 Okay, so today I'm going to be speaking a little bit about this, but I thought I would adapt the content a little bit considering that this is a FOSS Asia and not FOSS Node. So this is a talk that I've given before a couple of times. I gave it in Korea, I gave it in Australia, and I also gave it here. Here's a picture of me giving the same talk in Singapore. These are some other things that I've done. I do SingaporeJS, CampJS, which is a conference in Australia. I work for NodeSource and sometimes I host the Node podcast. Those are my, that's my Twitter handle, don't try to pronounce that, and that's my GitHub. So the primary thing that we're going to be talking about today is over-engineering and how we can take some steps, get some ideas on trying to curb the urge to do this. Also what is over-engineering, it's the adding of unnecessary complexity. And as engineers, anybody who's involved in any kind of complicated projects, you're going to be 100% aware, you will have experienced this at some point in your life. You may be experiencing it right now. So complexity is, in my experience, the absolute number one killer of all projects that I've ever worked on. If there's a problem with it, it's usually related to just complexity getting completely out of control. When you've got a complex system, it's difficult to understand it, which makes it more difficult to fix bugs, makes it increasingly difficult to add new features. Over time, projects end up becoming slower and slower. If you want to add a new feature, it ends up taking three weeks instead of what you would expect to take a day or a few hours. This is just, overall, a bad problem. Complexity is something that you want to do. It's your enemy number one as an engineer. But unfortunately, you as an engineer, you are the thing that's creating it. We have a very strong urge to over-engineer and hence create complexity in our projects. So this urge to over-engineer, it probably comes from experiences that we've had in the past. So almost all of this have been bitten by projects which have been under-engineered and perhaps and also all of the documentation that we read, books and blog posts, most of them are about trying to solve the problems of people who have under-engineered. But in my experience, there's been a lot more people who are suffering from the curse of over-engineering than under-engineering. And in fact, I think that a lot of these books and blogs are harmful to developers' minds because they encourage them to generate solutions which may have been designed in a completely different set of constraints. So what I'm trying to say is that there's things which work for a multi-million dollar startup which cannot have failure at all, it must succeed, and it may have hundreds of employees. It's a very different set of constraints to what you have if you're working in the startup. Or most typical projects have a very different set of constraints to what I believe a lot of these books are written for. And in trying to follow the books, I've ended up creating a lot of problems for myself. So I've been on both ends of the spectrum here and I'm trying to find a nice middle ground. So one of the problems that we experience is that we often build things that we want to build rather than building things that we need to build. So there's a good comic which covers this. So this is something that I think every engineer can relate to. And it might be a feature that you want to build because it seems cool or it might be using a new technology because you think that it sounds cool or it's something that you want to do to help you learn, but it might not be the appropriate tool for the job or it might not be the most efficient way of actually solving the current set of problems that you have right now. And so, yeah, we often build things that we want, not the things that we need. And actually a good way to figure out whether you're potentially doing something not for the right reasons is to detect whether you actually want to do it. Often the things which we need to do are the things that we don't want to do. It's an unfortunate thing. So if you think that you're going to enjoy building something, you're probably going to build it wrong. It's a horrible reality, I'm sorry. So another problem that we have is that we'll often spend hours in order to save ourselves minutes. And also saving ourselves, we'll spend hours solving problems that just will never happen in our production systems. This is something you're encouraged to do. If you're going to write a routine, a function, you want it to be able to take into account all the edge cases, the extremes, and you want it to be able to handle weird input. But reality is in a lot of applications that function will never need to do that. The things that you build don't always need to be completely robust. There's a good example here, another comic. So again, engineers should find this very familiar. It'll save time in the long run. So yeah, not everything needs a military grade solution. We don't need to be always designing everything to be at 100% quality, 100% of the time. This is a major problem that I experience. It's very hard to resist the urge to make something clean and better and solve a problem perfectly. When you know there's a way to make it better, yeah, it's difficult to resist that urge. I find that I'm often building the wrong thing for the right reasons. I can justify why I need to do this. I've got all the excuses in the world, but reality is it's often not the right thing I should be building. I don't need to build yet another package to publish on NPM. I don't need to create all this infrastructure when a lot of the time you can solve simple things using just a couple of lines of code. It's important to always be asking yourself these two questions. This will keep you on the right track. So always keep in mind what problem you're trying to solve and is my solution to this problem the simplest thing to solve that problem. And often we'll find ourselves solving problems in these big complex ways when, in order just to, for example, just to pass the test or to get something displayed to the user, you could have just done something simple. And I think sometimes we fear things which don't, which will never eventuate. So we think that we need to design all this stuff because, you know, scalability or whatever. Maybe once you've actually implemented it and it's working, you'll feel better about, you know, your crappy solution because you'll realize that you've actually got bigger problems to be, to solve. Another thing I've discovered is that when I go to do a proper solution, this can take hours. It ends up involving an awful lot of yak shaving. I end up diving deep into, you know, solving problems which will expose more problems because I'm trying to do everything properly. But in the end, I could have solved it simply. And I have discovered that five to ten times is about the order, it's the amount of complexity and time that I'm adding on top of the simple solution. So over-engineering is, I find, much more costly and more common than under-engineering. And one of the reasons why this is, is because when we over-engineer, we often end up with, we end up building the wrong structure. And the wrong structure can be more costly than having no structure at all. The most flexible piece of code is code that which hasn't made any assumptions. And you can come in and just grab some pieces and move things around. I found that sometimes when I'm working on a piece of rubbish code, I'm actually a lot more effective than something which has been, you know, very well designed because it's much easier to understand a straightforward, simple, you know, not very under-engineered piece of code than it is to understand something which has got a thousand design patterns and things going everywhere and code going all over the place. So I have a comic for this. Now this is an attack on object orientation. This is just what happens in pretty much every software project. It starts out like that, ends up like that. You know, this tree structure is pretty much, this is the ideal structure. In fact, a line, a linear line of dependencies and organization would be ideal. But obviously things require a little bit more complexity than that. So if you can get things into a tree structure, that's the ideal situation. But flat's better than that. As soon as you descend into a graph, that's when the nightmares start happening. So in order to be able to, you know, keep things as close to a tree as possible, we want to defer decisions about any kind of architectural things just because they're going to be more difficult to reverse. So, yeah, defer those decisions because in the future, you're going to have more information about whatever that problem is. And once you've got more information, you can make a better decision. But if you make a decision up front, in order to make a different decision, you need to first, yeah, back out of that decision, go to the new one. And if you are going to make a decision, make sure you've always got a recovery plan. You're always thinking about, how am I going to undo this? Because, you know, always assume that what you're doing is wrong. And make sure that you can always back out of things. So this is one of the reasons why I try to stay away from framework, framework code, because frameworks often, we can't back out of using that framework without making incredible changes to your code base. So try to do that. And another way to say that is optimize for deletion. So when you're building things, wherever possible, make it so that if you need to suddenly remove this stuff, it's going to be easy. So what that means is that if you've got a thing and then it's reaching into all, well, this is about managing dependencies, I guess. If you've got a thing which is reaching into all different parts of your application, it's going to be very difficult to remove or change the application. So ideally, you sort of try to keep everything together and make it easy to remove. So let's talk a little bit about Node. So Node has a culture of extreme modularity. And what that means is it differs a little bit from a lot of other software environments because everything's a module. There's a module for that is a common phrase which is uttered in the Node world. So there's modules for everything. There's modules for reversing a list. There's modules for, I think there's a module for no op, like literally a no op. But that has actually been recently superseded by no op 2. So anyway, Node has been, when you build a Node application, you build it up out of a hierarchy of lots of little pieces. And you might see, okay, well, if all the stuff's on NPM, this might be how to build an application. I should build my application up out of all these little pieces. But I've discovered that trying to do that upfront is very expensive because you'll often modularize something in the wrong way and you might need to reverse that or perhaps you've modularized it in one way. But you need this thing over here needs access to this so then this needs a reference to this thing and then now suddenly you've got a graph and that's when things start to go to hell. So the point is you should try to avoid premature abstraction until you really need to break something out into another file or into another package or to another module. Don't do it. And certainly don't start by building different applications. This is another common piece of advice. You know, the way to avoid building big applications is to not build big applications. But reality is that if you've got multiple applications, you're incurring a huge amount of overhead and making sure that those applications work well together. Much more than just making simple function calls. So the point is abstractions, they're often claimed as, you know, just if you're going to, when you're building something, you should just always be abstracting it and making it more and more abstract. But abstraction is not always a good thing because it does this to it. You decrease the local complexity but you increase the global complexity. So, you know, you may have experienced this in, you know, having very difficult to set up environments and things like that. That's because all this, you know, you've got like docker containers and all these bits connecting to each other. That's bad. So just be aware that any time you make, you add modularity or create an abstraction, you're just moving the complexity around. Sometimes it's worth it, sometimes it's not. It's just not always good. So the point is publishing to N, in the context of Node, publishing to NPM is not the only way to do modular code. You've got all these other options. So you can, even within, sometimes these days, rather than trying to find a good way to break something up into multiple functions, I'll just keep them all in one function and just break it up by having fancy comments in between the different sections. Sometimes later I'll break that stuff out into multiple functions. But the point is that when you, one of the key benefits of doing modularity is that you get this organizational benefit. It gets cleaner. But it's actually possible to get that organizational benefit of modularization without any of the overhead. And again, I mentioned microservices. Those things give you so little, you know, the same benefit that you would have got from just breaking that thing out as a comment. But, you know, all this additional overhead. Anyway, flat's better than nested. That's a common thing. People know this. So you should try to avoid creating wild goose chases for your colleagues. I mentioned that a little bit before. You know, we've all tried to debug problems where the thing goes into this thing and then goes into this thing, comes back around here. That's no good. And often these things are created by people designing overly complicated solutions. The simplest thing is probably flat. Let's get back to the main point, though. Modular versus monolith. The point is the sweet spot's probably somewhere in between. Maybe it's called a monolith. That's a pretty stupid name. I don't know. But anyway, usually the best plan is to go monolith first. It's kind of like the idea of you prototype your ideas first and then you can progressively modularize out as you need to, but only when you need to. And doing this progressive modularization you should end up doing less over-engineering and it should help you build the simplest thing that will work. Avoid unnecessary complexity. Build the simplest thing that will work. Thank you. Any questions? Do we have questions? Yeah. Questions for Tim. Anybody completely disagree? That's okay. Was that a question or were you disagreeing? Oh, okay. How do you make sure that it isn't misused? I've used as an excuse for just laziness. So the question is, if you use this approach, how do you stop it from being abused as an excuse for laziness? And I guess the thing is that you don't break things out until you need to and it's deciding when you need to. When things start to, when, I've discovered that in my apps when things start breaking and I'm like, whoa, what the hell's going on there? When things start happening and I don't expect it to happen, that's usually a sign that the thing's gotten too big and I need to start splitting some stuff out. And usually by that point, the things mature enough that there'll be some obvious slice points that you can make that you know, if I slice it here, this isn't going to have a huge impact on the app. So I guess it's just diligence. It is hard, but I feel like having an under-engineered app as long as, in both cases, if they grow out of control, you can write bad code by over-engineering and under-engineering. It's striking a balance. So the answer is I don't know. Sure. This article called the Majesty Monolith. It says why because Basecamp Code was kept in one project, the team, small team as his, had so much more benefit and when I read this, I actually thought of Tim's one. And I think last month, the Code, which is a Node.js application, open source application for blogging, the founder came over to Singapore and we asked him the same question because Node is a very modular thing and he was criticized by the core Node team for making a monolith project. So both DHH for Basecamp and John for Code, the two big projects that you can take examples for Tim's talk. You'll find that the people who are good examples of open source contributors aren't necessarily going to be good people to develop apps for you because they're going to, and this was me before, you know, I was trying to build, my allegiance was with open source and so I would sacrifice the app in order to build more open source modules. This isn't efficient. It doesn't build apps better or faster. It just sort of builds them different. Great for open source, not so great for the app. It's unfortunate. Anything else? All right, cool, thank you. Thanks Tim. Oh, one more thing. Hi, hi, hi. Cam.js. Just a little plug. I didn't run this conference anymore but it's one that I started happening in Australia. It's in Sydney. The dates are here, 3rd to 6th of June 2016. Go to this website. This is a good example of over engineering. It's all 3D. So I didn't build it. But anyway, please come to my conference. It's a lot of fun. It goes for the whole weekend. Lots of good stuff. I think, has anybody here been to Cam.js? No. There are people in Singapore who have been to Cam.js, just they're not here and I've heard that it's good. So thanks. Thanks very much.