 I want these epic shoulder pads. Look at these. I thought when I saw that picture come out that I was like, oh man, I live in L.A. So I thought, should I go to a costume house and see if I can find that in some old Battlestar Galactica relics vault somewhere? Anyway, so hi, my name's Evan Phoenix. I'm gonna talk about services but a little bit about myself first. I just have started a new company called Vectra which I'm happy to talk to anybody about afterwards and it has lots to do with services so that's sort of why I'm talking about services today. But my first job is really Zoe's dad mostly so I couldn't resist putting a picture of her in here. So this is a little weird, I can't see the next slide so we'll just see what happens here. So what am I gonna talk about? Services, services, services, services, services, services. Sorry, this is my best Steve Ballmer impression there. So everybody talks about services now. Ah, we should write services and sometimes people talk about, you should write microservices. A lot of this is sort of, the talk is mostly meant to talk a little bit about microservices and to talk about the fact that you already write services, you already are working in a distributed programming, distributed network programming environment anyway. So there's big services and small and medium size and people throw around these sizes of services like all of a sudden we know what a 44-ounce service is versus a 12-ounce service. Like all of a sudden everybody just understands what a big service means. We're gonna try and boil it down a little bit and talk about mostly what the smaller, what kind of idea of a microservice is and maybe why you wanna build them, maybe why you don't wanna build them. Microservices. So Martin Fowler has a really great quote here which is that there are yet another term on the cruddest treat of software architecture. They don't, I mean people throw it, you'll see this a lot of times in blog posts and that kind of thing. And one prevailing idea will always be present in anything you read about microservices. No one agrees on what they are. No one agrees on what their size is, they just agree that there's this term that they'd like to argue about mostly. So hopefully we'll come to some conclusion that I don't necessarily know and you won't really know and that it's kind of a meaningless term anyway. So the crux of the idea is that you wanna structure your application into some composition of services. So rather than having a big monorail that has all of the code inside of it that has a hundred controllers that's doing everything, you wanna try and structure whatever you're doing as this composite, this mosaic of services that are gonna talk together. And the idea is that you can isolate this functionality into different code bases and that sounds great. It's just, let's just do that. And once you've done that, that's obviously just gonna work out great. It's just gonna magically come together, right? Well there's a lot of work that goes into it. It's probably more work than building a big monolith anyway, but there's a lot of benefits you can get out of it. So the first of all, what's a microservice? Like how should we try to define that? There's generally these sort of three definitions that you kind of will come across. One is this idea of a single data type. So you wanna have like a user service, something that maybe it's got endpoints to like get a user's name or add a new user or something but it kind of collapses around this single data type. Another is a single operation. So it may not be, it may be something that operates on one of those data types, like a login operation. And all this service does is do login. It doesn't do anything related to getting users information or anything like that. It just does login, right? And another one is sort of this logical grouping of things. So you may have one that does email. So an email service may be able to send email and be able to compose. It may be able to receive email. And obviously that is sort of similar to a single data type, although we don't necessarily think about an email, the actual form of an email as a data type. It's usually something that's created and sent and serialized and brought back in. So it's more of a logical grouping of operation. So as you kind of go and you're breaking up things into services and you're thinking about them, the big benefits we've kind of hit upon that earlier is one of them is this responsibility isolation. The general don't break my shit rule, which is when you get into a big, and I should have done this at the beginning, who here has a web app that feels too big? Raise your hands, right? So that feel too big comes from someone else in some other team working on some features that you're not working on, breaking your shit, basically. They add a scope that, or they say like, oh, logged in. Well, there's already a logged in scope, but actually logged in for my idea of logged in means something slightly differently. So I'll go change that scope. And now all of a sudden, your code that expected the old behavior of the logged in scope now breaks because of that, right? And so as you kind of get larger, you kind of, you start to break things. The other thing is that it forces a modicum of discipline. We'll get into this a little bit in the next slides. You can't instant see Val, which you can't see. And one of the third benefits here is you've got these separate velocity vectors. When you have one big code base, everything moves forward within that code base. Even if you have separate features that maybe are worked on and that are committed back in, they actually, they maybe can move at a different speed, but once they're committed back in, they're now on the same release schedule as everything else. And so you'll get teams that will say, hey, you know what, I just finished this feature. I'd really love to get it out. And you'll have another team that says, well, that's super. But you know, we really need to wait until next week to deploy because we're working on this other thing and we're not really sure if the code that was already merged in, which we sort of merged on accident, sorry, is broken or not, right? That happens a lot. And part of that is the discipline, but part of that is just that that's life. We're fallible humans. So that first one, responsibility. This is this idea of the surface area of your application. It's the behavior that the application is known to have. And the, you wanna constrict it. You want it to be as small as possible, but you don't want it to be too small. There's that saying that you, I can't remember the saying, so we'll just give that. But basically, if something is so small that all it says is like, go get this thing from someone else that you don't really know much about that anymore. If that's all of the surface area is that you have to go off and look at something else anyway, that's probably too small. But you wanna be able to have, you reason about it and look at it and understand what its specific behavior is, hopefully in isolation. And if you kind of, as soon as you get to an area where you're, oh, there's this control over here and it does something I don't understand and over here I don't understand that, that's kind of where you're getting to a large size. It's kind of too much surface area. The discipline factor. We're lazy programmers, that's fine. That's literally what being a programmer is. It's being too lazy to do something so you write a program to do it for you, right? And these service boundaries really keep us honest as programmers. So if you've, like my example here sounds ridiculous, but I've had this conversation before, which is, you know, oh, this thing's being really slow, what am I gonna do? Well, sometimes in Ruby you may say like, oh, okay, well, I know that they store this little token where that I need somewhere weird. So let me go write this big crazy thing that digs in into an object and does a weird Instasy file and grabs it out, or maybe uses an API that's deprecated. And I do that, I know I shouldn't do that, but honestly, I'm being kind of lazy. I don't really want to, I don't have time to go fix that thing. Those service boundaries set up these things that are very rigid. You can only go right to the edge of what the expected behavior is. And so that allows us to sort of be creative within those constraints and not, you know, okay, great, maybe I'm hitting this boundary because I really want to do, I really want this piece of data and it's only exposed in this really slow way. Maybe that means I need to talk with the other team and come up with a new API that allows us to do something better. So you drive development of the actual application in a more reasonable way. And velocity. So like I said, unique services can have their own unique release schedules. You can do feature flags, you can roll things out at whatever rate you think is reasonable for your individual thing. And let's be honest, shipping code, putting it in out there is really what makes people happy. And so if you're getting into a rhythm where you can have continuous deployment, you can have services that are constantly running off master all the time and they get deployed 10 times a day. That's a pretty great place to be that everybody's comfortable with that. And that's really almost what you want to strive for. So great. Wait, is that the next slide? Yeah, it is, okay, cool. So great, awesome. Let's just do that, right? That sounds amazing. Everything's fixed. We can go home. Everybody gets off work early that day. We're doing services. Yeah, well, there's plenty of pitfalls and that's really the majority of this talk is about is what those pitfalls are that you're gonna hit as you try and go implement this kind of idea. But let's do it, right? So from here on out, this talk is called Evan's Practical Guide to Services, TM. We're gonna go over the dos and don'ts and what pitfalls you're gonna hit, what decisions you're gonna have to make and hopefully when you're gonna have to make those decisions. So you've got a big monolithic application. You wanna break it up into services. You gotta figure out the boundaries of those services or even if you don't have a big monolith, you have to figure out still, I've got some thing that I wanna build. Where are the divisions? Where do I draw these lines about how to actually build the individual services? So the natural place to do this feels like is the controller and this is typically where people start. I'll just take this controller and it's associated code and I'll move it over here and sometimes maybe it makes sense to move one controller. I've got this controller that does the user service or whatever it might be. Perfect, let's just do that. The minute that you do that, you hit this problem. Oh, okay, great. So I wanna move this controller over here and this controller over here and this controller over here but those three controllers all use an active record model called user, group, whatever it might be. It's usually those kind of generic terms and people will say, hey, let's just share them between all the services. Never do this. Never. It will feel like, oh, it's not that big a deal. They're obviously gonna have the same code, right? So it's easy, let's just share them between them. It will be okay for the first week and probably about the 10th day that you do this, someone will add some code to the shared active controller or active record repository for this thing that will break something else and then you will turn into this big cycle about what exactly is going on and everyone will be now afraid of the models. You get this shared model pane and I actually, this obviously is hyperbole but I contend that this is totally true because I've seen shared models and the bit rot in those shared models is so much worse than everywhere else in the system and it's because everyone is deathly afraid of the models because now they're not being used by this one giant machine, this one giant code base. Now they're being used by 10 different code bases that you don't really know how quickly those other code bases are moving and what they're doing and everyone gets really afraid of touching anything inside the model and so individual apps start to, individual services start to get like wrapper classes and all kinds of stuff in order to deal with these shared models. Just don't do it. So maybe we go back to what we were talking about before about doing controllers and maybe we say, just ignore the controllers. Don't just forget their controllers. The controllers are usually stateless anyway and really put each model in its own service and then move all the controllers that are associated with that model in there and that ends up being a much more comfortable way of starting, breaking things up into services. Now when you do that, you will get a point where you'll get touchers. You basically, you'll get controllers that will say like, let's say it's the, I don't know, some random controller and it will know, it will wanna know the currently logged in users full name. It's a really common one, right? And so it's, maybe it's gotta, it can read the session, it can see what the user ID is and then it's gonna wanna go, all right, I wanna display, hello Evan, good morning or something like that from the user ID, great. This point in time, you know, okay, well this doesn't really need the user model but it touches the idea of a user and so this is the place that you wanna say, okay, now I need to go talk to this other service. So now not only have you split them up but you've gotten, you now have this really nice place that you know where to actually put the calls to do the different services. The first time you do this, you'll say, oh okay, well what I'll do is I'll have another user class and that user class will just basically go and do services over the wire. That, this is generally what happens. Be super, super careful when you do that. Because what you get is something that looks equivalent like this. Oh okay, in this particular example, I'm getting, I wanna print out the names of all of the users in the system. So I'll go ahead and I'll just call, I'll do users all and that's gonna go and it's gonna hit the user service and it's gonna give me a list back of all of the users by IDs and then I'm just gonna loop through those and I'm gonna say, okay great, give me the names of all of those users and now when I do that, I'm gonna be doing a 50 millisecond network call to another service for every single name. And so you get this exactly the same as you would an active record, you get an N plus one query problem, but now you can't see the query. Now it's just this thing that's just like going over the wire and coming back and maybe it's an RPC. And so you, people, this happens a lot and this will kill the, this is generally the first performance problem you hit when you start to do services and when you hit this, you'll have half the team saying we never should have done services in the first point, first place, this shit sucks basically. That's just what will happen. So you should know going into it that you're gonna hit this if you're generally not careful. And this can easily bring down your efforts to break things into services. People will say, this is too hard. How am I gonna know how to do services? Obviously it just causes huge performance problems. It's 10 times slower than we were just using active record to forget it. I don't wanna do it anymore. And we'll get into a little bit how you solve this problem. So let's say that you have some green field application that you wanna do as services. I present that what you should do is just do a whiteboard architecture because if you go in and you basically get your team together and you have them draw out, okay, great, we're gonna do a social network, God forbid. And you're gonna go and you're gonna draw boxes on a whiteboard of the different components that in a social network, I'm gonna have a time feed and I'm gonna have chat over here and I wanna do images and people need to send pokes to each other. Whatever the boxes are, it doesn't really matter. But you inside there, that team is gonna have some intuition about what those boxes look like. Great, make those your services. Start there, that's the perfect place to start. And they may be wrong, but I contend that your intuition about this is actually really good. And the intuition comes from a lower cognitive burden for knowing how they fit together. So you're gonna have these services, you're gonna have a service called Send Someone a Poke. And that makes sense, people are gonna understand that because that was the box that you drew on the whiteboard and maybe we're gonna have a timeline service. Again, that's gonna make sense because that was the intuition about what this thing looks like. You can get a lot of mileage out of this. And once you finish with this, you're gonna say like, oh great, there's all these services that we didn't think about that we actually need perfect. You gotta get the ball rolling before you get there. So you've decided how you wanna break stuff up. Great, wonderful. Now you gotta decide how you're actually gonna get it out there. I contend that deployment ends up being a huge burden when you try to switch the services. Now you may be saying, well, I already deployed one app. What's the big deal here? Unless you're really comfortable with that deployment process, you're gonna, you are gonna, it's gonna get 10 times worse because now you're gonna be dealing with it for 10 other apps and they're gonna get 10 other weird configurations of things. You need to be really, excuse me, excuse me, you need to be really confident with what it looks like. Because after all, this is my tech version of if a tree falls in the forest, does anyone, if a, you know it. You need to have a very comfortable, reliable, boring deployment process because you really want it to be something that fades in the background. You wanna just be able to deploy and go on with life and not worry about it. You don't wanna be like, hey, did Unicorn restart again? Is it crashing? What's going on? You really wanna be super, super comfortable with this because you're gonna be doing it a lot more than you used to in the past. Try to have one deployment system for all of your apps. Don't have one thing that uses this overheating, one thing that uses this over here. If you have to say, have a Ruby app and you have a Java, you have a Ruby service, you have a Java service, at least put them in the same wrapper. So be able to type one command for all of your different apps and be able to deploy them no matter what language they are. Because you don't wanna be able to have to switch apps and have to remember, oh, do I use Maven or SBT to deploy this app? You never wanna be making that, thinking that at the end of doing some work and trying to deploy it because you basically will lose hours out of every single time you wanna do it. There are generally good deployment or good-ish deployment solutions out there. It's Capistrana, there's Paz's. You can write your own custom thing but figure out what that is and be set in your ways about that particular solution, whatever it might be. In your deployment system, you really want one configuration system and when you start off, when you have your big monorail, typically you have a config directory and it's full of YAML files and those YAML files have a whole bunch of config values in there. That's not gonna fly anymore. You're gonna have to pull those out and put them in somewhere more generic because you're gonna wanna share those tokens amongst a whole bunch of different services and you're not gonna wanna have to go and edit a bunch of config files in 10 different places when, say, your air brake coke in changes. You really wanna be able to do that one place and have everything pick it up. Quick tip, this is related to logs here. Do a fire drill once a month once you start to get this going. Do a fire drill where you pretend in the morning, just do it Friday right before lunch, take an hour, half an hour, whatever it might be and pretend that the site is down and go through all of the steps that you would do to diagnose why the site is down with your team. Sit in a room, pretend, have it, maybe do it the end of the day on Friday with beer, whatever it is, but do it at least once a month because as you start to do it, then when the site actually goes down, you will have practiced. You will know what everybody's supposed to do. Everybody will sit in the room and they'll go like, hey, who's, okay, Joey, you're checking Redis, okay, Rachel, why don't you go pull up the logs for service number four again and you'll all have done it before and it'll be a very easy thing to do because when you get services, you have to go check six or seven different places with the different logs for each individual service and you have to correlate them and the last thing you wanna do is be doing that for the very first time when the CEO is standing behind you. Don't have that be the time that you're figuring out was the, did I put the logs on this machine or this machine, are they being our syncs? Figure that out and just practice it. It goes a really long ways. So, great, you've got your services out, they're all running in different compartments wherever they are, somewhere in cloud space, excellent. That's really not super useful until they can talk to each other, right? And this is really the big thing that your team will discuss a lot. This will become a very core part of the discussions that go on. Is that whole member sun was like, the network is the computer anyway. Really, that's what you end up doing. That whole sun marketing campaign was the fact of like, let's build services that talk to each other, awesome. Remember, the network is kind of fragile, so you're gonna have to do things like expect failure from your network. So, if you go to talk to some service, whatever it might be, it may not be there. Something may have happened, it may have crashed, whatever. Don't have some reasonable guidelines, some reasonable deadlines for talking to a service. So, John Hodge has a really great, John Hodge just has a really great talk about, talks about back pressure. This idea that what you really wanna have a service do is return errors rather than just try and continue to accept requests and crash. And a lot of that is around just time. So, you may say, okay, I'm gonna go talk, I'm this service, I'm gonna go talk to that service, and I'm gonna set a reasonable limit where I'm gonna wait for the reply, and that may be a second, it may be 100 milliseconds, whatever it might be, but you have to figure something out. You wanna basically just say, that's how much time I have to return that result. And if it doesn't come back in that amount of time, I have to figure out how to continue on without it. And continue on might be return an error to the person who asked me for my operation, because that's usually how these things are going. You've got some service in the middle and time's not talking to something. Bubble that error back up and have it be presented somewhere else. But don't just block forever waiting for someone to do something else because that's basically how everything just grinds to a halt because Redis ran out of log space. That's typically how services grind to a halt by the way. Also be aware of priority inversion. So it's not uncommon as you start to get more services where you get service A talking to B talking to C and C needs something that A also has and A starts talking, C talks back to A. You can get these priority inversion deadlock loops within your services, depending on what A was doing when it needed to talk to B. So if it had some resources that it actually had a mutex on or maybe it was doing just something generally that had to do with locking or who knows what it might have been doing. You have to be very careful with being sure about those services talk to each other. And the easiest way of figuring out exactly what's going on there is to add tracing. Add when you, let's say you get a request, a web request and you're gonna go and you're gonna go send someone a poke or whatever it is. Great, get the request ID or generate one, just a random one, a random UUID it doesn't matter and start passing it down in every call that happens to every single service. So that every service that wants to go through and perform that operation can log why exactly am I going and I figuring out is this person pokeable? At least then you can print that out because you're gonna wanna go back and look at the logs and correlate them to those request IDs to figure out why am I deadlocked here? Oh, I know exactly why. This request came in to talk to A, talk to B, B talk to C and then C talk back to A but I know that in that code path there's a lock. Bingo. But you have to have those request IDs in order to be able to trace that through to understand exactly what's going on. If you try and do it any other way you will basically have to just tear the whole system apart and put it back together and that'll take like a week. So we've been talking about how services talk to each other but or the general idea of services talking to each other but not the how and the what. So the how they're talking to each other and what they're saying, right? It's gonna be tempting to do some real easy solution. Some will come up with like, oh, let's just use X, Y, and Z whatever that solution may be. Be careful. You really wanna have one convention, one mechanism, one protocol that all your services use to talk to each other. Don't have service A and B talk to each other in a way that B and C don't. And the reason is that you really wanna have tooling around all of those communication mechanisms and if you have more than one of them you have to have double the amount of tooling and double the amount of understanding and now you have to be sure that request IDs are actually flowing from system A to system B and it's just a lot of additional cognitive burden. And you really want that idea of the A talking to B you really want it to melt away. Like people are gonna talk about it a lot in the first month but you really want in month three people are just sending messages back and forth between services and it's no big deal. You want it to be second nature to your team. You want them to just say, okay, yeah, we're using services and we pass JSON between SQS, it's no big deal. You really want it to be boring. You want it to be this thing that just exists within the system and it just is sort of there. Sort of like whether or not you've got cat on the system. You want it to be this simple traditional, like just don't worry about it. My example here is you don't really debate how a method call works in Ruby, do you? I hope not, right? It's just the thing that's there. You just use it. You really want the communication mechanism between services to be exactly the same kind of thing. So, the how. I thought this talk was gonna be short. I was wrong. Okay, so how? There's generally three ways of doing this. RPC doing rest and documents and doing message passing. We'll talk a little bit about each one. RPC. RPC is really just, there's a method or a function in some other process and I'm gonna make a request to it and I'm gonna wait and I'm gonna get my response back. Be careful here. Remember the deep RPC problem that we hit before where you get this N plus one kind of stuff. It's tempting to make RPC. You're moving from a system where you could just make a method call and you get a response but where that hop to hop look like that and that was like six nanoseconds, now it looks like this. And then it just waits around over here and that guy's just stalled. He's not doing anything. And then oh, maybe I get the response back. You really wanna be careful about how if you're using RPC, don't hide RPC calls inside innocuous methods. It's the best way to get fired. This is bad. This first one. Don't make a user object that contains a method called name that happens to go over the network from where you are to Singapore to go grab the value and return it back. You basically will cripple any program that uses this because no one is gonna expect that name is a 800 millisecond operation. They're gonna expect it's a two nanosecond operation. Make the calls obvious. Anybody who uses the second one is gonna be pretty clear that I'm going and calling a service to go get some value and return it and they're gonna be aware, oh, you know what? I probably shouldn't do this in a hot loop. That would probably be bad, right? And so they're gonna come up with some better way of doing it, talking with the team, coming up with a service that will get them the bulk reply, whatever it is. Just be careful here. As you're, when you're using Marshall or when you're using RPC, a lot of times you end up using Marshall. It's easy. We're in Ruby. Oh, I just, I will Marshall the values back and forth. I don't pass them over here. That will be fine for the first week or maybe the first month, but you basically, you'll hit, you vastly limit the amount of work and the kinds of architectures that you can build by doing that. I mean, don't, by doing that, you're basically inventing your own protocol. And unless you're, like I said, unless you're in the protocol business, it's not really your core competency, they say. A good example of this is Finagle. It's a Twitter library for the JVM. It's not, if you're on JRuby, you can use it. I bring it up mostly because of the idea, not the exact applicability here, but it's an asynchronous RPC library for the JVM. And the idea is that you call something and you get this future back. We'll talk a little bit about futures here in a second. But it's a very good example as you kind of go look through of what a RPC layer for services would look like. This is their diagram. I'm gonna talk about each, I'm gonna talk about testing in each one of these ones because you have to do it. You have to know, you have to build a mock for every single service you use. You have to build something that someone else is gonna be able to use to test against. And RPC testing is really easy. It's just, you just use a normal mock, generally. Because in the language, it looks like just making a method call and getting a value back, right? Even if it's against a module like user service, it's still just a method call. So RPC testing is by far the easiest. So the next one, rest and documents. So this is the APIs that we're all accustomed to now. You're building some API that is externally consumable by users, clients, whatever it might be. You're gonna build a rest interface probably. You're probably gonna have it return JSON. Great. Your data is hopefully gonna be denormalized. You're gonna be returning large documents, not things that say, oh yeah, this is for user number four. And you're gonna have to go, okay, great, what? Well, okay, super, what's user number four? You know, you're gonna basically return the user information in the user spot, right? Be careful here. This is the idea, HTTP JSON and rest is a very vague idea. It is not a protocol. There's a lot of different ways of structuring your data in JSON. There's a lot of different ways of structuring your URLs. There's a lot of different ways of using authentication. If you don't set something down in stone, you will have a constant debate about what the APIs are supposed to look like. So you have to standardize. If you go around this method, you have to have a standard. You have to say, okay, great, services that we're gonna go ahead and use HTTP JSON. Here is how a URL must match. You must have the resource. You must have the action. You must use get and put and post for these exact kinds of things. If you wanna represent a collection in JSON, it must look like this. It must not be an array at the top level. It must be wrapped in an object. You have to define those things because if you don't, someone will decide that they wanna do it a different way and they will go off and do it a different way and it will cause constant havoc within the team. Don't leave room for flexibility. Make this a constraint that people have to work within, not innovate. The other thing that you get to when you start to do REST and HTTP JSON is clients, specifically client libraries, client gems. So I've got some API that exposes some HTTP JSON service. Super, probably whoever is using this thing is gonna find it, they're a programmer, remember? They're lazy. They're not gonna wanna formulate an HTTP request every single time and write raw JSON or do this thing. They're gonna wrap it in some Ruby client library to do it. When you do this, it will, okay, first of all, if you don't put something down, you'll get three or four competing client gems within your organization for different services. So service number one, we'll be using their own version of this thing. And service number two, we'll be using one that was actually originally copied from this other one and it's been slightly modified. So you'll get this really crazy thing where which client is using which version of what thing and it'll be, it's kind of a mess. Also, you'll be forced to write your API twice. You'll write it on the service. The service side, you'll write this HTTP JSON layer in the middle that may be super elegant and just absolutely delicious. And then you'll hide your deliciousness behind a Ruby API. Why did you do all that middle work, right? So be careful here. Standardized again. If you can auto generate your client code, make a standard for what those HTTP JSON APIs look like. Have code that can read some document about them and generate the Ruby code that everybody uses. Try not to hand write it. Try to have them all feel and look the same. You'll derive infinite benefits from not having to hand write them every single time. Testing. We're doing, this is the same kind of testing you do right now for a service. You can use a gem like VCR, or anything that really looks like a remote service. I have made an HTTP Fetcher class and just mocked that to return a response before. Whatever it looks like, but that's what it's gonna generally look like. So the third one, just wanna see where I am on slides, okay, we're getting pretty close to you. Message passing. We talked about this in the very first talk. The idea is that you're gonna make all your communication about what happens asynchronous. You're gonna send a message to some thing. It may be a pub sub, it may be some, just some name, but you don't really know who you're sending it to. And you're gonna get some response back. That decoupling forces all the messages to have to be really clear and concise about exactly what they're trying to convey because you don't know who you're talking to anymore. So you have to be very explicit in your messaging and your format and all that kind of stuff. That's good, you want that. It allows you to do other unique sort of communication patterns. Whereas before, we're generally talking about request response, request reply. Now you can do stuff like broadcasts. You can say to all services that are interested that someone just logged in, hey, someone just logged in. You can do pub sub. You can basically, which is very similar to broadcast. As you get into doing message passing, use a broker. Don't use zero MQ. And I tell you this because when you need to use zero MQ, you will know, right? It will come, you will know, you will feel that. You'll wake up one morning and be like, this is a problem for zero MQ and that's great. But as you're getting started, don't do it. Use a broker. Use SQS, use rabbit app Q, use something. Use a broker that's gonna manage the message passing between all of your different components. It's just a much easier way to get started. When you're doing message passing, it can be very cumbersome because what you get is this idea that, okay, in this particular example, you need to get some user's details. So let's say I have a user ID and I need to go talk to the user service to get that user's information. How do I formulate that in terms of message passing? You can do it one of two ways. You can make a get user details message. You can send it on the wire. You can have that code block and wait for a reply on some kind of ephemeral queue, which you get into. You'll see this as you kind of get into message queue, message passing. And you can just basically turn it into RPC at that point. Or you can write all the state, all the information about why you wanted the user details in the first place somewhere. When then when you get the user details message, reconstitute that state to go out. This is cumbersome. I won't lie. It's very cumbersome when you're using message passing because of this kind of thing. You have this additional overhead of trying to figure out why was I, why am I getting a message that says user details? That you have to figure it out every single time. And so it's very awkward, but it can be at the same time very powerful. That's why message passing and message passing frameworks have different mechanisms. They have, you know, if you go look at the documentation for RouteMQ, you'll see, okay, here's how to do RPC and here's how to do broadcasts and here's how to do pubs up. Great, use those different ones for those different cases as you need to. And it's very difficult to master how to do the message passing the right way. Don't worry about that. Just get started. Just start doing it. Get a feel for it. That's how you're gonna get good at it and that's how you're gonna integrate into your application. Testing. The best way of doing this that I know of is in your code, you abstract away the idea of sending and receiving messages. So you basically have an abstract broker and you substitute in testing in something else that can receive and reply to those messages just in a testing-like way, basically like as a mock. And then you can basically drive individual components by, I'm gonna inject this message and I'm gonna see what that person says. The person said, oh great, I injected get user details and it admitted a user details message. Excellent, I can go on with life. But you wanna, again, be sure about how you're gonna do this as you start to roll it out because you wanna have your testing strategy in hand before you start. So as you start to talk between services, you're gonna have, there's a delay in there. Remember we talked about that deep RPC problem. So you have to think about how and when you're gonna go off and talk to those services. You generally don't wanna talk to them in a loop. You don't wanna go off and synchronously do this hot loop of I'm gonna do a call, I'm gonna do a response. I'm gonna do a call and they get a response. One way of dealing with that is as I mentioned earlier, denormalizing documents. So if you're using HTTP JSON, arrest, don't have the response to whatever the person asked for be complete in this particular document. Don't return references to other services. Return the whole thing. So if I'm with the user details, the user details document that I get back should be complete. It shouldn't have things like a reference to say, you know, a parent ID. Okay, great, that's super not useful because if I need the parents, now I have to go back over the wire and go get this thing again. It's better to just include the names of the parents right there. And the reason is that the data that you send back over the wire, even if you send redundant data, that is a ton cheaper than formulating a new method call or a new RPC or a new HTTP call, formulating it again and getting a response back. The additional cost of say, you know, 30 or 40 bytes is way cheaper than figuring out how to do yet another call and blocking. Another way, and I mentioned this with Fnagle, is to use futures. Also called promises. And the idea is generally just you have an asynchronous method call and you only block when you actually get the value back. So here's a quick example. So I'm gonna go off and fetch this document. At the same time, I'm gonna go often, I'm gonna go basically do another service call, but in this case it's to my database. I'm gonna go get a user. And then I'm gonna set the document basically to whatever I got from this remote service, right? And if both of these things take 50 milliseconds, then I can do this whole thing in about 50 milliseconds. Whereas if I didn't use a future, it would take 100 milliseconds. And it's because I'm not, I'm starting operations and I'm only blocking when I actually need the values. I'm kind of getting them started and then wrapping them all up at the end. And as you kind of get more and more services, you start to see this a lot more. You may say, okay, to answer this question, I need this value from this service and this value from this service and this value from this service. And then I'm gonna combine them all at the end. Okay, great. So you'll do three futures to go talk to different services and then you're gonna do some composition operation at the end. And it becomes a very teeny little cute version of MapReduce where you're setting off all these things and you're just gonna wait from all to come back and do something with them. We're almost done. Service discovery. Or where did I put that? You've got all these services and they know how to talk to each other or they know what to say but they don't know how, who to say it to. You need some kind of service discovery. I don't care what it is. There's a lot of different solutions out there. Config files, certain deployment, internal DNS using tools like console or STD, whatever it might be. But you wanna make the decision early about how your service discovery works as soon as you can and you wanna use it everywhere. You really wanna have, if it is the wrong solution for what you're doing for your problem domain, you wanna know that as soon as possible so you can switch. So you wanna induce as much pain in it as possible so you can say, we tried doing it in config files generated at deployment time and we're having all these problems. Okay, great, that's fine. Go on and try something else but you wanna force the issue. You don't wanna say like, oh, well, this service is using service discovery right now but this service isn't because, well, I don't know if I wanna deploy it yet. Just do all of them at one time and kind of feel the pain and migrate them over time. You'll get to a solution that works for you much faster. And again, you wanna eventually come to something that's comfortable, stable, and generally boring. So you've got these services that can talk to each other, they know where to find each other. One of the benefits you get from having all these services is you can have multiple languages. And this is again why you wanna have very rigid standardization about what stuff looks like. You can use different languages for different tasks. The big drawback is generally that you can't reuse code between your services and this can be a big issue. It's really different than the shared model problem because what we're generally talking about is tooling. So if you've got services talking to each other you're gonna build up tooling code for how those services work. So if it's maybe you're using SQS you're gonna build up some tooling code for how to send messages between SQS and how to serialize them and that kind of thing. And if you decide I wanna write a new service and I wanna write it and go, well now I gotta redo that tooling. And if you do make that decision, write the tooling first. Don't cobble something together without the tooling it just is a half-ass measure and just put it out there. Write that tooling at the very beginning because you're gonna use it a lot. And so take the time to write it. If you can't be burdened with writing the tooling then you probably aren't ready to be using multiple languages because that is a very important part of having services in all these different languages. So we're at the end. So should you start breaking your thing up into services? Well, I mean you already have services. You generally use a network database. So you have your big Rails app and you have a database and maybe you have your Nginx or something on the front end. You already have services. It's just about the scale and the size and the composition. You have to be willing to put the right pieces in place at the beginning. Don't start to put it together. Don't half-ass it basically because you will get yourself in such a huge mess. You'll wish you never even started. Get your tooling solid. Don't skimp on the tooling. I say this a lot but it makes a huge, huge difference as you get down the road because just like the fire drills you're gonna need that tooling. You're gonna need to go and say, okay, how do I, why are there no request IDs from this particular service? Oh, that service isn't using the tooling. Oh, great. Now we can't figure out why this thing is crashing. You need that tooling. You need that infrastructure in order to deal with the added complexity of having services. But the big thing is that it's not a question of should you do services. It's a question of when will you do services? And Brian talked about this at the very beginning and it's almost like we synced up on this and we did not. Just a happy coincidence. Brian said there's, it's 2,000 lines, it's 20,000 lines but there's some barrier with which you get to that a single human being and a single human cannot hold the surface area of a particular problem domain in their head anymore and they have to add layers of abstraction so that they can not worry about other pieces. And that happens with a large application. You get these large applications that are larger than, I just made up this term. Do you like it? Human cognitive mass is sound good but you get this big thing that you can't reason about anymore and you have to get it under control in some way and generally the way you get it under control is you break it up into services. You give each individual piece, its own surface area, its own piece of responsibility. And the corollary to this is obviously that every sufficiently complex domain is gonna have an app larger than this threshold. So if it's a quick little app that maybe sends a text message to Twilio and turns on a light, it probably doesn't need to be broken up into services. But anything that you do that's sufficiently complex that you're probably gonna run some kind of business on is going to need to be broken up into services at some point in time. It's sort of a matter of when. The other thing that I wanna point out is that I contend that the team threshold for this size is lower than the individual threshold and that's because you have the ability to communicate between people is much worse than being able to communicate with yourself. And so those apps, that app that seems too big for a single person is really, really too big for 10 people. So microservices, should you do them? They're wrapping up here. Really just ignore the general hype and the name because it doesn't really matter. It's all about what's comfortable for your team and your size, what makes sense for how to break something up. And for Peek's sake, don't just make services and decouple stuff because that seems like the cool thing to do. You have to be comfortable with it. If you're not comfortable with it, it will just cause everything to burn to the ground because it's a lot more complex than having a big monolith. Do your homework. Be ready for the network to fail. Spend time on your deployment environment. I literally meant it so much. I wrote it in the same two slides twice. Really don't do it just because it feels like it's something that everybody's doing and you wanna hop on the service wagon. Be ready to do it when you do it. Otherwise, don't just, but don't worry about it. Tooling matters. You got to figure out all the details. Don't skimp on the details. I feel like a preacher up here. Really, you have to do those things. They make everything happen and you will build services because you already do. And I'll leave you with two quotes as you start to beckon onto this great world of really complicated pieces of software. First, one of my favorite quotes by Voltaire. The perfect is the enemy of the good. Don't worry about getting this excellent, super amazing service-oriented architecture together before you start using it. You'll never do it. Get something that you're comfortable with and just start going. Go down that road. Don't feel like you have to get everything right to begin. Just be comfortable with what you've got enough. And this is about your big monorail app. That sacred cows make the best hamburger. Don't worry about the, people will say, oh, but it's a million dollar app and I can't break it up and that kind of thing. That's the best reason to go off and do that. You have to figure out how to do that because while it may seem scary, the million dollar monorail app could probably be a $2 million set of services. And with that, thank you.