 My name is Chris and I work at Square. And today I'm gonna talk about how we make a new service today at Square, specifically a Ruby service, because we also have lots and lots of Java services and we do those very differently. So Ruby, let me start first by talking about Square. So I'm on our risk engineering team, so we have a bunch of really small teams. One of those really small teams with two engineers that I work with now is risk engineering. So we have, because we deal a lot with money, there's a lot of risk with that. Some people do really, really large transactions and we wanna make sure that we kind of vet their business a little bit before we just give them all that money because it could potentially be risky for us to allow it to go through automatically. So the team I work on, we write tools to make that really fast. And then we also sit with our analysts and we try to automate a lot of the work that they do so that they don't have to do as much work. Also so we don't have to hire 5,000 people to actually look at all the transactions that come through Square. So the Square story kind of started. We had, well I wasn't at Square when Square started. I've been at Square for 11 months. But the idea was we wanna allow anybody to be able to process payments, right? So that's a pretty lofty goal, but it was really defined. So we said, hey cool, let's make an app for iOS devices, let's make an app for Android devices and let's make an API for those to talk to. So of course we use Rails for that and people started using it right. So we have this one API that we're using. Everybody's taking payments, it's working out really well. And then we're like, well maybe we wanna send people receipts now, right? So then we had five more controllers for receipts, couple more views, appreciate you guys' service is going cause I'm sure many of you have this problem, right? So the next thing we wanna do is make it so that users can log in and see their history of payments and sort them and search them and things like that. So we had 10 more controllers, so a bunch more APIs in the same Rails app. And it starts getting really, really big. So this is what it looks like today. This is our like monolithic Rails app. We have 183 controllers. We have 197 models and 657,000 lines of code. So this takes like an hour for our CI to run and every time you wanna add a feature, it's like at least two days before it's gonna end up in production cause we gotta run the CI once, it fails, you have to fix it, then you run it again and then you have to deploy and you can't deploy any time you want, you have to deploy in the middle of the night because people are trying to like run money through your system, right? So this is kind of the reason why we try doing small services, why we're trying to do that today rather than adding features into this existing app that we still use but we're trying to pull stuff out of here. Cool, so I'm using impress.js for this so I'm gonna do a lot of like animations cause it look really cool. So this, I'm gonna start just by loosely defining what service oriented architecture is. I'm not gonna try to talk too much about it because this is not one of those conferences but this is a service. So this is kind of what a service should be. It's just a little sphere that you can move all over the place. It has really defined functionality. It's exactly like you might do object oriented programming. You wanna define a service so it does one thing and does one thing really, really well. Whenever you have another feature you wanna work on, if you're not gonna, if you're trying to decide should it go in this service, should it go in this service, should it go in this service, generally you just make a new service unless you really, really, really think it relates to what's already there. Another nice thing about services is because they're individual pieces, right? You can put one in one data center over here, you can put one in one data center over here or you can move them around, you can put them on the same host, different host, like you can put them anywhere you want because they're isolated. So in this case I have my first service and then I have a second service on a different slide and we see the transition there as well. So the little dots, the little dots are kind of like that interface that we would use to talk between services. At Square we mostly use JSON. That's worked real well for us. Other people might use things like a messaging system where you have one giant queue and you just kind of throw all your messages on there and everybody can look at them and that's really good too, but we are not doing that. Maybe we will someday. Cool. So now we kind of know what services are. The way that we deploy our services is we use Java. There we go. So we use Java. So the reason we've decided to use Java is kind of like our infrastructure is because we don't just do Ruby, we also do Java. We also have some Clojure stuff and we also have some Python stuff and all that stuff runs on the JVM. So we use JRuby when we use Ruby. We use Java when we use Java. If we were to use Python, we can use something like Jython. I don't know, I don't think we actually do that, but I mean we could. So the nice thing about having one infrastructure for all your apps is you just start up a new app container and you can put any of your services on it. It doesn't matter what language it is because they all run on Java. So the only requirement for a new host, if we're provisioning a new host, is that it has the JVM and then we can stick any one of our services on there because they all run on just the JVM and the way that we deploy them is that's the only thing they require is the JVM. Cool. So now that we kind of know what services are and why you might want to use them, we're gonna pretend like today is your first day at Square. Congratulations guys. And you're gonna start working on a new service and this is gonna be a payments map service. Oh, there we go, okay. So what we wanna do is we wanna make a service that shows a map of all of the user's payments. So a good use case for this might be some kind of receipt. Maybe when you log in, you wanna see all your payments. We don't actually have this service so I just kind of made it up. So I don't really have a good use case for this, but the idea is we wanna have a user and we wanna be able to show their payments. So we're gonna go through the steps that we might take to build this if we're at Square today using the tools that we use. So step one is Rails new, right? Okay, so not exactly, but very similar to that. A couple of months ago, some very smart engineers smarter than me went through and they typed Rails new and they set up all this stuff in that app that does nothing. They documented it, they added code quality metrics, they tested it, set up the testing framework. This is all boilerplate stuff that you usually do for a new service, right? They set up the deployment infrastructure for it so we got it all set up so you can do like a cap deploy for example. They set up monitoring and then security for that service and all the other services it's gonna talk to. So this app actually does nothing, but it serves as a template or a starting point for our new service. So if we actually wanna start our service now, we're gonna use JRuby and for the reasons I talked before, we're gonna be deploying on Java. That's why we're using JRuby. So the command we would run is make me a Rails payment map. So make me a Rails is that magic template app that I talked about. This is just a script and I'll show you in a second what this script does. It's actually really simple, but then we just go in there and we get a net and this is a empty Rails app ready to be deployed. We can deploy it right away. So what does the script do? Search and replace. All we do is build a template app. We search for template app constant and we replace it with our new app constant. We get twice, that's it. So you can build this too. Just make a new Rails app, get it deployed and then save that off as a good starting point for all your services. All right, so have you guys heard of Professor Farnsworth at all? He's from Futurama. He's like a very eccentric, crazy person. We're going, so the tool I'm gonna talk about here is called FDoc and it's named after Professor Farnsworth. I didn't name it, but I know who Professor Farnsworth is, so. This is something we made a square to make documenting really easy. So generally if we're gonna make our new payments map API, we wanna start with documentation first and then we'll write our tests and then we'll write our code. So I don't know if documentation driven development is a thing but that's kinda how we try to do it. And the nice thing about FDoc is if you do not document your API in the way that you're using it in your tests, your build is gonna fail. So it kind of just reinforces itself. If you don't have docs, you're not gonna get your stuff deployed because the build isn't green. Okay, cool. So let's look at that payments API I was talking about. This is how we're actually gonna get those payments. So this is something that FDoc would generate for us based on what we did. So looking at this, it's pretty self-explanatory. We have at the top the URL we need to hit to get our payment data. We have an example request which just has a user ID and a limit and then below those parameters are described and we can see the type, an example and whether it's required or not. And then FDoc also shows us an example response. So we can see we're gonna get a success Boolean which is probably gonna be true hopefully in most cases and then we get a array of all the payments. So now we know how this API works. We know we just need to give it a user ID and we're gonna go back to payments. So now we can start writing our own documentation for our new API for our payments map. So now we're gonna go through how to do that. Again, like I said, we're using FDoc. So here's the pretty vanilla payments map controller spec and all we need to do is the magic sauce line right there where we include FDoc spec watcher and that's gonna look at all the requests you make in your controller spec. If it's not documented, then it's gonna break the build. Something you can do with FDoc is you can run your specs with the scaffold environment variable and that's gonna generate documentation for you but it's very empty documentation. It just has a whole bunch of question marks. So you need to fill these things in. For example, describe the payment map service. That's a top describe block there and then we have all the parameters that have been used in our specs and so that we need to describe those as well. In this one spec, we only passed in a user ID so it saved that off as the example there. So we just need to describe it and say whether it's required or not and then it also recorded our response so we can describe what the response looks like. We have the example URL. So in this case, it looks like we're just gonna be returning a path to the image. That's fine. But we do need to describe and say whether that's a required response or not. Makes sense. From this, at the top, you see the file name. The file name describes the request so it's gonna match the path. So this is gonna be available at payment slash show with the method get and this will be compiled into that HTML that we just looked at. FDoc will do that as well. Do the compilation. So this is available online as well. GitHub. So I'm gonna actually get a drink of water so memorize the URL. Cool. So now we've got our API documented. And like I said, we're gonna do documentation driven development, right? So now it's time to write our tests. And our test is very, very motivated by this book. I'm sure many of you have read this book but if you haven't, you should check it out. Something it really talks about is unit testing and testing your things in isolation. So with services, that's especially important because on my laptop, for example, I can't even run some of the services that we have. So it would be impossible for me to test using those actual services. So a really good way to do like your acceptance test and not actually have to talk to all of your other services is to make fake versions of them. So this is an example of a fake version of that payment service we looked at. It's super simple. We see we have a payment service module and a fake server, that's Sinatra. And then we just describe that endpoint. And we can even stick in the example response that we see in the documentation. But this will allow us to use a cool tool called Forman to boot up all of our fake services on our development machine so that we can write our tests and do actual coding and try it out. So let me show you how to set this up with Forman. Here's the rackup file, which this is gonna be the same for every single service. You just require your fake server and mount it to the root. And then you can take this rackup file and include it in your proc file. So lots of files here. Proc file is what Forman looks at to start your services. So Forman is gonna start two services for our new map application. It's gonna start Rails server, which is obvious. We want our map server to be running. But it's also gonna start the fake version of the payment server. And it's gonna start that on port 3001. You also notice on that top line there with the server, we're starting out with this weird isolate environment variable. So the reason that's there is so that we can allow ourselves to actually use real services if we want to. So inside of our Rails app, we'll have probably a YAML, and we do use a YAML of all of our services that we depend on and their URLs. So the payment service in production will be at this URL, the payment service in staging will be at this different URL, and the payment service in development will be at this URL, but if it's set to isolate, then we actually wanna look at local host. So every time we start Forman, we'll use the fake server. If we don't use Forman, then we'll be talking to the real server, which will live wherever we define on this YAML, which will probably be on our laptop. And that's how you start Forman. This is actually a lot like when you start Rails server and it starts showing the log files. This will show all of the logs for all of the services you've defined in your proc file, and they'll all be prefixed with the name of the service. So if you have like five different services, you'll see five different colors of logs. It's actually really, you should try it out. It's really cool. So this is where you can get Forman. This is not something we've made at Squared, but we use it a lot. All right, so next up, we've gone to testing. So now we're gonna actually start writing our code. And something that's really important that we found when making all these different services is keeping some kind of consistent code quality among them, because you're probably gonna have like two people over here working on this service, two people over here working on this service, and they're gonna have vast different opinions on, you know, little things like line length and empty white space and all that kind of stuff, right? So a good way to kind of just take that out of the equation is to define that in advance and use a tool like cane. This is cane. This will break your build if you violate style guidelines that you've set. So I'm actually really, really into the less than 80 character thing. So this is like a godsend, because if somebody does a pull request and it's like 90 characters long, I don't have to pass it aggressively say, well, maybe we should shorten the line. All I gotta say is the build broke, right? So it's way easier for me. Like little things like that are huge. Another good example is if you're working on Ruby 1.9 project, maybe you don't wanna use hash rocket syntax, right? You wanna use the new fancy Ruby 1.9 syntax. There's not really a good way to enforce that, but you can build a rule in cane to enforce that for you. So if somebody does a pull request with using the hash rockets, then the build's gonna break and they'll have to go back and change that. Cain is also available and that's where you can get it. I'm also gonna try to make these slides available later if you don't wanna write these down, but they're pretty self-explanatory. Okay, so now we've got, I didn't really put the code for a payments map in here because that's not really interesting. So we kinda just coded and now we're going to deploy our new service. There's a bunch of different ways you can do this and this probably isn't the best way, but this works really well. So we have Java that we're deploying to, so we need to use a Java web server, right? So the one we've decided to use is Jetty because it's really fast and it's really small and you can actually kind of embed it into your Rails app. You don't need to assume that it's already on the host you're deploying to. So again, it kinda serves that purpose of being able to provision a new host right away and then put your service on it and not really worry about what dependencies are there. You can kinda just stick Jetty inside of your Rails app. A really good tool for this and the one we use to use Jetty is called Jetpack. So Jetpack actually does all this for you. You just need to create a config jetpack.yaml and in here you can define all the things for your app. So here are some examples where we have, this is the user on the machine you're deploying to, the user that's gonna need permissions to start this Jetty web server. This is where that's gonna live, right? So we want it to live in user local apps payments map wherever we deploy this thing to and then we can also specify things like the ports, HTTP port, HTTPS port, and then our Ruby version. So this is obviously an old YAML we're using Ruby version 1.8, but you can use 1.9 and I think with JRuby 1.7 that just came out, 1.9 is like the default, so that's pretty cool. So with, again, so this is Jetpack, how this works, edit the config file and then, that's pretty cool, there we go, run Jetpack in the current directory, it'll package it all up and so what this Jetpack does is it downloads the JRuby runtimes from the JRuby site or wherever you tell it to, by default, it'll be the JRuby site, it'll download the Jetty runtimes, it'll bundle your gems in JRuby and then vendor them and it puts all that in the project directory and then when you do your cap deploy, all that needs to do is bundle up your, tar up your directory and SSH it up because everything else is already set up. Jetty already knows how to start your server, your ports are already ready to go, all your gems, JRuby itself, that's all in the package, so you don't need to even assume JRuby is on your host, all you need is Java, you run Jetpack in the current directory and you're good to go. Jetpack is also available at square slash Jetpack and there's a pretty good read me to get you started because I probably totally did not cover something on here. Nice, so we're at the point now where we're deployed and there was a lot of voodoo in that deployment so feel free to ask me questions later, but it's up there, so now we need to monitor it, right? So I'm sure a lot of, how many people are using, this is the air brake logo, how many people use air brake now, like a lot, how many people use hop toad? All right they're the same thing, so okay. So we actually use hop toad, but I'm saying we use air brake just because it's the new hotness, but it's actually the same thing. So what air brake does, you configure your Rails application with, you just stick an initializer in there with your API key and maybe some other info, but that's really all it is so I don't really need a code sample on here. And then all your exceptions are sent to air brake and this will email you if there's an exception, it'll kind of track exceptions over time so you can find things that are kind of becoming more problematic than they were. It'll also let you comment on exceptions so it might look typically something like here's a timeout error, these happen all the time. There you are. So this is really useful for us for like interrupt driven triaging if you're, it'll tell you if something's breaking. It's not really good for a monitoring tool I don't think, but it's good for interrupt driven stuff to warn you if something's about to happen. Another really awesome tool, this has nothing to do, this works completely different than air brake, this is more of a monitoring tool. Cubism.js, you can make really, really cool dashboards with this. How this would work is you take all your critical metrics like foo bar and foo plus bar and foo minus bar, but probably more stuff like in squares case we like to monitor our payment APIs and make sure that payments are actually going through. If this graph starts looking really weird, like we're getting a bunch of 500s on our API host or we're running out of memory or our CPU's bottle rocketing, that was really cool. Or our CPU is like going through the roof. This is a really good tool for figuring out that we need to start looking at something. You'll probably start seeing activity on here before you get exceptions, we've noticed. Cubism.js can take information from graphite, it can take information from cube, which is similar to graphite but not, but I haven't actually used cube, but we use it a little bit at square, I haven't used it. Very useful tool, this is available on GitHub as well. Okay, so now we are at the point where we are deployed and we're monitoring our app. Splunk is the tool that we're gonna use to figure out what's breaking, right? So one of the huge problems with having services all over the place is you never know where to look for your log files. So we're getting an exception, even if you think that, so say for example we have five API hosts that are taking requests from handheld devices, right? And we know there's an exception, but we have no idea which host it's actually on. Splunk is a really good tool for that because you can search through all your log files in one place. So all your services send their logs to Splunk and then you can search for say a payment ID and you'll be able to see that payment go through all of your services and you won't need to SSH into each box individually and look at the log files. You can also generate really cool graphs like this that kind of break down whatever you want. So this would be different types of queries stacked on top of each other across time. So like successful versus failed payments, that kind of thing. So here's an example of a query that I use when I'm on call because what this will show, for all of our API hosts, any process that involves payments, tell me which ones are 500ing. So, and then at the very end, we got count by action. So Splunk is able to parse your logs and detect the controller and the action, any other thing, you can kind of teach it what you want it to know. It knows Rails by default, but if you have your own custom log file, you're gonna need to probably teach it. But you can do intelligent searches like this. So, excuse me, so this would show you a graph of all the actions that are failing and so you can kind of gear your attention towards those specific actions in your controllers that are causing the most trouble. Okay, cool, so this is the last thing I'm gonna talk about and this is security between services. And I probably have more details about this than anything else because it's more interesting, in my opinion, as far as looking at code. So, lots of people with internal services don't really care about security, right? Because it's behind your firewall. Who cares? Nobody's gonna be able to get to your services anyway. We trust our developers. We don't think that they're gonna write malicious services. But when you're working with money, you have like certain requirements that you might as well just secure everything just to be safe and it's actually really easy to do. So, why not do it? So, for security, we use SSL certificates and we just use, we do what's called mutual authentication. So, I'm sure everybody in this room, well, most of you in the room probably, are familiar with setting up an SSL connection for if you're making a secured website like bank login or whatever. You wanna make sure that that session's secure so you can't see the traffic. But usually that's a one-way thing. You're not actually verifying the user's browser, for example, making sure that they say who they say they are. You are with the password, but you're not doing it with SSL. With services, we can do that because we have complete control over the services. So, every time we generate a new service, we also generate an SSL certificate for that service. And then on the SSL certificate, there's a little key called organizational unit, which you can put whatever you want in there. So, we put the name of the service in there. So, basically, we have, for our payments map service to go back to that example, we would create an SSL certificate for that we'd sign it with our square private key just so that we can verify that that is a service we made. And then we would set the organizational unit of that certificate to payments map. So now on all of our services, when a connection comes in, we can look at that organizational unit, say, okay, this is the payments map. Is this payments map allowed to access this API? If it isn't, then we'll return up like a 403 forbidden. If it is allowed, then we'll just let the request go through. So we can do like a per service kind of security there. So I'm just gonna go through, like in a Rails app, how you would do that, like how you would get the organizational unit out of that certificate, and then how you can check on that, pivot on that. And I'm gonna take a drink. So where you would start is making some kind of access control list. So we just use a simple JSON file for that. So in our payments app, this is what the ACL might look like. We would have our resource, which is slash payments. We'd have the method, get, and then we'd have a list of all the clients that are allowed to talk to this resource. So in this case, just payments map. So as long as we have this in here, we know that our payments map is going to be able to talk to our payments service. So now we'll go through kind of how to actually use this ACL in the Rails app, because right now we're not using it for anything. So the first step is to have some kind of, we're just gonna use a API controller for this example. So all your API controllers can extend this API controller. I don't know if you're into inheritance, but for this example, we're gonna use inheritance. You can go and define a before filter on here. And all this before filter has to do is say, if this request is authorized, let it go through. Otherwise, return a 403 and an access denied. So just block the request. So now anything that extends your API controller is gonna be protected by that ACL. So there's a lot of magic happening in request, and less request authorized. So we'll go look at that and see what that does. So in request authorized, you see we're gonna have the, let's start by looking at that bottom organizational unit. I was trying to think of the best way to explain this. It's kind of hard to go without seeing all the code and we don't have enough room on the screen. So let's look at organizational unit. This is how you actually get the OU, as it's called, out of the SSL certificate. When the request comes in, you grab it out of the environment. You can pass in the certificate as SSL client cert. You then create an open SSL certificate using that raw cert that you got out of the request. And then that's gonna let you easily access those things. So then you can search through your certificate for a name that matches OU, which is gonna be the organizational unit, and return that. So this will just give us that payments map string that we've saved in the certificate. And then if we look at that top method up there, we'll take that result, in this case payments map, and we'll take the request method, which in this case is gonna be get, and then we'll also take the path info, which is gonna be slash payments. So we take those three things, right? The organizational unit, which is payments map, slash payments, which is the path, and the request method, which is get. Those are exactly what we define in our ACL, right? So then we take those three things, and we look at the ACL for something that matches that. So we find an entry that matches those two things, and see if the current OU is in the allowed list. So while you look, I'm gonna drink more water. Cool. Like I said, that would be the end. So we now have security. We talked about getting started, setting up all your stuff, including documentation, testing, how to get it deployed, and then taking that off and putting it into a template app. And then we talked about how to monitor the app once it's actually up there, and then do security between the different services. So I can probably take, like, I mean, I have lots of time, but I can take maybe just a couple of questions, more because I'm scared. Yeah. So the question was, what do your clients look like? So like the iOS and Android clients. Oh, I'm sorry, then can you repeat the question? Got it, okay. So the question, not like the handheld clients, the question is, how are your consumers of these services kind of getting at the data? Are they using JSON, are you doing, or is there some kind of defined client that you're using? Is that right? Okay, so generally we just treat any client service like another service. So we create an SSL certificate for it, and we go through the same APIs, and most of our APIs are JSON APIs. So like a client you might make, we would just treat that as a new service, and give it access in the ACL, and talk to it using the same JSON API that we would use for service to service. Does that make sense? I wanna talk to you later because I'm sorry your question's over my head, but I will talk to you afterwards. Yes, sure, yeah, so the question is, have you had any luck to paraphrase, chopping up your giant Rails app into smaller services? And so like I said, like the screenshot I showed earlier of the lines of code and stuff, like that is how it is today, but it did used to be bigger, and as we work on more things we are trying to chop things out. So a really good example is the logged in portion of our website. That used to be a part of that main Rails app, you know, it's just another controller in there. You log in, you go to this controller, you see this view, you can manage your stuff, but we decided we wanted to redesign that portion of the app. So when we did that we just made a new service. So usually what we're doing is anytime we're gonna redesign something or just do some kind of large refactoring or really work on any part of the app, do a lot of work, then we'll probably just do that work in a new service and then use our, I guess we don't need this anymore, use our large app that still exists to redirect to that new service and then eventually just chop it off. So does that answer your question? Usually only when we're working on it, we haven't actually gone back to mess with stuff that we're not currently trying to change. Yeah, that's a good question. So the question was, do you have just really, really small services or do you have, that do really specific things or do you have like a bunch of larger services? Is that what you're, is that what you're, like the dependency between the services and oh, I see what you're saying. Yeah, like some deep call stack, right? That you could go like service A, call on service B, call on service C, call on, yeah, so no, we don't have that right now. Because that's usually like, that would almost be like a smell, right? Because you have, if you have one service that needs to call, it's kind of like if you have, it's the same way like if you're writing, doing object-oriented programming, if you have one object that's calling into like the 18th attribute of this other object, then you should probably just wrap that in something else. So that's kind of what we're doing. I mean, that's what we try to do. The deepest call stack I could think of is maybe two. Yeah. Okay, one more question. Yes, on the front left. Okay, good. Okay, so the question is, I talked about doing service isolation using something like form and to create a fake version of the service. How is that different from making like a mock? Like are you talking about like an R-spec or something? Oh, okay. Sure, yeah. So our approach is like, it's very simple. Like you described, we're always gonna get the same response back. In some cases, we actually have some logic in the fake service. So if it's supposed to do something like approve a payment or something, then we'll actually take the payment that comes in. We might actually, in our fake service, we might actually be doing a lot of the logic that's in the real service. We'll read it up, switch something, and then return it back. We won't usually persist any of that data, but there might be some logic in there. Generally when you're writing a fake version of your service, you test it with the real service and we put it in a gem. So with the payment service, for example, all of our tests that test our payments API will also test our fake endpoints and then that'll be in a gem. So I wouldn't actually have to write that fake service. I would just include the payments gem and then that fake version of that service is available in my tests. That way I don't need to worry about, typically if I'm talking to like 10 APIs, I don't wanna have to write the whole payment service over again every single time I make a new app. So you would just kinda test it the same way. But yeah, often the data is like static. We return the same data all the time for the fake service. And then for full, your other question was, do you do acceptance testing for all your services, like all together? No, we don't, so. Other than doing acceptance testing against the fake versions of them. We test them by themselves completely and then against fake versions and that's pretty much it. We just assume that everything works and then the conversation between doesn't actually get tested. Okay, I think that's it. Please find me if you have more questions.