 is your hand if you like working with legacy code? All right, well, you guys are missing out. Legacy code is awesome. It tells us everything not to do. And if it wasn't for that, we'd go and repeat some of those mistakes. So I'm Jeffrey Mathias. I'm an engineer with Sengrid, and I'm going to talk to you about future-proofing your third party integrations. This is a collection of practices. I'm not going to claim their best practices, but they've been working really well for us, and that's why I put them together to share. So my team's job is, Tim, just a minute ago mentioned that working from a monorails app to splitting that out, well, we have the same kind of situation. Sengrid was a pretty fast-growing company. We have a monolithic PHP app, and that is awesome, because it gives me a job. Our team, one of our core responsibilities is extracting our third-party services from being spaghetti dependencies into a manageable pattern so that we can repeat this. And we use Rails to do it, but the stuff I'll be talking today kind of applies to any Ruby application. It's things you can use across the board. A lot of the patterns that I'm going to be showing today or a lot of the ideas came from our work with Pivotal Labs about nine months a year ago, and then we've built on top of that. So you're going to see a blending of some different ideas here. So when we're looking at a third-party integration, we have a few main goals that we want to do to manage that interaction. The first one is keep third-party logic out of our code. Probably don't have to make a case to you guys about that. The whole idea behind that, of course, is that as soon as somebody else's implementation of something starts making it into the way that you are writing your code or even thinking about things, you start becoming dependent on how they have implemented it. It's scary. The next part is making our third-party behavior predictable. And we do that through test driving our code. And then the last bit is making our third-party interactions replaceable. And being an engineer at Sengrid, I'm perfectly aware that I just told you to make my company's product replaceable. And I fully believe it. Any third-party API you have should be. So let's take a look back. I actually have a degree in art. I can prove it. I have an amazing drawing here. My professors would be so proud. So this is a general look of what the architecture looks like. The Music Box app, obviously Sengrid doesn't have anything called the Music Box app inside, nor do we have a My Little Tea Pot third-party service. The biggest, most important thing here is that you're going to see there's this adapter sitting in the middle. And then we have the internet or something. And the reason that I said or something is that this doesn't just apply. When I say third-party integrations or your third-party services, I'm not just talking about outside of your company. Any time you're engaging, since we're all moving to service-oriented architecture, we're breaking things up, you need to look at the behaviors and how dependent you are on the logic or how wrapped up and how wrapped up are spaghetti, your actual dependencies are with your own internal applications as well. So for example, right now, we have a database abstraction that at Sengrid that people like or people don't like. But the whole point is, is we're writing a new app. We're following this pattern for our internal service too, because someday somebody may agree with us and not like that service enough to replace it. So we want to be in a position that we're flexible. So this Tea Pot adapter here, this adapter that's designed for our third-party service, the most important thing to understand about that is that it is just a Ruby gem. As in, we run Bundle Gem Tea Pot adapter. It spins up a bootstraps of gem for us. And then when you go to do that, you don't need to push it anywhere. You can keep it in your repository in the directory that you have in it, and you just need to know the path for your gem file. Hopefully, most of you have already been using this as a way to organize your code for various things. I could not recommend it more here. Part of the reason for that is that it gives us a very obvious separation line for our concerns. It means that the SendGrid logic stays in the application, and the third-party specific logic stays in the adapter. And then all you have to do is make sure and focus on the messages in between. So that isn't to say that you should ignore it if a third-party provides an SDK or a gem for you to utilize their service. They've put some hard work into it, and you should definitely leverage that. I'm not telling you to reinvent the wheel when I say roll your own gem, but what I am saying is that the messages going into this adapter should be your language. The way that you implement that should absolutely take inside of that a gem, should absolutely you take advantage of any existing code. So your gem, your adapter gem, can use the SDK or the gem from that third-party or anybody else who's developed one for that third-party. So let's go back and look at this thing again. When I talk about this gem, it's actually a really basic architecture. It's just a big bunch of service objects. And so your messages are going into those, they're calling out to the third-party and then returning the objects. And so one of the most important things when we are setting up this kind of architecture is to make sure that our main application really doesn't know what adapter it's using. It knows how to talk to the adapter, but it doesn't know which one. So the only place that you're actually gonna see the name of that adapter is gonna be in some sort of initializer. This reflects our current version of Rails initializer. But the whole point being is that these bottom lines where you see us handing over the service as that service name should be the only place in your entire main application that you see references to that thing. And then at that point, the way we have it, we have a configuration class and so then we actually just refer to our services by configuration instance melody service. And instance for those of you who aren't familiar with it shows that we're actually using a singleton here, which this is one of the few places I definitely would recommend it. So again, I wanna emphasize that this is the only place in the application that you wanna see teapot show up in your music box app code. So how do you move towards this or how do you get this started going? The first thing is to clearly define your internal vocabulary. This couldn't have followed a better talk than Mike talking about how important understanding your domain and what your concerns are. This is the exact same thing. We need to know how we look at how our business is structured and not think about that third party service. And that means actually getting people to agree on terminology. It's amazing when it comes down, it seems like, oh yeah, of course, we know all these things. When it comes down to talking to your business owners or your product owners, your stakeholders, getting them to all actually say, oh yeah, when I say this, this is what I mean and realize that they're on the same page or not, they're not. Almost every time there is gonna be some variances. So you get them to agree on all that and that sets the vocabulary if you're messaging between your application and the adapter. And even if you never plan to replace a third party service, first off, you should always plan to replace a third party service. But even if you never plan to replace it, you should still have your own language and understand, make sure that the way that you talk about things is the way that they're meant, is the way, make sure that the definition is what's relevant to your company has nothing to do with another thing that you're using. So for an example, from our world, when we talk about a package from a billing perspective, our third party provider actually is talking about a product rate plan. When we talk about an add-on, our third party provider is actually talking about a product rate plan. And then it's a little less confusing. When we talk about a coupon, our third party provider is actually talking about a product rate plan charge, which is good because it's very obvious difference between there and when you're having conversations with people, it's very easy to talk about. So no, instead we are able to refer to packages, add-ons, and coupons when we're talking to the people that we work with in our teams and only the people who have to deal with that third party service, the developers on our team really should have to know the details on that next bit. So, oh, and so to get an idea, we actually then have a package service that handles the product rate plan. We have an add-on service that handles product rate plans as well, but in a slightly different context. That doesn't mean we can't reuse code. And then we have a coupon service that handles the product rate plan charge. So I've been talking about separating these concerns and we actually have a couple things that we use to force this separation. One of them is data transfer objects, or DTOs. What DTOs allow us to do, they serve as a codified contract for the data going into the service and the way that it comes out. So it standardizes what that formatting looks like. The way, and it's basically just a Rails object. I'll get into that in just a minute. The second part we have are abstract classes that serve as documentation for the interface. So let's first take a look at what a DTO can look like. It really is a fairly basic object. Most of the data tends to be passed in through Initialize. And then one of the most important things about our DTOs is that we handle our validation for the data there. So in our case, most of what we're doing is coming in from a route, a Rails route, and the controller is what's kicking this off. It puts the data in Initialize as the object and then looks for errors to see if everything's good to go before it makes the next call. If not, then it grabs the errors, they got off of that and returns them. But the whole point is, is that because of this, the same thing can be used. It doesn't have to be used specifically in a controller. Or more importantly, as we're versioning things, we can still maintain the validation lives in a single place instead of us having to worry about making sure that validation moves between, like, is maintained or copied over to whatever our next version of that endpoint is. One of the other things that's really nice about it is when we first started implementing this, our services were getting a whole lot of data. And one of the things we were passing along was an options hash. And options hashes are great when you are, you have a really tiny amount of things that you're trying to pass around. Actually, that's not true. I just don't like options hashes. And the reason why is because for me to know what's supposed to be in an options hash, I have to go look to see where my method got called and actually look at it there, and then it might actually be different between places. So I know that Ruby 2.1 has given us keyword arguments and I promise you, as soon as we bump this from 193 to 2.1, we will be implementing that, but we will still be doing that with our DTOs, not trying to skip around them. And one of the biggest reasons is because we really like having this portable validation. The other thing too is to notice is that it's got results and errors. That is an opportunity for once you've passed the object into the service, for the service to do its thing, it actually uses this object to hand the data back. The response from whatever that service happened, whatever happened during that service, that response comes back through the DTO. That object is just handed back. So next we'll take a look at what I mean when I say we have abstract services. We actually have a service that's entire purpose is just to raise errors if you call its methods. What it does is it defines for, in this example, the Lyric service, it defines what those incoming messages are and how many parameters it takes. We even left ourselves a note that we expected to take a Lyric's DTO. Some of our older versions before we were using DTOs will actually just have multiple parameters until they get refactored and that's what they look like. So when you go to implement your actual service, you inherit from that and then you know which methods you're supposed to replace so which you need to implement. So why are we doing this? Why are we giving ourselves more overhead? We've written empty classes and the reason for this is that the DTOs combined with the abstract classes, the abstract services, give us documentation in our code on how to replace that service. So with these two things, as long as they're there, I actually can come along, scrap the entire implementation I have for another third party as long as it's a comparable service, I can build an adapter using just these things to replace that service and I never have to look at the way that it was implemented in the previous adapter with the previous service because that code is all going to be specific just to that third party. The problem, of course, with this is that most of the developers I know are human and it means that they forget things. Meaning that when we go and we're working on our teapot adapter Lyric service, we're not gonna remember to go, deal with making sure that the interface is all matched up. So I've addressed that and I'll get to that in a few minutes but just so you know, I am aware that's a thing. But before that, let's jump back and kind of look at a slight variation. So this is what I showed us before and what we've ended up doing is we've wrapped the DTOs and the abstract classes together into another gem and we call that an adapter interface gem and so for every adapter, it has an adapter interface gem that it refers to and so it actually sits in there and it finds that very surface layer of the adapter class. So when the Musicbox app talks to the teapot adapter, it's actually using it in the language defined by the adapter interface that implements language that belongs in the Sengrid domain, our domain, our apps domain and not the domain of the teapot API or whatever API that we're wrapping. So the catch is that both the main application and the teapot adapter will include this gem. The main application will use it for the DTOs and the adapter will use it for both the abstract classes and for the DTOs and then as a side note, we test drive our DTOs but since our abstract classes are there as documentation, we don't test those and I mentioned I would get back to that whole thing about being human and forgetting stuff. By the way, nobody laughed when I said most of the developers I know are human. You guys really disappointed me. Okay. So I wrote a gem that actually through testing we'll go through and look at the class, the service class, look at its parent class and make sure that the interfaces match and it's simple enough that we've included it is part of our generator for our adapters and so we don't even have to think about it now. If there's something missing, we can just go update the documentation and it's done. I don't know, raise your hands if you like writing documentation. Okay, there's a few of you that are nuts. For the rest of us, this is the way that we figured out to have to maintain the documentation on this the very least way. So I know that the concept of enforcing an interface is about as anti-Ruby as you can get. I mean it just goes against the core DNA of Ruby and hopefully having named the gem uptyped will give you guys some awareness that I feel the same way. Sounds like uptight. It is available as of this week on RubyGems. Even better as I invented a word apparently if you Google it, it seems to be the first result. So it is a super early version. It looks at anything that is not, if you implement it the hardcore way, it just looks at anything through the testing that doesn't inherit from object and makes sure it matches. We're actually looking at moving it so that we will do things, it'll do it on everything unless you allow you to whitelist namespacing instead. So private for example. For our purposes of the adapters, this makes sense because we don't wanna be writing code that's not supposed to have it, unless it's private that shouldn't have that interface. By all means grab it, tell me what's wrong with it and even better submit pull requests. So there's the general architecture of how we do it. The last component is how do we test these things? I'll get to the complexity of dealing with third part, testing third party integrations a second from the standpoint that it's on the other side of the internet, right? That's obnoxious, it definitely causes some problems and I'll jump to that in just a few minutes. But first let's just talk about testing. There are four parts to test, right? There's your setup, your kickoff and your expectations depending on your framework, the order of those may be different and then your tear down. And the tear down is something that we consistently see people forgetting about. And in fact the application that we're deprecating right now really loves to dump every so often when CI runs about 400 new accounts into our sandbox for one of our providers. Which means that as we're sitting there just working it's the same place where we play as developers, suddenly you'll have to go searching for the data you are working with instead of being able to just pull it up. And that's never fun. So tear down is a very important part of this. So a couple basic rules is remember to never assume that your data exists in your third party. So when you're testing just like any other time if you're testing you need to make sure you put your data there just because it's a third party service, doesn't mean you have the excuse of like oh I don't wanna have to go stage this stuff every time. You really do need to do that to be consistent, be able to be predictable on your testing. Use a sandbox if one's available and if one is not you need to either talk to your third party provider about getting one to be available or maybe look at other options. Unless it's a situation where having test data somehow mixed with your production data doesn't hurt but I don't really know many situations where that would make sense. So this is a pattern that we actually got when we were working with Pivotal Labs. We call it a test environment. And what it does is it just creates an object that has all the tear down methods sitting inside of it. But when you create that and I'll show an example of how to use it in a few minutes but when you go through and you create it you'll see that we're actually hanging onto for example we call create user on this thing. We're hanging onto that user ID. And so when you initialize it you end up giving it a block. Once that block is done executing the insurer will go back through and clean up the data and it just uses every time you create something you hang onto that ID the clean method will then go through those various kinds of IDs and remove that data again from your third party. My best advice on this is as you continue to develop your app switch calls. You see me using the teapot gem here user create start dogfooding your own services because those are actually test driven at that point. So as your application develops if you're going back into code and either adding something or you just have you know it's a decent chance to refactor make sure that you do but go back and refactor switch it to use your own services instead of having kind of some crazy code going on there but that will initially happen because you can't stage the data with services that don't exist. And so with that the next oh I was trying to figure out why this it wasn't as big on mine so I put that there so I absolutely wouldn't forget cause every time I practice this talk I did integration specs you can actually use the same environment with your integration specs but the problem is of course is that your app theoretically has to know about this thing in order to use it but I told you that it shouldn't show up anywhere else the app shouldn't know what service it's talking to. So what we've done is we've actually created a pretty thin layer around a thin wrapper that lives in the adapter gem itself that makes sure that no matter what we're using SendGrid language when we're making these calls to the setup environment and then we're handing that over during initialization to our application. So it knows that it has an integration level testing environment it knows that it can excuse me make certain calls but it still doesn't know what service it's actually dealing with. So back to that thing I was talking about the internet being a pesky thing it's not as fast as we all would like so you inevitably are gonna have to deal with stubbing your third party services. There are two options that we've come across and that's making a fake service and the other one is using VCR which is a handy gem that captures your interactions between your application and whatever's happening through the internet basically records all that stuff as a YAML file and then can replay that. So as long as you keep sending the same responses or the same requests it'll give you the same responses as it did the first time. Well after a long discussion we've seen people very successfully go in both directions. We agreed we really didn't wanna have to be responsible for maintaining an interface for something. Yeah this is from the guy who just told you that I had brought a gem to maintain interfaces but VCR definitely looked like it was gonna be the best match for the way that we wanted to develop and so I'll show you what that looks like and then I'll talk about some of the rules that we've developed for that. So combining our VCR and our test environment you can see that we're wrapping everything at the top in a VCR cassette. And then in VCR if it doesn't have a cassette and the cassette's just the name for the YAML file if it doesn't have a cassette it will go and record it and otherwise it will just assume that you're trying to hit the cassette and it'll work against that instead. So what we've done here is you can see the top couple lines inside the block are creating the thing. We set up our you know so and then we've we've got some setup on our services the line that starts with response CTO is our kickoff and then we have an expectation. By the time you hit end from the block sitting inside of a teapot environment new you've hit that ensure block which means it's gone and cleaned up the code so you've run your test successfully and then you've gone back and cleaned it up. And again I wanna emphasize that we wrap the entire setup, test and tear down inside of the cassette. So that means that yeah if you're running the cassette later and hitting that your code is running through those steps but it is the only way to be consistent and predictable and not leave cruft behind in your in wherever your hopefully your sandbox wherever you're testing. So I mentioned a couple rules so always never forget that cassettes must be deletable. You have to be the state of your cassette and this is why it's important to wrap the whole thing like all of the setup in there is that if you cannot delete your cassettes you've now created fragile tests. You have to be able to go through, delete the thing and just be able to re-record the whole stuff. Yeah, your tests are gonna run a little bit slower, right? But we don't commit code on certain, most of our projects, the ones where we, after we learned to make sure our cassettes are deletable you make sure that it will run with that. Even more, even better is that make sure, as I said, you set up everything you need inside of the cassette and then if you're using CI, kill your cassettes on a schedule. So VCR has the ability to set a time to live or an expiration date on a cassette. You can actually do that globally. So what we've done is by environment it knows if it's sitting in the CI server that no cassette lasts longer than a day. And one of the reasons is that when you go and if you delete your cassette and all of a sudden your specs are failing, turns out not every third provider is super awesome about maintaining their interface or making changes. Especially things that you may have unintentionally been using that have been undocumented as part of that versioning. They're like, oh, that wasn't really technically part of our version. We can remove that and it won't hurt anybody. So this is your canary in the coal mine. This is typically the first place that you're gonna see things break, especially if you're in your development cycle and you're working against existing cassettes, then you may, so that way it enforces it without having to make it so that you're always constantly deleting your cassettes and rerunning your tests because the internet will slow things down. So to recap, your app should never know what service it's talking to with exception of that one time when you hand it into the initializer. But what I mean by that is your code from your app, in your application itself should be specific to your domain. The language there should be your application specific and have nothing to do with your third party services. So that starts by defining your vocabulary and then spin up the adapter interface gem. And again, we have generators that we've built custom for what we're doing with these things. And so that interface gem includes a DTO and the abstract services. Create your adapter gem and add up types because I want downloads. Create your test environment and use VCR. And delete your cassettes a lot. So that is the way that we have set ourselves up to be able to switch to other third parties as quickly as possible. Does it mean that it's gonna be easy or a walk in the park? No, but what it does is it gives us a predictable amount of work. We can actually look at what's there, get an idea of what it's gonna take to implement it. No, we have documentation of how much we have to implement when we're trying to estimate the project and turn around and talk to our product owner. That was future-proofing your third-party integrations. I'm Jeffrey Mathias. You can find me on Twitter and Instagram and I guess L.O. now at idle hands. And if anybody has any questions, we have a few minutes. Yeah, so the question is can I talk about the pros and cons of using VCR versus stubbing out a web object using something like a struct? And that's coming from somebody who I know really likes structs. So yeah, I think the biggest thing comes down to is that it means that if there are any changes on that third party, we have the ability to react faster, detect it as quickly as possible and don't have to worry about making sure that our data behaves the same way. And that really was the deciding factor for us. As far as the cons go of using VCR, I mean, you're buying into somebody else's library and somebody else's implementation. So sometimes when you're debugging, you have to end up kind of looking into it. And then the errors from VCR are a little questionable sometimes. You just know that whatever you did didn't make sense in the cassette, but you don't necessarily know what you did that wasn't right as far as the interaction across that third-party service. That's a good question. So the question was whether or not we have found a way to, if we've got a test failure, identify whether or not it has something to do with the deletion of the cassette. And so the basic response is deleting a cassette should theoretically never cause an ultimate problem unless you don't have the internet available to you. But to tie in with that, it's interesting, one of the things we've seen, and this actually can come back to the question you asked as well, is that sometimes you have a spec that's failing and it turns out that if you just delete your cassette and re-record it, it is not a problem. That something about the way that the cassette is looking for the data to come out causes a issue, but it turns out the third-party service itself is totally cool. Nothing has changed in terms of the way that you're hitting it, or nothing about the way that you've changed the way you're hitting it actually is causing a problem with the response or its interaction. And so, no, to come in short, no, we have not automated too much of that. We haven't come across too many problems yet. And so, admittedly, our first problem, if we start seeing issues is we just go to delete our cassettes and re-record. And a lot of times that means that we change the way an ID is going out and it's not in our test or something and it's not getting matched. We didn't properly deal, like, handle an our cassette in VCR in the way that we were randomizing some of our data for our test stuff. Okay, so the question is how do we deal with the cassettes on the CI server? And by the way, this is the last question. If anybody else wants to talk to me, well, you'll have to grab me either after this or at lunch. So, once the whole point of even using VCR in the CI environment, ultimately, it really, like, it should just be hitting the third-party service. But if we're gonna be running the same specs and we're sharing resources, we don't want it to take forever. So we're not worried about making sure that those cassettes necessarily end up back with us. One ends up happening is if you end up with a failure over there, then that gives you an opportunity to just sit there in your dev environment and figure out instead what the problem is. So, great. Thanks, guys.