 Good morning. So my name is Wesley, as he said. I go by GMS online. So that's what you'll find me on Twitter and GitHub and that sort of thing. And I call myself the hacker laureate of Heroku. I don't know if anybody else actually calls me that, but it's my story and I'm sticking to it. So to give you a little background, some of you maybe have never heard of this library. It's a library in Ruby that provides client interactions with a bunch of different cloud services. So this is kind of one of my biggest introductions to the world of APIs because it turns out if you want to interact with, you know, I think it's something like 20 or 30 different cloud services, you interact with a lot of different APIs and a lot of different ideas of what a good API might or might not be and, you know, kind of everything in between. And so this was actually the second or third thing that I ended up working on right in a row where I was interacting, like writing a lot of clients, interacting with a lot of APIs. So coming out of that, I was in the position where I was able to get a job with Heroku. And so when I first started actually kind of continuing with that theme was working on the command line tool, what we now call the tool belt, which again, yet another API clients, which gave me the opportunity to actually become pretty intimately familiar with the API that we had at Heroku at the time. Before I then transitioned onto the actual API team and undertook to perhaps fix what I perceived as many flaws in the API that was there when I joined. So it didn't take long to sort of realize that consuming and producing were quite distinct, even though people were like, oh, well you know all kinds of stuff about APIs, you use them all the time. And it's like, yeah, but using them and actually making them is a pretty different thing. Consuming I think is a lot simpler in most cases, although it can sometimes be really frustrating because some APIs are not awesome. So I think maybe the saving grace and some capacity is I sort of have this philosophy that in a lot of ways it's not really what it looks like that's gonna matter with the design of an API in particular, it's how it feels to actually interact with it, like what it is like for a user. And so having interacted with a lot of them, I at least have my gut reactions to fall back on. So when we're talking about a new API design or something, I can often say that doesn't feel right to me, it gives me the heebie-jeebies, I'm concerned about that, even though it's taken me quite a while after that to necessarily be able to work backwards and say, well, I don't think that this is necessarily a good idea because of X, Y, and Z, like turning that gut feeling into concrete reasons was difficult. So about the feels, so I think it's really important that the user should feel empowered and in order to better do that, I think it's really important that we borrow from things like user experience design and information design, because ultimately that's what we're trying to do is provide a good user experience for the user, the client that's interacting with their API. It should be relatively simple for them to get on board and get started. I also pull a lot, I think, from the philosophy of Ruby, which has been very influential for me, so I think it's important to think about things like the principle of least surprise that people talk about, although people sometimes jokingly call it the principle of Matt's least surprise within the Ruby context, but this idea that if you already have some knowledge about how something works and you're going in and working on something that ought to be pretty similar so far as you can understand, it shouldn't work in a completely different surprising way. If it does, then you have a problem because now you basically have to apply yourself to that learning curve every time you're going to do a new thing instead of the curve being perhaps a bit steep at first and then leveling off, which is what you hope for and something that's more consistent and less surprising. So that being said, working at Heroku on this big, important API is actually the first API that I've ever really produced, which, you know, it's no pressure, not a big deal. You know, what's the worst that could happen? I still am at this job after a couple of years, so I guess I haven't done anything too horrific, but some of my colleagues might disagree with that at times. So let me tell you a little bit about this API that I sort of inherited. What we had was, I would say, it was organic, which because it was organic, it was somewhat inconsistent, and it was also private, and I think all of these things have a tendency to go together. So for us, the only real consumers that we had concerned ourselves with of this API were the command line tool and the webpage that we operated. Since we controlled both sides of the whole equation, in some ways it didn't matter a ton what the API actually looked like as long as it got the job done. And so what we'd had over time is that when people needed to add something new to the API, they would just add sort of whatever was, you know, sort of necessary even if it wasn't sufficient, right? Like as long as it did the job, that's what mattered. And it was a different person adding the new API almost every time. They weren't too worried about going back and reviewing the existing API to see if it fit in. They just wanted to get the job done. In the world we came from that totally made sense, but I found that I wanted something that was more carefully crafted, that was more coherent throughout, that from going from one resource to another, as I mentioned, was not surprising, that it made sense because I'd written a lot of these clients. I'd even worked against this particular API and it was really frustrating for me that half the time when I was working on the tool belt, I'd actually need to open up the API code base and read through code to even understand what a particular resource did because it would be so different from the resource right next to it. And also then I wanted it to be public and for it to be public and for it to be something that I was willing to stand behind and sort of put my name on and be proud of, I thought it was really important that it needed to be more carefully brought together. So the way that this ended up working is this sort of divide and conquer thing of like I started working through the API and started building out a new version of the API that had the same resources, but in this more consistent format by trying to extract some of the common patterns and destroy all of this sort of like beautiful and unique snowflakes that were within the system because that's not really what we wanted. We wanted things that were relatively uniform and then just sort of repeating that process over and over again. As it turns out, this is incredibly tedious, especially with something that has the large surface area that the Heroku API has. We ended up kind of doing it in this checklist driven way, which is actually pretty helpful for something like this where there were basically this set of steps of like, does this resource endpoint provide these things? Is it in this particular format and this order and so on and so forth? So it helped me to be more consistent than I would be if I was just doing it off the top of my head and I would probably forget things. But I did not love this process by any means. It was not exciting particularly. I thought it was important, so I kind of trudged through it, but it quickly became clear that, you know, we were in this process of growing the API as we were going to be adding more and more endpoints, things were gonna be moving faster and faster and that ultimately it wasn't something that I was going to continue to be able to really do independently that I was gonna need help, right? So I needed to like recruit other people to be more involved in this process, but up to this point, the vast majority of this process had been basically in my head, right? It was a process that I had, it was working, I was making progress, but it wasn't something that I could easily bring other people into. You know, I couldn't invite people into it. Other people within the company were not going to feel particularly empowered to work on this API. So from there, we started working on some different tools and documentation to make hopefully this process of API design and implementation more approachable. So I wanted to walk through some of those tools now, hopefully because they might also be useful or interesting to you guys as you approach your own API concerns. So the first one is the HTTP API design guide. We have an organization called Interagent that we decided to create on GitHub to kind of house these different API design tools and patterns just because basically the Heroku org has a bunch of open source projects on it and we felt that if we just threw all of these additional things on there, they just get lost in the noise. So by having one organization, we hope it's a place that we can kind of consolidate and keep all of the things that are closely related to this area. So some of you maybe don't know what 12 factor is, but so I'll just describe that very briefly. It's something that came from the original CTO of Heroku. It talks about the 12 factor app that kind of gives 12 like first principles that are good for like web scale type apps, right? It kind of describes in a lot of ways what the ethos of Heroku was in terms of what an app is and how you should run one in the modern context. So it's like this great set of first principles. It doesn't really say like you should use Rails. It's more like, you know, you should use environment variables to set temporary configuration. You should have development and production systems be as similar to one another as possible. Like these other, you know, top level kind of principles. So my hope was that I would do something similar in the API space where I talked about best practices and the things that were working for us and why they were working as much as possible. The explain part is definitely still sort of work in progress and that people tell me all the time that I've not done a good job of explaining why some of these things are. It's sort of like I was describing of this sort of like, I have the gut feeling that this is the right thing. I can't quite put my finger on why that is, but I'm pretty sure it is the case that we should do that. So I've been trying to go back through and explain some of this better, but it's an ongoing process. And the big thing I think that's important to realize there is that it's more about patterns than necessarily solutions. And I think that's how I try to do a lot of my job too. You don't really want to solve this particular problem once. Ideally you want to try to solve that sort of class of problems because inevitably that same sort of problem is gonna come up again and again and again as you work with more and different people and more and different resources. And so if you have just a pattern that you can say, well actually in that case we use this pattern. Use this pattern in the way that makes sense for this particular thing. It's a lot more powerful than saying, well maybe this time we should do it this way and maybe next time we'll do it a little bit different way because that's how you get back to this sort of organic and consistent and ultimately private state because it's so unusable that people publicly aren't gonna use it even if they can access it. So that being said it was important for us to have an easy way to actually describe the APIs that we were working on, right? Like actually just trying to like write out an API from scratch is like everybody kind of does it a little bit differently, they leave things out. So we wanted to have something that was human-writeable but also machine-readable because we felt that having it be machine-readable would bias a lot of capabilities because if it's machine-readable we can write programs around it, we can automate things related to it. So there are a number of different formats in this space and I'd be happy to have a discussion with people about the different ones and pros and cons but I'm not gonna do it right now. We settled on JSON schema and JSON hyper schema which are both in the RFC world of things. There are RFCs that describe both of them. There are ways to basically have a JSON document that describes an API and JSON hyper schema basically adds links to what would otherwise just be a description of JSON blobs to let you describe the whole basically surface area of your API which this gives us a lot of traction because now when somebody's proposing a new endpoint they can actually write a schema and you can very concretely see exactly what they mean. It's not sort of like well, I thought it would be kind of like the app's resource except for this other part which would be kind of like the user's resource and it's like I have no idea what that means. Please just point to a schema that actually spells it out so that we know that we're on the same page and we're actually discussing the same thing. The one thing that I would say that is maybe different is that we actually use YAML to write the JSON schemas because at least most of the people on the team don't wanna actually have to hand write JSON for obvious reasons. So a little bit of an optimization there which I would recommend because it does make it a lot nicer to actually work with. So I'm gonna get one sip of water. I don't know if people have any idea what this is but I will explain in a second. So this is my sort of inside baseball for you today of why the thing that I'm about to describe is named what it is. This is a pyramid scheme. I don't know if you guys are familiar with pyramid schemes. There's a very famous one recently with Bernie Madoff, right? He got all these people to give him all this money. Supposedly that they were gonna get all of these returns and then he like disappeared basically. So we have this tool called pyramid that I basically named in effect because I wanted to sort of remind myself to remain somewhat skeptical and that the schema is very powerful but it's not the sort of be all and all. I don't think it will solve every problem ever but it is very helpful. Anyway, it's shortened so people usually don't realize what it actually is supposed to be because it's PRMD but it's supposed to be pyramid. So this is a command line tool that provides a lot of how we actually get work done with these schemas. So there are generators. So when you're creating a new resource from scratch, it turns out there's just a certain amount of boiler plate that you're gonna end up with pretty much every time especially with resources that have CRUD. Well, CRUD looks pretty similar on most resources. You don't need to specify a lot of unique stuff. So you have a generator that will spit out that file for you. We have verifiers that will take one of these files and check to see if it actually matches with our expectations or not until you when things are broken. If you look at the documentation for the Heroku API it actually all is just automatically generated from our schema which helps us hopefully at least keep things more in line. Like if we have a schema that we agree upon and we're pretty sure our implementation matches it then the docs ought to match everything else too because it all comes from the same place. We also have stubbing capabilities there. So you can actually basically spin up a server that will pretend to be your API that you can do client work against or just play with to like do curl commands and make sure that it's doing kind of what you think it ought to do. So anyway, super useful tool. If you're getting into the world of JSON schema I think there's still a lot of work we can do here to improve this tool but it's already been really helpful to us in terms of breaking things down and making it easier to actually work with these things. Basically like empowering people that don't want to use JSON schema to use it more easily because the learning curve can be a little bit steep on it. All right, so we were still in this sort of situation. All right, there's this sort of idea of like you know on this side we have basically a JSON schema or something on this side we have a fully formed API, right? Not super helpful for people that haven't done this before there's still a pretty big gap between these things, right? Like just having a schema that describes the API and actually implementing and operating an API are quite different things. So in order to kind of narrow that gap at least we have a couple additional tools. So the first one is called Pliny and in simplest terms I would say it's kind of like how Rails is this sort of big opinionated framework that makes it really easy to make front end like web pages, right? Pliny is intended to be sort of a somewhat smaller but still very opinionated framework for making APIs specifically. It doesn't really have any notion of views. It doesn't do things with templates or whatever. It's all just about doing APIs. And a lot of the patterns that are in here are basically extracted from our API code base that we haven't done like a grand rewrite to make it all beyond Pliny. It's still like a mix of all of this crazy old legacy stuff but there are certain patterns that as we worked on it we found to be incredibly helpful to make designing and implementing these APIs easier. And so when we're making new APIs within the company now which we do on a somewhat regular basis which I'll touch on in a minute we usually start with Pliny as our foundation and it makes it again it just lowers that learning curve. It empowers people to build and deploy their first API because there aren't all of these crazy things you have to figure out to start with. And one of the things that's like at the core of that that you can use independently and that we do use in our existing API is something called committee which is a rack middleware that can take these JSON schemas and do things with them. So there's a couple important things that committee can do that are very powerful. The first is you can use it for input validation. So inside of your schema you actually say when somebody creates one of these objects these parameters are required. This parameter needs to be in this particular format. So for instance like an email address, right? You can say this email address needs to match this regular expression or it's not an email address. And committee can actually take the incoming request compare it to the schema and return valid like validation errors directly. So it can say this is 422 unprocessable and see the thing that you pass that's supposed to be an email address clearly not an email address. And it lets you, it sort of alleviates needing to do all of that over again by rewriting basically the same thing that you put into the schema into your code base. Should you still have it in your code base somewhere? Probably because I don't know, trust but verify but it should actually handle it and in a lot of cases I think does a great job of it. On the output side it's also able to basically look at all of the responses and check the responses to make sure that they're also valid format but you wouldn't want to raise an error in that case because it would be bad for the user. So in that case we usually would look at grabbing that error and sending it up to like an error reporting surface. So like a roll bar or airplane or one of those kinds of things would get these errors that say, hey, you have this response that really doesn't match with the schema you have so you should probably either fix the response which is probably actually the wrong thing to do because you'd be breaking the API you'd be changing the behavior of it. You should probably actually fix the schema in that case and make sure that your docs have been regenerated then to match this actual schema and the actual reality of things. Okay, so umbrella. So the new thing that we've started doing a lot within the context of Heroku is something that we refer to as the umbrella. And the idea with this is that as we've continued to grow and the surface area of the API has continued to grow it has not seemed like a good idea to continue piling new endpoints onto the monolith. So more and more often as we add a new resource that's relatively independent from the existing resources we'll actually have a team inside of the company that is concerned with that particular thing spin up a brand new API service using Pliny and then we'll basically just attach it to the existing API via sort of like a proxy infrastructure. So that's the umbrella part where we have this umbrella that covers all of the things underneath and kind of hides away from people that it's actually a bunch of disparate parts. So we found that this has been really powerful and the Pliny stuff has been a big enabler for us to do that. So we are creating these sort of net new APIs but we're just making an effort to not make that be obvious to external users because you don't care, right? And if you had to use a different host name to get to every different endpoint in the API you would quickly pull all your hair out probably. So in terms of actually consuming APIs then we also commonly fall back to Xcon which is something that a long time ago I basically extracted out of Fog and this is a sort of low level HTTP library that in particular I optimized in a lot of ways to support usage as an API client because the use case of an API client is not quite the same as the use case of a general like net HTTP or whatever. You can certainly use this in that general way but this is kind of geared towards like using one connection for many requests, having persistent connections and also just like being crazy fast like as much as possible I tried to make it very fast and it tends to be quite a bit more performant than especially like net HTTP which is pretty slow for I could go on for longer than you wanna care to hear about why that is but anyway it's kind of slow. So we use this as one of the main ways that we communicate between these internal services. So the umbrella uses Xcon clients usually to talk to these backing services and get the responses for the end users. So I wanted to talk a little bit about some particular lessons that we've learned and hopefully to give you a bit of clear idea of some of my philosophy of this as well as like how we concretely have created some of these patterns because hopefully that will kind of drive home what we actually do with these tools and how some of these pieces fit together. So one of the first ones that we've kind of gone back and forth on this but I feel relatively confident. I have like a varying confidence level on all of these things. Like I don't know that I'm certain on anything which is disconcerting to say the least but it's this pretty common pattern you'll see of like these deeply nested things. I think this comes from Rails. I don't know what version of Rails or whatever it's this DHHism maybe but we had these like really deep nestings pretty commonly and it comes up a lot in the Heroku context because at least historically almost everything belonged to an app. So you'd have app slash whatever slash some other things slash whatever maybe slash some other things slash whatever to infinity, right? Because everything belongs to an app so let's just nest everything. Well, I find over time that I did not really like this at all and that really we ought to I think be minimizing the nesting wherever possible. So basically just avoid going to a depth greater than this, right? And so there's like some implications of this though which is that like that child now probably needs to have the parent foreign key serialized on it because before the parent foreign key was implied by the path that you were going to but now you have no reference to it and so you probably actually need to know which parent it refers to but this has stronger information sent which maybe also makes no sense to anyone that hasn't studied information foraging stuff but the idea being that if you are foraging and looking for food and you get no sense that there's any food in an area after a short amount of time you'll give up and you'll go and look somewhere else in the context of an API especially if you have competitors in the space you don't want somebody to come in and try to figure out how to use your API for a little while not find any sense in how to do that and just leave and go somewhere else to find the same thing. So by having everything be nested at the same level it makes it easier and more consistent for people to figure out because they're like oh well I wanna interact with this resource where do I interact with it at? Well in a nested world it's like well does this thing nest? Is it a child object or is it one of the Uber parent objects? Well you have to figure that out on each individual resource versus a world where everything's unnested and you can just assume that everything is always at the top level you don't need to think about it it's much more consistent that way. That being said I can still see maybe a case for having a nesting like this just for the case of asking for a list of children that is scoped more narrowly but there are other mechanisms by which you can do scoping so I'm not sure that doing it in the path even makes sense I think maybe just having a single level of nesting is sufficient just about every case so anyway that's one concrete example that I think is helpful another one that we find has been really helpful is sometimes within the context of a REST API there are things that don't concretely fit in the sort of like create read update delete world in a lot of cases there are basically things that I think you could map to like state machine transitions and so we needed a way to sometimes represent that a concrete example within the context of Heroku would be dinos where you sometimes wanna like restart a dyno or stop a dyno and it's not clear that you should just like set the state of the dyno to restarting like because then it actually causes other things to happen so we wanted to have a way to like more distinctly call that out so in a lot of cases in other APIs you'll sometimes see something like this and over time I found that I didn't like that because it was another place where there was some inconsistency because normally and granted if you're in a denested world this is less true but normally you see things that have this certain cadence of basically resource identifier resource identifier and this has two identifiers in a row which is certainly it does not seem consistent and it's not really clear what that final one is other than through perhaps maybe some convention you know that people would get used to it but I don't want people to have to get used to stuff I'd like it to be relatively obvious to them so we ended up actually implementing this basically where we kind of treat actions as a collection that actually belongs to the resource and within that we want to call the particular action and we found this to be really helpful and to be a nice pattern that we can reuse anytime we have these actions and make it clear exactly what this thing actually is it's basically a state machine transition it's one of these actions it's more complex so call it in this other way the other maybe less obvious application of this is I think in order to get it to work if you need to do it at the collection level which doesn't happen very often and actually in the cases where we've ended up doing it I've been dissatisfied that maybe it's a bad idea but I think you would probably end up nesting it this way again this allows you to have the same like consistent style of nesting in order to show that you're actually operating on the whole resources collection another one that's come up is the Singleton pattern like it's not that uncommon that people would have like slash account slash whatever as one of the URLs like one of the account things basically you're operating on the current user and we found that that was really not great because more and more often we have cases where admins either in an organization or within Heroku need to modify a particular user in some way and in order for this to work you have to use impersonation right you have to pretend to be this person in order to have account mean them even though you're you and then that makes auditing awkward so we've actually settled on using this instead where tilde means sort of the default value so users tilde is the current user and in other contexts I think tilde might mean other things although I don't think we've used it anywhere but the user context so far so some things that were good patterns I think as we shoot for IO parity so for the vast majority of cases the style of JSON blob that you pass in to create something and the style of JSON blob that you get back from that same resource should be very similar that's not always the case in a lot of APIs and I think that there's a lot to be gained from having that similarity that makes it easier to reason about we use UIDs throughout I think that also simplifies some things and makes it more consistent and hopefully more scalable but we also provide friendly IDs that you can use so like for app you can either address it by UID or its name and having both gives you a lot of flexibility we also use request IDs since we have this complex like service oriented architecture when a request is coming in you can provide a request ID or even if you don't we'll provide one for you and anytime we're making calls back through this network of backing services that request ID has passed along which actually provides us a way to trace its route through the entire system which can be very valuable so we still have some development woes though like I said there's still a steep ish learning curve I think some of these tools and stuff help there's also this sort of idea of this seductiveness of specialization a lot of people would love to see an API that did just the thing that they wanted even though it might not be useful to anyone else in the entire world but we're not really in a place where we can manage to make tons and tons of special APIs we need to make a more generic one in order to serve everyone hopefully the other thing that's super challenging is change management right so we do this via stability right now at least this is how we say that we're doing it we've never really changed anything in a breaking way so we haven't actually had to put this into practice but our plan anyway is that we have these stabilities that are at a per resource level if it's prototype stability we basically say we'll give you a week's notice if we're gonna make a breaking change in development we give you a month's notice and production we give you a year's notice I think something like this is maybe reasonable but we've already found some places where it's like this is kind of hard that it's at the per resource level that the whole endpoint basically has this same stability contract but I'm not sure what to do about this I think it might be that we just need to try to find easier ways to version because since versioning is hard we are reticent to make changes because we don't wanna have to support multiple versions at once versioning by the way, super hard one thing I would recommend if you're making a green field API is to not have a default version I think that's an anti-pattern one that we're stuck with because of legacy reasons but that's another surprising thing right because if the default's version two now and you release a version three and you'd like to make it the default suddenly you will break it for a ton of people that never even realize that they're sort of opting into that behavior I think it's a lot better to ask them to explicitly specify which version they'd like to be locked into there's also the complexity within the context of versioning of like if I have version three now and I wanna release version four do I have to rewrite every endpoint to be version four can I just say this endpoint is version four and this is the only endpoint that's available in version four or do I say this version four endpoint is different so you can access it and if you say that you want version four and access an existing other old endpoint it will fall back to version three's behavior it's probably also bad there's a lot of trade-offs there supporting multiple versions also just has a lot of operational overhead like operating in effect like multiple apps for your API instead of a single one I don't have a great answer here it's something that we're still trying to figure out so kind of bringing that all together so what is my conclusion from all of this I think my conclusion is basically that I still feel like I have no idea what I'm doing for the most part I mean there's some things that like I said varying levels of confidence but I think that honestly this is kind of where almost everyone that's doing this kind of stuff is and we don't necessarily have an open discussion about it because who wants to like go up and stage in front of a bunch of people and admit they have no idea what they're doing but I think that it's valuable for us to have these broader discussions and talk about these tools and these patterns and what's working and what's not working so that we can move the industry forward so I would really welcome you to consider being a part of this and coming and joining me and being lost we have a lot of great discussions on some of the different interagent projects of the things that are working the things that aren't working a lot of discussions about the feels and where I try to explain why I actually have some of these feelings or APIs I've interacted with in the past that I didn't like and why I didn't like them or whatever that might be and so I hope that some of that will be helpful you can certainly find the various projects on Energy Agent you can find me at Jemus and that is a link to this slide deck should you want to pull it up to find these links or that sort of thing so I think that's it I don't know that I have very much time for your questions I'll be around all day though so thank you