 Hello. I'm Deepan Subaya. I lead front-end engineering at British Gas. In case you saw me logging in, you had Sebastian Wettel on my screensaver client. Do not want to race for like a year. He drives for Ferrari. Good example that a lot of money doesn't necessarily equate to success as Ferrari have the biggest budget in Formula One, but I haven't won anything for one and a half years now. So yeah. So anyways, so going into my talk, what I wanted to share in this meetup really is our experience working with JSON API, basically. Some of you have probably heard about it. Some of you might already be using it in protection. Some of you might have just played around with it, but we've just released a couple of microservices into protection at the moment that actually use the JSON API spec. And then we have ember apps on the front-end basically that consume those microservices. So I just want to share our experiences with JSON API so far. I would like to start off by explaining or having a look at Parkinson's law of triviality. This law actually states that given the choice, people will argue more over a trivial factor rather than the broader scope. So it's also sometimes referenced to as bike-shedding basically. So the concept being that if someone asked you, asked a group of people to actually design a bike shed, then there's a lot of chance that more people or most of the time will actually be spent on the color of the bike shed as opposed to the design of the bike shed basically. So you spend a lot of time on something that's trivial that can be changed easily and then you find that you actually lost sight of the broader scope which was to design a proper bike shed. So this problem is not actually specific to development. It exists in other areas as well, but you tend to see that it's a bit more pronounced in development as developers sometimes do have a tendency to kind of spend a lot of time on discussing trivial things, things that should be just working out of the box, things that you should not really be focusing on. As developers, I think our focus should be on building great products and not actually be debating or discussing trivial matters. The cost of a triviality is actually pretty high. Spending a lot of time discussing trivial things actually costs a lot of productivity and in turn what that does is that it costs your product time in the market basically. So you develop applications less quickly. You develop applications or you send applications to production less frequently. So ultimately it's something that impacts productivity and impacts the time your product gets to spend in the market. It's a problem that a lot of teams realize that they don't have. They only find out after a lot of time or when they explore specifically why their productivity is not great. Spending time on trivial matters is not usually recognized as a problem but then if you do sit and think about the conversations you've been having when you were developing a product, you'll find that actually a lot of time was spent on things which you probably shouldn't have been spending on and you probably lost sight of the broader scope of what the product was supposed to do. So frameworks like Ember obviously because of its adherence to the convention over configuration principles, because of its adherence, because of its opinions basically, it kind of tends to combat the bike shedding mentality basically because it makes choices for you that you don't have to worry about then you can actually focus on building the product as opposed to worrying about certain things. I wouldn't say all of the choices that Ember makes for your trivial, that would be a massive understatement but it makes some well informed choices for you so that you don't have to worry about what your build pipeline is, what your preprocesses are and things like that. So it helps you focus on building the product which is our primary job as developers basically. And building great products is hard, designing great APIs is hard and really tools like Ember that allows us to focus on doing that is actually a good thing. So for example, what we did recently was we wanted to move our API platform into a microservices based API platform. So the first thing we were trying to do is like come up with a resource mapping chart really where we wanted to put all the resources that we want, API resources that we want, obviously all these resources map to Ember data models on the Ember apps basically. So we wanted to lay out a structure of how we want our resource hierarchy to be, what the relationships between the different resources are and things like that. So there was a lot of time spent basically trying to come up with the right place, the right relationships for the various resources and things like that. The last thing we wanted to be discussing was what should be the name of the parameter you used to sort in querying to a resource basically. So that is where really JSON API helped us a lot or is still helping us a lot in this process where we get to focus on the real hard bits of the solution as opposed to the bits which really should be solved by a framework or a specification out of the box. And one thing we found with resource-based API design is that people for some reason tend to hate relationships, I don't know why, but API does specifically hate relationships for some reason. So they've always asked us why do you need relationships between resources. The best way I've found of explaining why you need resources basically or relationships is basically if you had a website that had fire related pages basically and if you couldn't get from one page to another related page and you had to remember the URL of each page to put in manually on the browser, that's kind of equivalent to having a set of resources that have no relationships between them. Yeah, you have five pages that work on their own but then the user has to work his way out and put the right URL in to get to the next related page. Anyways, we've been getting better at that. So taking on from the good things that Ember gives us with its convention over configuration mechanism, its opinions and things like that, how about taking convention over configuration to API responses? Now I think API responses are an area that gets taken for granted in API development. I think people sometimes API developers pride themselves in having the best API implementation. And they don't really care about the API bit of it, which is actually how the API is requested and what response the endpoint sends back to the client. So having a great API implementation and not worrying about how the client actually uses the API is like having the best pizza machine in your pizza shop but having a poor way of, or basically having rubbish way of letting people order your pizzas or having a poor packaging and delivery service. So you might have the best pizza machine but then the client is not necessarily seeing the values of the great pizza machine you have. So I think in terms of that, what we've seen is that the client actually doesn't really care about how the API is implemented. What it really cares about is how effectively it can talk to the API and how efficient the API can make it basically. So that really is the concern of the client. It doesn't really care about whether your endpoint is a microservice, whether it's built in Java, whether it's built in Node. All the client really cares about is how effective an interface the API provides to the application itself. So in order to take convention over configuration to API responses, JSON API we feel is a big step in that direction. Obviously there have been specifications that cater to some elements of it before as you might be aware but I think JSON API takes it to the next level. So JSON API as you know is a specification for building JSON APIs. Ember data now supports JSON API out of the box. In fact, it is the default model for Ember data at the moment. So before actually going to talk about the aspects of JSON API, I would like to share with you. I want to share a small application we recently developed using JSON API and then I can use the response formats that we used in that actual application to show you the different aspects of JSON API. So what we had to do recently was to build an application that allowed the user to complete a survey. So the purpose of the application was to allow customers to book a smart meter installation appointment. So smart meters, I don't know if you heard of them, a replacement for dump meters. So the thing with a smart meter is basically that you don't have to go and put in a meter read yourself. The smart meter sends in, it meter reads automatically to the provider. So it's not a hard thing looking at your meter every three months and submitting a meter read but apparently people find that hard. So we are going to replace it with smart meters that do exactly that. So by 2020, every household in the UK is supposed to have a smart meter and apparently if you don't have a smart meter in your house then you have to pay a penalty to the government. I don't know how that works. But anyways, that's not what this application is trying to solve. So this application's intention basically is to ask the user a set of questions so that it can find if your home is currently eligible for a smart meter and then based on your answers to that question, to the set of questions it'll tell you whether you're eligible and allow you to book an appointment for one of our engineers to come to your house to install a smart meter. So the first step in that application really was to build a survey which basically asks you a set of questions. So the way we designed that, so the first step towards designing that application is to design your resources, your API resources and your embedded data models basically. It's quite a simple design resource-wise. So you have a survey resource which has a collection of questions. Each question obviously can have multiple answers basically. So initially when you go to the application, the application would display the survey which is basically a list of questions and then every question would have a set of answers. I'm just going to see if I can show you that. I'm going to make the cardinal sin of trying to connect to a test environment on a live demo but I'm hoping it works. So that's our login page. So what you do is you get to put in your username, see if you can log in. So at the moment we have five EMBA applications. So that application you just saw was our identity app which we basically use as our authentication and authorization application. So obviously the intention is that in the future there are other companies that can actually use the British Gas platform as an authentication provider. So that's our new account management dashboard. It's again another EMBA application. We have about four EMBA applications, some focused on energy sales and services sales which are our core sales endpoints. This one is basically for customers who sign up online so that they can submit a meet or make payments, view their consumption and basically all like self-service kind of functionality. So you get to that page and then customers who can actually are eligible to book a smart meter upgrade get shown the style which you can click and then what it displays is basically a survey like that. So this is the survey that I initially mentioned about when we were showing the resource relationship document. So I'm just going to go over now and show you the requests we have here. I was wondering how would this play the survey without actually making a response. That would have been interesting. So this is the survey endpoint. So what I'm going to do actually is to switch between the actual endpoint and the actual spec because we haven't actually used all bits of JSON API. What I can actually show you from what's been implemented, I'll show you on the actual app and then we'll also keep switching to the actual JSON API spec as well to learn about a few other things. So here, as you can see, this uses the JSON API spec basically. So the idea here is with JSON API, what you can do is that you have... So JSON API basically defines a lot of different sections basically. One of the sections, for example, is document structure like how your JSON documents should be structured and sent back to the client. So every JSON API document basically needs to have certain top-level elements. It needs to have at least one of certain types of top-level elements and it can optionally have certain other types of elements. So in terms of mandatory top-level elements, your JSON API response need to have one of either data or errors or meta. So these are the three top-level elements and you need to have one of those in your response. Now, you can also have other optional things like included which we'll come back to. So if you do expand data here, what you see within the data element is actually what JSON API calls resource objects. So a resource object should have at a minimum an identifier and it should have a type. Now the type really is useful for people... Because we've seen not all endpoints that actually return a survey resource are actually called surveys basically. So you might actually have an endpoint that actually returns a survey resource but it's not called surveys. So it would be hard for anyone to actually goes through that endpoint to find out what type of resource is being returned. So the type actually allows you to infer what type of resource has actually been returned back to the client. In terms of interaction with the application itself, so it asks you a few basic questions here. So if you say, do you live in a flat? You say yes to that. It asks you if your gas and electricity meter is located within five meters of your property. Some customers have called us and asked what if my meter is like five meters under the ground? Does that count as yes or no? I don't think we know the answer to that ourselves. But anyways, we do ask the question. So the thing with this is that initially the survey actually has only... If you expand the attributes on that, you can see that the survey only has two questions, which were the original two questions you saw, which was if you live in a flat and whether your gas and electricity meters are accessible. Now when you select yes to this, if you remember the entity relationship diagram I showed you, answers in some cases can have additional questions. So some answers can have questions. For example, in this case, if you select yes to this, there is an additional question that pops up to the user. Now the way we handle that basically is that once the user selected that answer, we then push the extra question into the has many relationship on the survey basically. So that the survey which originally had two questions now has three questions. And then once the customer has actually answered all of those questions, we then post the survey back to the server which looks at the questions, answers, and then sends the status back to the client saying whether the customer is eligible or not. So that's kind of like the high level design of the application. So like I said earlier, there are some aspects of JSON API that we found really useful for us. One is the document structure which I've just showed you. It does impose a spec is quite exhaustive. So we're not going to go through all aspects of it. But if you kind of go through the spec, you can see that it gives you a clear document structure that your API responses are supposed to follow basically. It also includes support for something called compound documents, which you might be aware as side loading if you used it before. So side loading allows related resources basically be loaded as part of the primary resource. And JSON API gives you a standard way of how you should request for side loaded resources basically. So if you go back to our application, so you see here, if you look at the resource, if you look at the actual request, you can see that it actually includes, it specifies an include parameter there. So it asks for the survey because the survey has an has many relationship to questions. What that request is basically saying is that I want the survey and I also want all the related questions to be side loaded as part of that request. Now, we found that really useful in cutting down unnecessary HTTP calls because one of the problems you can have potentially with resource-based design is your application can get a bit chatty as in some situations you might have to make a lot of HTTP calls basically to actually show the customer certain bits of information, which obviously isn't great if you're like focusing on mobile performance and things like that. So side loading gives you a way to get around that by limiting the number of HTTP requests you basically make. So here, like I said, you actually have the ability to side load relationships and JSON API gives you a standard way of how that should be requested from the API. What you'll also see there is a secondary include, which says not just include questions, but also include question dot answers basically. Now, if you look at the entity diagram I showed previously, survey obviously only has a direct relationship with questions, but the questions are then related to the answers basically. So what you really need to display this thing here is that you need the survey, you need the questions and you also need the answers related to the questions themselves. So how do you actually retrieve relationships of a relationship? Now, these are kind of things that when I was saying earlier about bike-shedding are the things that can take a lot of time basically because people will have different views on how a relationship of a relationship should be requested and then you probably spend three hours trying to discuss and come up with an agreement on how you should be requesting a relationship within a relationship. But what JSON API gives you a specification that says if you do want to do that, then this is how you do it. And then both app developers and API developers have a clear understanding of how that specific thing should be done. So that's with the resources. If you see the responses here, you can also see that we use the relationships here. So there are two ways you can retrieve relationships here. One is you can actually specify the data element. So the object that appears within the relationship object is actually called a relationship object obviously. You can choose to return the relationship IDs here, but in some cases if you don't want to return the relationship objects here, what you can do is just specify a link to your relationship. So what embed data does actually is that if it doesn't find a data element here, then it actually follows the link. So it allows you to actually... So the data element really is optional there. If you want the client to retrieve your relationship through a relationship link, JSON API gives you a specification for how to do that. An embed data knows how to work with that specification. We found that useful in cases where populating these IDs are hard because we use SAP. So a lot of the calls that we make to SAP to fetch that information takes about like seven seconds. So obviously we don't want our customers to keep waiting for seven seconds to just pull down all related data. So what we do in some cases is we just specify the links for calls that are actually expensive. We just specify the link. And then when your application actually needs that actual relationship data, you can follow the link and fetch that data, basically. Included basically here is all your included relationships. Clearly in our original request, we wanted all the questions to be included. So the included object actually includes all your included resources that you can see here. As you can see, it includes both questions and the answers related to the questions because we wanted both in order to be able to display that. So that's compound documents, really. The other thing that causes a lot of conversation in API development is response codes. So what do you return? Do you return a 200? Do you return a 404? Do you return 200 with a blank array? Do you return 204? Do you return 202? HTTP codes are a common cause of lower productivity. So much so that someone last week when we were discussing response code, someone was so fed up that he said we should use 418. If you do look up 418 on the HTTP response code list, it's a response code that says, I'm a teapot. So response codes are again a cause of a lot of discussion, basically, on what you should actually return. Because we had scenarios where if you request for customers and there were no customers, there were no customers. Should you return a 404? Should you return 200 but with a blank array? What really was the right thing in that scenario? Especially when you go down to more complex operations like update, what do you find is that the response that the API needs to send back to the client is not immediately obvious. People always assume that it's 200, but it's not always 200. There are other response codes that actually represent the operation far better than a 200 does. And if you're following restful API development, you should really be aware of those response codes and make sure your API returns the right response codes for the right operation, basically. And because response codes are usually a tricky thing to get right, they result in a lot of brawls at the pub. We've had discussions about response code for hours together and people still can't make their mind up. Thankfully with JSON API, the spec is quite exhaustive. It tells you that if you're doing this specific operation, like if you're doing an update operation, then these are the different response codes that you need to return for the different scenarios, which we found really, really helpful because then you know immediately based on what operation is being done by the endpoint, you can just look at it and you know what response code you need to send back to the client. And because embedded data plays nicely with that as well, because it complies to the JSON API spec, you don't have to worry about whether the response code is going to be recognized or things like that. Querying for data, like I said earlier, things like pagination, things like filtering, things like sorting, what names do you use for your sort parameter? How do you pass multiple sort parameters? So all these are things that generally result in a lot of discussions, basically, that probably waste your time. So what JSON API does is actually define all that for you, basically, in terms of what those parameters should be called, what orders should be called, what are the response codes for those requests and things like that. So querying for data, obviously, is a key element of an API and the JSON API spec really helps you by effectively being a go-to point for you to understand the parameters and the response codes for the various operations. Now the good things we found is pass field sets in JSON API. Now, just pass field sets are, API developers basically hate it, but clients love it, basically. So the point with pass field sets is that if you had a resource and it had five attributes and say you only wanted two of those attributes, you don't want all the five, you only want two attributes. So pass field sets in JSON API allows you to do that, basically. So you can not just request for a resource, you can request which attributes of the resource we want, which is great because some of our endpoints actually talk to multiple SAP endpoints to get the information they need. For example, the first two attributes can be populated from one SAP endpoint. The next three attributes need to be populated from another SAP endpoint. So basically, the API is as slow as how those two endpoints are, SAP endpoints are, basically. But if you had the option, if the app only needed the first two attributes, then obviously, it gets a performance benefit because your API is just making the one service call as opposed to two service calls. So we are actually trying to, there are scenarios and applications at the moment where that would be really helpful for us. And I'm sure it's something that we would be implementing into our API stack at some point. The usual create update operations, post, delete, put patch. Now, if you're not used to JSON API before, JSON API supports patch instead of put for update operations. Patch was always the right thing for that type of work, but we generally used to put previously. So at things change, we make things better. It didn't work in I11 when we put it live, not surprising. But it was not because of patch. It was because the version of jQuery we were using was actually instantiating the wrong version of the XML HTTP object in IE, basically. So it didn't really support the patch operation. So it actually works across browsers IE9 and above. I'm hoping that none of you have to deal with supporting IE8. So we're not going to discuss how it works in IE8. I don't know. Errors is the last thing we found really useful. Again, errors very trivial. So if your API hits an error in this processing, what do you send back to the client? What format does the error needs to go back in? It's very easy to spend loads of time on it, saying, OK, should we call it error code? Should we call it error ID? Should we call it error description, error title? What is the right thing for it? So it's easy to spend loads of time on it. But what JSON API spec gives you is a cleanly defined object with different attributes that have specific purposes when you want to give an error back to the client, basically. So again, anti-bike shedding, spend time building the product, don't focus on spending time on things that are not really useful for the broader scope of the product. And finally, embassy alignment edge. So if you're worried about how you would actually mock JSON API responses, you don't have to look any further than embassy alignment edge. So version 0.2 of Mirage actually supports JSON APIs out of the box. So it has a built-in JSON API serializer that allows you to mock JSON API responses really, really quickly. So it's the tool to go to if you're interested in using JSON API and you need a mock framework for your application. So yeah, that's our experience with JSON API so far. It's definitely a spec that we found really helpful. And I'm sure it would be helpful for your apps as well. Go have a play around with it. Look at it. Some people complain it's a bit too verbose and too exhaustive. But we found that if you get beyond that initial phase, it can really help you with building applications really quickly and help your APIs be neat, concise, and follow a convention that's useful for all of the consumers who consume that API basically. Yeah, that's it. Thank you.