 Hello, my name is Alec and I'm a software developer here in Edinburgh. I work for a company called Administrate and we make a training platform for organising and running your training business. We're one of Scotland's fastest growing tech companies. We have millions of students, thousands of users and hundreds of customers use our platform around the world. We also work four days a week and we're hiring so give me a shout if you're interested in that. I'm not here to talk about that. I'm here to talk about APIs. Hopefully. There we go. Here to talk about why we rebuilt our REST APIs with GraphQL. This isn't a REST bashing session or even a comparison between the two. I want to talk about some of the issues that we were having with our implementation of our REST APIs and then how GraphQL was the right choice to help us tackle some of those issues. We'll cover what the basics of GraphQL is, how the tooling enables our development and then we'll dig a little deeper into how you can use Python with GraphQL and specifically the way that we do it at Administrate. Here's an age-old battle between time and money and in a software development context that usually boils down to writing beautiful code or shipping features. When you're starting out, the trade-off between quick results and money sort of makes sense. You ship the feature, get the customer and stay afloat, but that turns into technical debt pretty quickly, especially considering how quickly you can build APIs in Python using the frameworks like Django or Flask. We had some issues with this at Administrate. Some of our problems were that we had two versions of a public API, a private API, and even though one of our public APIs was deprecated, we still had to support it. Keeping our documentation up-to-date was really hard. We had a really tight coupling of the front end and the back end to save ourselves some time in the beginning. We had automatically generated some endpoints by introspecting our SQL Alchemy models, which meant that we got a lot of functionality really quickly, but it was pretty difficult to work within the front end. A cognitive load was high for working with our APIs. The namespacing was awful. I couldn't really find where anything is. It was just pretty confusing. There was basically just not enough thought giving to APIs before we wrote them. Our private API had tons of ad hoc roots in it, and it was sort of a get the data to the front end by any means sort of attitude. Starting again with an API from scratch might sound a bit risky, and it is, but hopefully you'll understand why we chose to do this and agree with some of the reasons why after this talk. GraphQL, what is it? It's an API specification. It's been built and open sourced by Facebook in 2015, and it's now used by many companies as part of their APIs. GitHub, PayPal, Twitter, and of course Facebook, just to name-drop a couple. But what actually is it? Let's have a look at the definition. It's a query language for APIs and a runtime for fulfilling those queries with your existing data. That's still fairly vague. In practice, what does that mean? As a developer, you're going to have to define some data types in your code. You need to create resolver functions to help you resolve those types against your existing data, then you pass the whole lot into a runtime, and there's your API. That's pretty easy. It's still pretty vague. Let's dig in a little bit deeper. GraphQL is not language specific. It's just a specification, just an idea. That means you can implement GraphQL in whatever language you like. That makes it really flexible to integrate with your systems. There's implementations in JavaScript, Java, Python, go lang, you name it, there's probably an implementation. The data retrieval part of it's also up to you. You can use SQL alchemy models. You can use Django models. You can use text files on disk. It's up to you. Next up, there is a single endpoint. When you talk to a GraphQL endpoint, you need to build a query with the types of data that you want and the fields on those types, and then you post that off to the GraphQL endpoint and data is returned to you. This makes data retrieval far more efficient. It means you're not having to reach out to hundreds of services or many services to get the data that you need. It cuts down latency over the network immediately, which is especially useful when bandwidth is at a premium like when you're doing mobile development or when the internet is just slow. The single endpoint also means there's no versioning pain. There's only one endpoint, so there's only one version. That cuts out the maintenance pain of maintaining two versions. You just keep adding types to your API and evolving it. For us at Administrate, it was a really huge mindset change. Going from having a private API where we could just literally throw in whatever we wanted and it didn't matter to having a public API that meant everything that was going in the API had to be right. With GraphQL, you get what you want, so as you mentioned, you create a query, you send that query off and data is returned to you. There's nothing extraneous there. The data that you ask for is the data that you receive, which means that you get a sort of efficiency on the back end too. If the client's only asking for the data they need, then you only have to fetch the data they need from the database, which means that you can make your queries more efficient. Another part of this is the format. When you define your query, the order that you define your query in and send off to the server is the exact order that comes back in. What that does is it shifts the responsibility to the front end or the client who's making the query, shifts the responsibility of getting the data to them. It really uncouples the front end and the back end from having to know what data each one needs. Once you have a schema defined, then the front end knows exactly what data it can get and how it can get it. I just want to do a little illustration of this. Here's a typical GraphQL query. It's a JSON-like object. We define a type on it. It's event type. We pass in a parameter, event ID, no more query strings. Then you've got some fields on there, title description and talks. Then the response is going to come back. It's a JSON-like object. It's in exactly the format that you've asked for it in. Compare this to making a request to one of our API endpoints. This is our events endpoint and being a training platform. This is one of our biggest domain entities. You'd expect to get something back about events. We're not really sure what. It could be anything. There's an illustration of what comes back on our events endpoint. Phil expanded it's about 400 lines long. It's a biased example because there's a lot of data attached to our events. The main thing here is that there's no guarantee about what you're going to get back from your API when you're just querying an endpoint like that. Processing a response like this in the front end is an overhead and it's also a risk. Something in the back end could change and that could be a breaking change for the front end and you wouldn't know about it until it was too late. Unless you had a test, I suppose. The next thing I want to talk about is mutations. I mentioned queries in GraphQL. If queries are the adenpotent equivalent to making a get request on a REST API. So queries should never change data on the back end. Mutations, on the other hand, are the way that you would change data. Let's just take a look at the structure of that. This is a typical mutation. Basically, you need the keyword mutation. It's a JSON-like object again. You drill down into the type of mutation that you want to run. Here it's account. Then on the account type, we have the create mutation. That's not keywords. You can call those wherever you want. It's really useful for adding workflows to your API. For instance, we have a mutation called add learner to event, which is far more explanatory about what that action is going to have, what effect that action is going to have in the back end, instead of just sending a post request to an endpoint and hoping for the best. Once you've decided on the mutation that you need, you need to add an input object with the data that you want to pass into the system. Underneath that, where we have account and then name an email address, that's a query fragment. That's going to be the data that the system sends back to us once we've run the mutation. As before, it looks pretty similar. It comes back in the format that we've asked for it in. Next thing is that GraphQL is strongly typed. The benefits of typing for those who don't know is that you can do static code analysis to catch your errors early. It makes your code more self-documenting. You don't need to add comments to explain typing. It makes auto-completion better in IDs. The explicit type definition is also quite a nice way to organise your API, so every type that you have in your API needs to have a type in code. That was a much better way of organising it than we were organising it before in Administrate. The strong typing also allows some really powerful tools to be built against it and to aid the development for it and the development of your API. On the theory, let's have a little go at building an API. We're going to follow roughly the process that we do at Administrate. I'm not going to do any live coding because that would be terrifying. First, to make sure that what's going into the API is right, we want to make sure it's all well planned out and thought out. To do this, we use a tool called GraphQL Voyager. GraphQL Voyager is a tool that allows you to visually explore your GraphQL API as an interactive graph. It's a great tool when you're discussing or designing your data model and it can be connected to your own GraphQL endpoint. Hopefully, we can have a look here. This is going to be our design. We're going to have a root query type here. We're going to have an event and we're going to have a talk. Once your GraphQL schema gets more complicated, you can see this would be a really great way of just having a visual representation and making sure that things are working in the way that you want them to work. The process at Administrate is that we need to have a design in code and review that design before anyone does any work on it. We have our design. Let's look at some code. There are a few options for creating a GraphQL schema in Python. The first is GraphQL Core. This is the core implementation of the GraphQL runtime in Python. You can create a type like this. Here, we're creating our event type. You use the GraphQL object type. It needs a name. It needs a dictionary of fields. Those fields need types themselves. You see we've got our GraphQL scaler types there with a string and list. Then we're just going to resolve the data there. It's going to be a simple case at the moment. This is just going to resolve some static data. Once you have your type, you can then add it to our root query type that we saw from the design. It follows a similar pattern. You need to have a name and some fields on there. The type that we defined before event type, that is going to be a field on this root query type. That's all wrapped in a GraphQL schema. That generates a schema for you. Once you have your schema, how do we make this into an API? Well, we can use the Flask GraphQL package. Basically, just initialise your Flask app. You add a URL rule. Use this GraphQL view object from the Flask GraphQL library. Then you pass in your schema. That's it. That's as easy as it is to get going. That's kind of messy, isn't it? That's going to get messy really quickly. We can do better than that. Next, in the ring, we've got Graphene. Graphene is the most popular abstraction of the GraphQL core implementations by GitHub Stars. Here's the reason why. It allows you to write GraphQL schema in Python in a much more Pythonic way. Here we've got a Python class. We've got some static variables on it. For the slightly more harder to... Well, if it's a scalar field, then Graphene will just resolve that data type for you. But if you need to resolve something a little bit more complicated, you need to define a resolver on that class. You can see we're just resolving our talks endpoint there. Once you have those, it's a similar pattern again. You need to add those to your root query type, which basically just collects all the types in your API. Then you generate a schema by using Graphene.schema. Once you have your schema object, just pass that straight back into the Flask and the Flask GraphQL package there. We decided not to use Graphene at Administrate because we like to make things hard for ourselves. We decided to roll our own. GraphQLIs is the name of the library that we are writing to help us with our GraphQL API. Why? Why do we do this? Graphene allows us to write GraphQL schema in a more pythonic way, but we asked ourselves why should you have to write GraphQL schema at all? We took the inspiration from Java's implementation of GraphQL and Java being strongly typed allowed schema to be built through introspection. You pass your types into that and it generates a schema for you. We found this is possible to do is type annotations. Type annotation isn't real typing but it does allow you to pass extra information when you create your Python classes. We consume those type annotations by introspecting the types that we declare and then we build a GraphQL schema for you. The benefits of this are that our developers don't have to learn any new paradigms. You don't have to learn Graphene or anything like that. We just write pure Python and that's it. The cognitive load is much lower when you're doing this. It also makes the implementation easy to change out. When the new GraphQL comes out in a couple of years, we'll just be able to swap that out and use the classes that we've already written. What does this look like? It looks like a couple of normal Python classes. You just have to make sure you've got your type annotations there. The methods on those classes are going to be the fields of those types and then executing those methods is going to be the way that we resolve the data. Once you have that, you need to add them to your query type again. It's the same kind of pattern. With GraphQL eyes, you need to initialise a registry and then run the query type through the registry and then generate a schema. That GraphQL schema object is from the GraphQL core library. Once you have your schema, you can just use that in your flask and GraphQL package again. Can you use it not yet? Sorry about that. It's a real bait and switch. It turns out open sourcing stuff is hard and we want to do it right. If you are interested, just follow me on Twitter. I'll post when it's released. Or you can sign up. We've got a type form there. We'll send you an email if you prefer that mode of communication. Sorry about that. In the meantime, the stuff that we're relying on heavily is the inspect and typing modules of the Python standard library. We use functions like signature and get members from inspect to help us with their introspection. If you have not already seen the typing module in the standard library, it's worth looking at too. It provides you with all the Python types to do annotations with and it's got some useful helper functions like get type hints to help you do type processing on receiving annotated Python objects. We have got our design. We've written some code. What's next? I'm not sure that it's working. I'd like to introduce another tool to you if you've not already seen this. It's called Graphical. Graphical is basically an in-browser IDE for exploring your... Uh-oh. It's basically an in-browser IDE for exploring your Graphical API. What we can do here is actually some... There's a little bit of live coding, I suppose. You can start your... You can start your query there. You can see that it's got some nice auto-completion features. That's going to be a one. Once you've created your query, you can run it and you can get the response from your Graphical API right in the browser there. It's a really powerful tool for development. It gives you great visibility over what's happening when you're building your API. Another thing that I didn't mention before is that because GraphQL is strongly typed, it means that it can generate your documentation for you. In Graphical, here, you can see that we've got the documentation for our little toy API here. You can drill down into the types and right down into the scale of types there. It makes it really easy for keeping your API documentation up to date. API first, given the game away. Not all these tools help us do at Administrate. It helps us develop with API first in mind. For those of you who don't know what API first is, well, it's fairly self-explanatory. But it sort of means that we develop our product as well. Our product is our API. This is the thing we think about first. This is the thing that's most important to us. You might say that your product is your web interface or your mobile application. But for us at Administrate, we're building a platform which means that we need our customers and third parties to be able to integrate with us really easily. So our API is our product. What does that mean in practice? Well, we need to have some upfront design. As you can see, we did that with Graphical Voyager. It just means that we are able to decide that our API is going to work before anything is built on it. We also release publicly and regularly. Well, quite regularly. We're moving towards a continuous delivery type situation, but at the moment it's every week. When we release, there's nothing feature flagged in our API. When a field goes in the API and it's released, it's public to the world. This is a really nice constraint to focus our attention on delivering good value incrementally every week. Of course, there are times when we need to deprecate something. Stuff does go wrong. But Graphical has got us covered there. Deprecation is built into the schema. Also, the benefit of there being one endpoint means that every request goes through your Graphical endpoint, which means that you can track what types are being requested for and even who's requesting them. There's going to be no more pain about wondering if deprecating a field is going to be a real problem for somebody if you can see that it's not been used for the last six months. Next up, dogfooding. Dogfooding is the process of using your own product. We consume our own API to build our own features. This is really great for understanding the developer developers of pains and the developer experience when developing against your product. If there's a problem with your developer process, if you're using your own API, you're going to be the first to spot it. Documentation. We had a real pain with our documentation being a different process from our code process. Now the two are completely merged together. Documentation changes can follow the same process as code review. It's much harder for it to get out of date than before. Let's go back to the problems we were having. Maintainability. There's one API version. Our documentation is easy to keep updated. The coupling of the front end and the back end, that's completely separate now because the front end needs to know what data it's going to get. A cognitive load is less. We're just writing pure Python classes and not enough thought was given to our API. That's the first thing we think about when we wake up and the last thing we think about when we go to sleep. I think that's about it. Thank you very much for listening. Any questions? Thank you for the talk. You said that documentation is easier with GraphQL. Is that just because you generated it from the schema? It takes a guess that it generates it from the schema and it can tell you what types are in it. But there's also the ability to add your own documentation which I didn't show there. You can provide documentation as you're writing the types as well. GraphQL takes you part of the way and then you can add your own as well. Thanks for the talk. Could you talk about how error handling works in GraphQL? If you add permission errors, getting a particular object or the entire type that's wrong or something like that. You sort of just get a runtime error and it's up to you to handle them. You can handle the errors yourself. It is just whatever the error is from GraphQL core. It'll be probably pretty nasty if you've not handled it properly. I know next to nothing about GraphQL, but you've shown this flask application that takes the GraphQL Cree and runs it against something somehow. But how does it do that? Does it do it for like joining to an SQL data store? Yeah, absolutely. Sorry, I maybe wasn't clear on that one. So when you define in the types that we saw there were resolver functions and maybe we can go back to them. In your resolvers we were just as the example say for instance here. This was just returning some static data but that could easily just be a call off to your SQL Alchemy database with the corresponding type. So say you have a SQL Alchemy model event and that has a name on it, that resolver function there could just be a call to get that particular So I just do the mapping myself in this. Okay, thanks. Hi, hello. So you say before that in case of deprecation GraphQL got you covered since I don't know nothing about GraphQL except your talk. So can you elaborate on that please? The library like Graphine allows you to mark things as deprecated and when the so there's a few front end libraries as well which I didn't get into but they're able to introspect your GraphQL schema and they can see which things are deprecated and then in your documentation or your developer portal you'd be able to see these things come up as deprecated. So it's supported by the Python libraries that we have available to us so you can sort of mark them as deprecated and also I mean there's this yeah. Thanks. Cheers. Hi, thanks for the talk. Maybe it was cut out because you didn't want the talk to be very lengthy but did you have any issues with pagination when you're Yes. You have to write you have to do it yourself but it's it's just something you need to implement yourself think I mean same with same with sort of pure rest of sorry I can't be Yeah, very interesting talk, thank you. About documentation again I wonder in the world of GraphQL where would you store some recommendation about metadata in terms of fields description even more per object table description so forth you could still store it in like the code level in the declaration of the wrap so forth or could you go more in the backhand side database in the kind of traditional catalogue so forth. So what so the library here that you so basically this documentation is generated from an introspection of the schema that you've declared so every time you make a change to the schema in the backhand the library in the front end will introspect the schema and the documentation will be updated automatically so you don't need to store it in the database documentation can be added to your types I think in GraphQL anyway I think you can provide there's sort of a description flag that you can add here and so some of it will be in your code and then some of it is automatically generated by whatever is introspecting your schema Is that interesting to answer your question? Yeah kind of Sorry Thank you for your talk How do you deal with some form of like either like malicious queries like how do you deal with somebody basically requesting a full database dump or sending you tons of queries that cause a lot of load on the backhand or on your data store? Yeah I guess we would try and deal with well we would try and deal with stuff like that maybe before it ever got to GraphQL so we try and maybe deal with that in our load balancer or Nginx or something like that so ideally we would try not to let that stuff get to a point where it was executed so you can add custom stuff because it's one end point you can add a bunch of middleware to do whatever kind of security checks that you want to do on requests coming in so it's it's up to you I guess Thank you Hi you said you were moving towards a continuous deployment sort of world at the moment we do this we also use GraphQL and one of the issues that we're seeing in the future is a place where our clients and our server have different ideas of what the schema is I was just wondering if you had any comments on that any tooling that you knew about or any way of checking those things before the issues happened That's a good question and I guess we've not quite got there yet I think there are things like the relay specification which sort of allow automatic communication between the front end and the back end that's a React library sort of out of the scope of my expertise but yeah I guess you I'm not sure how you deal with that we've not quite got there yet but let me know if you come up with anything good because it sounds like that's going to be a problem we'll have in the future Maybe one last one have you explored doing this in Node instead of Python because I think there's like a lot of support in JavaScript and what was the pros and cons So yeah original GraphQL implementation was in JavaScript and that's maintained by Facebook the reason that we're doing it in Python is a long and sad story about how we try to write something in Java and nobody wanted to do that so it was out of necessity that we sort of ported it to Python which is sort of the main language that we're using at Administrate but we find that makes people a lot happier so yes, developer happiness is the main reason for that Okay so thank you again