 Yeah, so it's great to see everybody here and it's nice to be in Portland where I live and I love the city, so I'm glad you guys get to experience it. This talk, how I envision this talk, is giving some insights from real engineers in a production systems that Safari Books Online has deployed about various REST, our use of various web services libraries to build Django APIs and it's sort of an evolution because as we started kind of with one thing and then we tried something else, now we're doing this other thing and it's changed over time and we have a variety of libraries in production right now. So we'll kind of talk through three or four of these things and what I'm going to do is give you some real words from different engineers, not just me and not necessarily like Safari's position but what we've experienced, like the good things and some of the bad. But at first I had a question, are any of you creators or authors of the Django web services library? Okay, good, good. So tone down the stress a little bit. So we will be talking about some cons and it's not a judgment necessarily. All right, so quick introductions before I get into it. So a little bit about you. I think you might be for this talk. You're familiar with Django and REST. You want to hear about some real world uses of Django like I just described. Django web services libraries to write APIs. And quickly about me, at Safari I'm a senior software engineer. I write front-end code and I write some back-end code and mostly back-end right now. I also like to write text editor plugins for some reason, I don't know what it is for different editors. So there you go. And a little bit about Safari since they sent me here and I got to use this nice PowerPoint template to make this presentation. You know, sort of, you might know us by the longer form, Safari books online. We're kind of thrown around Safari now, it's a little shorter. And just sort of our catchphrase, you know, we're bringing the best books and courses to life. So we want to give you guys access to videos, books, whatever it is you need to learn. Come and look at our site and learn. But as far as I learned, if you need to jump out, I want you to leave with something from this talk. So I will tell you the end of the talk before we do the entire talk. And that is that we use Django REST framework for our new APIs. And we'll get into some reasons why. There's a long list of things we like about it. But if you need to leave with something, and I'm sure you've already heard this many times, this probably isn't new, but Django REST framework is good. So let's start at the beginning, right? And this is in some ways like the beginning, maybe it's the beginning of your prototype. We can also look at it as before we did, before we used a library for doing our web services. We used some plain views, and I've also used some prototypes as well as I mentioned. So let's look at some pros of using just vanilla Django views for your API. We won't spend a lot of time on this, but as you may know or may not know, they're good for returning snippets of rendered HTML from a template, or something really simple like you want to just expose a read-only API for some JSON or XML data. So all good. They give you complete control over the structure and output of your data, and you can do whatever you want in your view function or a class-based view to get this done. And a little helper I find nice, if I'm doing a prototype and I want to use vanilla Django views is JSON view. It's actually called Django JSON view, and basically it's this little decorator you put on top of your view, and it helps manage the serializing, deserializing Python data structures into JSON. It's just like candy, you know, cut down some of that boilerplate. But of course, there are some cons, like why would we have all these library frameworks if there weren't some nasty things about using vanilla views. You wind up with a lot of boilerplate, sure, if you've written, if you've done this, you know, you wind up with a lot of stuff. A lot of it is around validating input data over the line. It's a lot about setting up the deserialization and deserialization for multiple content types and so forth. Having total control can lead to inconsistency of your responses across and within projects. So maybe you have multiple apps that have APIs. You might have developers doing stuff differently in different apps, and suddenly, you know, the errors you get back when there's an error are slightly different structure, and the clients have to code differently. Blah, blah, blah. It's just not that fun. And you might end up with multiple views. Well, you will end up with multiple views for different content types that help manage, you know, I want my stuff back in XML or JSON. It's not so easy if it's plain Django view. You might have to do some extra stuff. All right. You've probably already know all this. I want to look at Piston mostly because it's what we used, I think, as our first real-like web services library. Most everybody knows, you know, Piston isn't the greatest choice at this point, not to malign Piston, but we'll talk about it because this is a history, this is an evolution, and so we'll talk about pros of Piston. There's probably more than this. This is not very nice, right? But the major pro, if you were to delete use it today, is you can get a simple API up and running. Probably any library can do that, but there you go. I have a lot of cons, so keep this short. So we ran some problems with Piston. Another one problem, of course, if you use Piston, it's really not being maintained. It's kind of a dead library at this point. The last release on PyPy was in, like, 2011. So, you know, we perceive that the community is drying up. We still have APIs that were written in Piston, but we're probably going to migrate them over to Django REST framework. I think we have migrated some over to TastyPy, and we might end up migrating those to DRF as well. So, real problems, like, one of the real problems we had is there's not an explicit serializer really for your API classes, so it kind of makes it, you know, it's a little bit sticky to manage the, or to mangle the output of the data. So if I want to add in some, you know, dynamically calculated field in my JSON that I'm returning, a lot of that logic winds up inside of this one monolithic API class. And that's just, it doesn't, it's not good for maintainability, you know. This is a problem that actually is in a lot of different libraries, but Piston is one where we found it the first time. And there's no pagination out of the box, so Piston didn't ship with the ability to paginate results that your results set in the API. Turns out that's pretty useful if you are turning lots of, lots of items in a list. And there's more, right? There's other ones, but why should we really spend a lot of time on Piston at this point? I think Piston was a great library when I first started using it. I was like, dang, this is way better than writing vanilla views, you know, this is awesome. And it paved the way, so props to Piston. The next thing we settled on was TastyPy. So what are some pros of TastyPy? People are still thinking, oh, do I use TastyPy? Do I use DRF at this point? So what can TastyPy give you? What did it give us? It gave us pagination out of the box, which was awesome. And then, so it was a little bit, a little bit better, right? There was a couple of improvements over Piston. One of them was it was easier to change the output data we were sending back because TastyPy has this thing, these methods you can override for hydrating and dehydrating, which is another, you know, it's just another, for deserializing your data structures. So that was nice. It was improvement. And we found the resource object that you subclassed to be relatively straightforward in terms of its APIs, you know, interacting with it. We'll talk a little bit more about it, but on the surface, pretty straightforward. And then one of the cool things about TastyPy, which is still cool, is that it will generate a schema based on your API automatically. So if you have a client that can do something with that schema, which is sort of a big if, but if you have one, then that's pretty helpful to do client-side validation. And you're also going to do your server-side validation as well. And it handles setting up the URLs for you, which is kind of a double-edged sword, right? It's like, that's nice. It gets fast. It is not necessarily pythonic, just an automatic URL thing, in my opinion, because I kind of like having, you know, my URLs pretty explicit. But a lot of frameworks do that for you, and Rails does it as well. And these other libraries, like Django REST framework, have a router. Anyway, so a couple cons. We kind of ran into some big problems using TastyPy, and they weren't really things we expected. Safari, a lot of our back-end APIs are like, it's like a pipeline of content, right? So we get a bunch of stuff from publishers, we get lots of material from them. A lot of it is like EPUBs and PDFs. Some of these files are pretty large, even when you chunk them. And TastyPy didn't handle binary types very well. There's an open issue I linked to. I didn't want to just be like, ugh, this didn't work that well. So I put a link in here if you can get access to the PDF version, and now you can check it out. And then I think this is more like, when I'm having to do some custom stuff, yada, yada, yada, to get a file upload working against the API. If you read that issue, you know, there's like, oh, I got this fork, and you can use it, you know? So one of the problems we ran into was like high memory usage on receiving a file upload, because it wasn't doing any kind of streaming, it would just buffer the entire thing into memory, and they were uploading huge objects a lot of times. So another thing was that we ran into was it turned out to be, while the resource object was easy to use, it turned out that it was not so easy to extend lots of parts of TastyPy, in our opinion. And again, this is, multiple engineers give me feedback on this, so it's not like, we all had the same problems, but this was a perceived difficulty. And the same with Piston is, and this I've seen this is, you generally end up with a single API class that's kind of responsible for doing everything, like it does, it handles all the custom serialization, deserialization logic, and a bunch of other, you know, fields that control authentication and stuff. So that is what it is, you know, but it's generally a lot of code in one place responsible for a lot of different things, which causes a smell about it when you have to maintain it. And one of the engineers told me that TastyPy had too much magic, which you can take with a grain of salt, because actually this person used a lot of generator expressions, which, if you've had to maintain, are interesting to debug sometimes. So another one, this is kind of an interesting point, right? So a big thing about Web Services Libraries is, you know, you have all this validation that you need to do, like you need to give, like you want to give the client a schema, so the client can do some validation before it uses any request to send any data over at all. But you also need to do all that same validation, you know, on the server side when you receive data. And Piston, I think, and TastyPy both approached this by using Django's excellent already existing code to do input validation, which is the form code, right? So you create a form and you check your impound data against this form schema and this form acts like a schema, right? So you can say, oh, this is not typed correctly, yada, yada, send it back however you want. But that's kind of weird for us, like, we don't really like using forms to validate, like JSON data coming in from an API. It's not specialized to that purpose. They're kind of specialized to the purpose of handling, like, form inputs, right? So one of the things, like, say you want to output the error messages that the form generated from validation because of validation problems, right? You have to do some funky stuff, like, get in there and kind of get the, get the, there's not a clean API to get, like, simple text out of the form error sometimes. Or maybe there is now, but there was. I think there is actually in 1.7, but a little bit odd. All right, so we were into these problems. We didn't know what to do. There's so many options and then we had a new engineer come to us and we asked him, hey, what do you think about Janger Rust framework? Like, is it cool? And to my knowledge, he told us that he thought it was cool. And because he's a new engineer, I think we want to really look at his input because of new people on the team. The audience, good. So we took a look at Janger Rust framework and turned out it was awesome. It was actually really awesome. So I actually had to break up a list of grid things about Janger Rust framework into two parts because there were so many things we liked about it. And I didn't really want to leave any of these out. I actually leave out a bunch of stuff out. And I put this one at the top. So the ordering of these is based on me. It's sort of arbitrary, right? But the number one thing about Janger Rust framework that I really love that we have found is how well documented it is. Like, you can go to the site today. You don't need to attend to talk about Janger. I mean, you should if you're interested, but you don't need to read anything but the docs to really get up and running and also beyond that to really go deep into understanding how parts of it work. So it's very well done. And the creator is very attentive on Stack Overflow and forum asking questions. It feels good. And, you know, you are everybody's, you know, one of the first things you see about Janger Rust framework is how's this awesome HTPrest API browser? You've probably seen this already, right? But Janger Rust framework, it creates this HTML view that you can access as a human being that shows you information about the API. It's awesome. I don't know, you know, if it's really numbered to you on my list, but it's something that everybody likes when they first start using it, right? But some real stuff. Like, what did we really find useful? Out of the box pagination and this filtering feature it has were really useful to us as we were upgrading some of our old APIs. And we upgraded a bunch of APIs from Tasty Pie to Janger Rust framework. And this came in handy. And as just as an anecdote, not as a bullet point, but as an anecdote, that API, I'm not saying it's Tasty Pie's fault, but there were a lot of pieces of that API that we upgraded that were very poorly performant in memory. And you could look in New Relic, we get the alerts, you know, and you're like, oh, God, spew. It's just crushing the server. So much RAM is being used by this stuff. And we try to fix it. You know, we get somewhere in here and there. That doesn't happen anymore with our Janger Rust framework version. So it's more complex than, oh, we switched to Janger Rust framework. We didn't have any problems anymore. We made some changes based on what we learned wasn't working. But also, it's my perception that performance is a lit better in Janger Rust framework. And so back to the little points. It supports token off out of the box, which is just nice if you want to get up and running with token off. Like, yeah, you just drop it in and it's going to start working. But there's more. There's more. So it is very easy to customize output from your APIs, including resource relationships. So in Rust, we have this thing called the resource, and they relate to each other. And it's easy to set that all up and get it the way you want it with URLs or IDs or whatever with these explicit serializer classes. So the thing about Janger Rust framework is you end up with a bunch of composed objects, right? So you have your thing and it has a serializer. It has a filter class. It's not just one big class. It does everything. It's composed into pieces that make sense, that are separately testable, reusable in different contexts. And one of those things is a serializer. Okay, and here's a cool thing. So server side validation. You're going to have server side validation in different stages. A lot of people will use validation routines in the model classes to do validation. So a great thing about Janger Rust framework is when it's creating something over the API, it will call full clean on your model. So you can stick all your business rule validation into full clean and use it. And Janger Rust framework will use that. You don't have to abstract it into something else and then, you know, use that in your API and call it from your full clean or whatever. You can put it where it sort of belongs and it will get used, which is nice. And sort of bouncing around here, but the community itself is very active, very helpful. We have found, which is very, very important, obviously. And, you know, more about like beauty, right? We perceive it to be elegant and that's important. This is Python. Our perception of beauty in our team is actually, you know, high criteria kind of important thing to us and looking at the code, but also looking at what we got out of it. You know, what does our API code actually look like? We find it to be elegant. We find it to be extensible. And that makes us happy every day. So plus one. And sort of related, your API representation is being pretty concise. And it's, you know, this is, I didn't write this book on myself, but what you end up with is, like I said, it's a bunch of composed objects. So it is concise in the sense that here's this one API. It looks pretty small when I look at the high level view, right? And then I'm going to go down and I'm going to look at the individual composed classes like the filter or whatever. And you're going to expand out into multiple classes. But it's nice to be able to see a concise overview sort of at the top level of the object hierarchy. Class hierarchy, we'll say actually. So, but I am not going to say that it's the best thing on earth. You know, that everything has a couple flaws or whatever. These are flaws. We found some cons. So one thing that we really miss about tasty pie is there's no built in schema generation. So clients can't auto discover that stuff. And we have to maintain our own schema, which is pretty common when you have an API, I guess. But I don't really like it. I don't like having to go in and like, oh, I changed the thing in the API. Oh, right. I got to go in and update the schema file. And yeah, that's fine. We have a client that can read the schema and knows what to do with it and like in-house rest client. So that's the thing. Maybe someone will release some package that does this for us someday. And I won't have to do this anymore. I hope. Then there can still be lots of boilerplate. So interesting thing about this, right? I'm pretty sure there's a bullet point I either removed or put on every single one of these sections that are bullet boilerplate. I think that's just programming like you actually have to write code with no matter what library you choose, right? So take it with a grain of salt. But exposing filtering mechanisms we found can sometimes be convoluted. And I think there's different things going on here, right? As Django Restorimer gives you this thing that is the ability to filter results in the URL. And it kind of feels to me, it feels a lot like your filter parameter you're going to pass in to the Django ORM, right? But it's not really the same. And there's a little bit of cognitive dissonance there, I think. And you kind of have to do some stuff to expose these different types of filtering options within your URLs. So it can be a little convoluted. And somebody informed me that Model to Model Fields are read-only. Which I don't think is going to be a game changer. I think you can, you know, probably by default, yada yada, you do some overrides and you can, we do set related, Fields are related objects, but you have to do some, you have to actually write some code, I think, it's terrible. And some of us like the way that routers are implemented. And if you're not familiar with router, a router is this thing you can use to, like say you have this API code you wrote for a book or books, we'll say. It returns information about books. And you can use a router object to, like, automatically generate some URLs for it. I don't really have a problem with it, but, you know, we're a diverse group of folks. I wanted to give the voice, honor the voice of the engineer who did not actually like that, how they were implemented. So I'm almost out of time here. I think we'll do a quick summary. So sometimes you can get away with playing Django view. I often, when I'm prototyping, we'll use just straight up views. And then if it turns out we're going to do something, like maybe we go back and add it. You do it right with a library. Which I got to stop doing, I think. It might be easier just to start with the library. The Django rest frame is pretty simple to set up. And I want to re-emphasize that I don't think that Piston is bad. And even that, you know, Tasty Pie was bad. Piston got us rolling, you know, it was this library that I started with, like back when I worked at Dark Horse for Wilding, digital publishing. It was great, you know, it was a good start. Sort of paved the way. Tasty Pie was a great improvement over Piston and gave us all these nice features. And, you know, everything depends on something else, right? You can't get up here without starting here. And all these libraries sort of build on what we learn. Django rest framework I think has learned a lot from Tasty Pie. It's learned a lot from Piston and other stuff. I can't speak for the author to say, I know how he figured out, you know, how to do this. But I think we learn. And we are using Django rest framework for new APIs because we think it's great. So that's the takeaway. I am free after the talk to answer any questions you might have about any of these points or to chat. So thank you very much.