 Good afternoon, everyone, and welcome back. We are having a speech today with James Blair, your speech on REST APIs. So could you give him a warm welcome? Thanks. I'm here to talk to you about console apps and why I'm so excited about them as well as REST APIs. And as was recently pointed out to me, they are not the same thing. We'll be covering both of them in this talk. So I'm going to do that by talking to you a little bit about a program that I recently called GERDI. And to give you a little bit of background on that, I work for a Silicon Valley startup running the developer infrastructure for the OpenStack project. And as part of that developer workflow, as you can see, it's illustrated here and it's exceedingly simple because I drew it that way. And also in the middle of the screen, because I drew it that way, is a tool called GARET. And GARET is a standalone code review tool written by Google for the Android Open Source project. And it looks like this. This is a screenshot of GARET. This is not actually a screenshot of GARET since GARET is a web application. And I don't know if you've figured out what's going on here, but this is decidedly a text-based presentation. But if you've ever used GARET, you would probably agree that this is pretty much what it looked like, especially if you used it when we first started using it. Honestly, we kind of had to really try to love it. And in GARET's own authors admitted that UI design wasn't really their cup of tea. They were more back-end folks. So we made some changes. So this is what GARET looks like now. These were actually so well-received that if you install any GARET, they pretty much all look like this. We added the open stack colors, and they said, wow, those are great colors. We'll use them. So we were able to upstream some somewhat minor changes, some aesthetic changes, things like that. But we were never really able to upstream some of the larger changes that I wanted to make, changes around workflow and how developers actually use it. But then GARET continued to evolve, and they produced this REST API. And it turns out it was rather fully featured. I took a look at it and realized that almost everything that you could do in the GARET web interface, you could do with their REST API. And so I thought, well, maybe I'd have a chance to address some of these longer-term, larger concerns in GARET without having to try to convince them to change their application in ways that they weren't prepared to. So I had sort of four big reasons that I started working on GERET. And one is around the workflow. I'm one of the more active reviewers in the open stack project. Bizarrely, I don't actually review any open stack code, but fortunately that doesn't matter. And I always wanted a workflow that was a little more akin to email. I really like email. This is a screenshot of my email client. This is actually a screenshot of my email client, because my email client is in Emacs and it's all text-mode-y. And that sort of workflow, like the zero inbox workflow, where you've got your messages going into an inbox and you try to dispose of them all in one way or another, whether that's answering them, deleting them, something like that. That's a workflow, a way of surfacing the emails that you need to read. And that's what I wanted to get out of GERET. These are changes that I haven't reviewed yet. I need to do something with them. Review them, ignore them, but do something. And so I wanted to build that kind of workflow and it was rather hard to do that with GERET's standard interface. And the reason why I keep comparing this to email and also news groups, I should say, my email reader is also a news reader, so I kind of think they're the same thing. And I don't understand when people talk about them differently. But we have all of these tools for dealing with massive amounts of incoming data, tasks that need to be done, things like that. There's a lot of tooling and processes around that with mail and news, it's very mature. And I just wanted to be able to apply that kind of thinking to code review as well. Another reason that I thought that this was important to write is some of us, for some reason, end up spending a lot of time traveling around talking at conferences or something. And so if... Yeah. Well, so yeah, if this is where you live, or even better, if this is where you live... That's a good thing. Then... I have not achieved your level yet, Bradley, I'm sorry. So if you're traveling around all the time, then Gertie is for you. I wanted to be able to have a system that we could do code reviews offline. We can write code online. We can write... Sorry, offline. We can write terrible code sitting on a plane and I don't know... Yeah, I usually do, and land and push it up and make all of our friends review it. And meanwhile, they get out of reviewing code because they can't do it, they're on a plane. So I wanted to remove that excuse. I mean, I wanted to allow everyone to be able to participate equally in all of the processes. And so I wanted to make sure this thing worked offline as well as online. So it turns out that being on an airplane and being on a slow part of the Internet are really just two versions of the same thing. One's an extreme, the other one's elsewhere on the continuum. So it's just different amounts of latency. So here's a schematic map of the Internet submarine cables. And if you happen to have a server located in the top left of that and you're located in, I don't know, say the bottom right, then it turns out that actually being able to reduce that latency in some ways is very important. And to my delight, some of the earliest contributions to Gertie's efficiency, as far as that goes, came right from here. So in fact, right there. So to address both of these issues, what I did was I could have written an application that directly used the REST APIs in Garrett. I could have synchronously had it post reviews or whatever. But instead, what I chose to do is to have a local database of all of the changes and then a background process that keeps it in sync. And so if you're online, that's very quick. You can post a review, it's going to immediately send it up, and if it realizes that it's offline, then it just saves it to its database and tries to catch up the next time. So there's a local SQLite database to accomplish that for all of the change metadata. But for the changes themselves, I had the choice of, of course, ask Garrett for a whole bunch of diffs and put them in text columns in the database or something like that. But it turns out Git is already a distributed system. And so I realized that what I could do is just point Gertie at the Git repos that you would normally use for development anyway and have it sort of behind the scenes without changing your working tree or anything. Have it download those changes into your Git repositories and it can then generate its own diffs on the fly for whatever combination of patch sets that you wanted to have it do. As a side effect, that meant that suddenly we actually get an opportunity to do things in Gertie that we would never be able to do in Garrett, such as bind a keystroke to check out this change into my repo or cherry pick this change onto the top of the current branch or something like that. Because while it's great to review code online and leave comments, that's an important part of it, occasionally you want to check something out and grep to make sure that somebody didn't miss something or actually run the change assuming that they didn't put a root kit in it or something. So just the ability to tell Gertie to check this thing out locally and do something with it is something that I think is really neat that we would not have been able to do in the web application. So those are all good reasons, I think, to write an application like this in the first place, but why did I make it a console application? So I think code review is text, or sorry, code is text. And over the past 30 years we've been developing some really fabulous tools for dealing with text, including the terminal. And I think that if you're dealing with that kind of information you should use the best tools for the job. So when I was a wee little lad, a company called Digital Equipment Corporation which should never ever be confused with the company that I work for currently, released something called the VT100, and to this day it is the terminal that terminal emulators are emulating. But fortunately things have come along a little bit since then. It's not the same as it was 30 years ago. We have, of course, mouse support and colors. In fact, quite a lot of colors. Honestly, I didn't even realize this when I first started working on this. 256 color terminals are pretty common now. Full RGB terminals exist as well, and frankly I didn't even realize that until I started writing the program to display these slides. So there, in fact, terminals are so sophisticated now. You can, it's perfectly appropriate, I think, to use them to show off your holiday snapshots. So here we are above Queenstown. Here's the Maraki boulders, Lake Wakatipu, the Kauora River. I mean, look at that color, right? That's the color of it, right? The Rootburn track, which is amazing. Cathedral Cove. Another, this is up here on the North Island, of course. This is another shot of Cathedral Cove. I'm there on the left. And Lengahangi on Waitangi Day. And Bethel's Beach, which is not too far from here, and I highly recommend it, though it's not generally that warm. So clearly this has gotten off track a bit. So let's talk about something that's actually in the title of the talk for a bit. Rest APIs. So an API, this is what Wikipedia has to say about it. I think we all think we know what an API is. It's some stuff that helps you do some stuff, which is sort of the problem with it. It's a little vague. It's maybe it's a library, maybe it's a protocol, maybe it helps you interface with a running process, a data storage system, a physical device. It's some stuff that helps you do some stuff. Maybe we can get a little clearer with what rest is. Fortunately, rest was defined in somebody's, a dissertation from a guy named Roy Fielding. And so he says that you need these things in order to have a restful application. You need to have a client-server relationship so that they can be developed independently. That's clearly germane to what we're talking about here. Stateless. So you should not have client context stored on the server and used between requests, which doesn't mean you can't have any state whatsoever. It's okay to maybe store it in a database and have the client refer to that state or something like that. There are ways around issues there. Cacheable and layered system, those both kind of relate to the idea that let's say you're building the worldwide web, you should expect to have proxies and other intermediary systems between your client and your server and they should be able to fit into this thing and understand enough about what's going on to do their job as far as caching and whatnot goes. Code on demand is an optional requirement of the REST API. Basically, as far as I can tell, that's there to sort of account for JavaScript. The idea is that your server can give code to the client that it can run. This is not required for a REST API or an application to be restful, but it can be there. If the system doesn't support it, it should gracefully fall back to not needing it. Then things should have a uniform interface which gets its own slide. That includes identification of resources. Basically, everything should be uniquely identified in some way. Typically, this is through a URI. The manipulation of those resources should be changed by changing their representation. If you fetch a resource and you change something about it and you put it back, that's how you manipulate something described by a REST API. Self-descriptive messages. Again, just sort of falling back on the idea of the worldwide web. This basically means things should have mind headers so that the client knows what to do with whatever kind of response it gets. Then there's this thing called hypermedia as the engine of application state, which basically means that all of the state transitions should be described through hypermedia, which doesn't necessarily mean hypertext or HTML or something like that, but it just means it's the idea that say you had a cloud API that describes a server or something, it should tell you all of the things that you can do to that server using hyper-references to other locations. The idea there is that clients should really only need to be able to understand the structure and then they can be dynamically extended to perform new actions. If you've been following along, you might be thinking about the REST APIs that you've seen and which of these things I've talked about they don't have. Honestly, I put this one at the top of the list. This seems extremely rare to me. If you look at the whole list, here's what you need to be a REST API, which curiously has one X missing from its optional requirement. Then going back to say you want to write an application like GERDI over on the right, that's what you need to make a third-party client. You need a client server. You need to be able to identify resources. Everything else that you can get on that list is helpful. Honestly, if you're writing a REST API, you should try to make it as much of a REST API as possible. It's a good idea and it's going to help people using your API. From my perspective, this is the bare minimum that you need. In fact, there's one thing that you don't need code on demand. If Garrett required running JavaScript in order to do something, I would not have been able to write GERDI. That's a strange thing to do. Enough about that except to say that if you want to get some beer and talk about whether there ever has been or ever will be a RESTful application other than the World Wide Web itself, I'd be happy to engage in that conversation with you. Going back to what an API is, one of the things that the Cal said that an API could be is a protocol. This is what Wikipedia has to say about protocols. When you think about it, actually a lot of REST APIs are very little bit more than just a thin layer on top of a communications protocol, which is great. There have been a lot of good protocols throughout history. I'm going to highlight a couple that I really like. This one is being picked on a lot lately, but I really like it anyway. It's IMAP. You can type text into it. You can tell that to it or use open SSLS client or whatever. It's got tagged responses so you can do asynchronous things and it'll tell you which thing you're responding to. It's got lots of annoying parentheses, which is unfortunate. It's not perfect, but it's interesting. SNTP, I kind of like this too. It's obviously very successful. It may not be as successful at what we want it to do as we would like, but it is. This is something you can tell that to the port and you can type help and it'll tell you what to do, though I'm told that can occasionally be problematic. Let's look at HTTP and JSON APIs. This is sort of what people are doing now for protocols. It still almost as easy to use. You can't tell that to the port and type nice neat commands, but hey, you can. It's a lot of typing. It's a lot of typing. You can use these curl commands, which make it a little nicer until you're starting to tell curl which extra headers and to attach the binary data and stuff like that. It gets a little squirrely. It's not quite as easy to use, but it's not bad. What it lacks in sort of online debugability, it makes up for in parsability because JSON is pretty awesome. To be honest, if IMAP returned JSON blobs or something, that would be swell. That would be way better than parsing all those friends. We're getting to an interesting place though with things like web sockets. A lot of the reasoning for creating these APIs using HTTP the way that they have been so far has to do with the way web requests were made from client applications. Now that we have things like web sockets and a lot more capabilities in JavaScript applications, I think we can be getting to a point where instead of making HTTP requests and getting JSON blobs back all the time, we could actually be streaming interesting data back and forth on web sockets, probably still in JSON. But imagine that instead of doing get or post requests all the time, you're actually just sending little commands to a server. It sends a JSON blob back. You could do that inside of a JavaScript application. You could do that with a client that isn't related to a web browser at all. I think we're actually getting to an interesting place as far as that goes. Toolkits. We're talking about console applications. When I was looking at how to write KERDI, I like writing in Python. I looked at a couple of different toolkits that I could use in Python. There's NUT, which is venerable. It's been around for quite a while. It's a little thing when you install Red Hat, it pops up little dialogues that ask you questions. There's apparently a Python library called Snack to help make that easier. I honestly don't know how well this is maintained at this point, so be aware of that. There's a neat thing called inpy screen, which I might have even used if I found it when I first started working on KERDI, but it's actually very new. And it's a really great toolkit for... I want to present a form to a user and have them enter in some data and then retrieve that data from it. So if that's sort of what you're looking for in the console application that you're writing, you might want to check that out. There's something called curtsies, which is far simpler than that. Actually, it basically aims to be a nice way of dealing with a terminal, but maybe not focused on form inputs or complicated applications or things like that. What I ended up using is a system called Erwid, which is more like an application framework. So if you're used to writing a GTK or a Qt application or something like that, you might find Erwid somewhat familiar. I did it first. So it's really designed for this sort of long-term event loop kind of application where it's like you build things out of widgets, you display them, you get some input, then you do something else. So this is sort of the Hello World application in Erwid. The interesting bit is actually at the bottom and you create a text widget with some text in it. You put it in some kind of a container and you tell the main loop to display that container. And then that function at the top gets passed keystrokes. And so if you hit Q, it exits. If you hit anything else, then it just changes the text in your text widget to be something else. And so with that introduction, I'm going to show you a quick little demo of Gertie. And there we go. So sorry, the type is a little bit smaller on this, but I'm going to squeeze some stuff in later screens. So try to bear with me. The words aren't important. So when you start out with Gertie, it's got this list of projects that you're subscribed to. So think about email again. And it tells you how many open changes there are for every project and how many unreviewed changes there are. As you can see, I've not been doing very well on reviews here lately. I've apparently been flying around or something and writing this talk. But you can see this is equivalent to these are how many messages you have in this folder or whatever. And the terrifying thing about Gertie is it tells you actually how far behind you are. So if you click on that, it shows you all of the open reviews. And this is something that I was working on right before I left and not many people have seen it yet. But look, threading, like email, right? So you can see all of the outstanding changes that you have to review for this project. And you can click on one of those and it brings up a change screen, which is in spirit not that different from the screenshot of Garrett earlier. But notably, this is text and that's on the web. So it shows you the information about the change. It's got a table of the reviews, so sorry, information about change, people who've reviewed it, the commit message, all of the results from our extensive test system. And down here we've got comments that people have left on the change in general. Right here it's got the files that have changed. There's only one. I maybe didn't pick the best thing to show you. And with a key you can have it show you a diff between the two. So here's the old and new side and you can hit enter and leave comments, et cetera. And let's see what else. So there's, you can have, pop up, whoops, sorry, pop up dialogs. And the mouse works, right? So if I click on the diff button, it'll go back there. So basically all the modern conveniences, these are also links actually. So if I go over here, these colors aren't great. But if I click on any of these, it'll open a web browser to that, which obviously doesn't work so well on the plane, but there's only so much data that you want to download. So anyway, that's what Gertie looks like and that's briefly what the sorts of things that you can do with Erwid. So going back to the slides. Oh, sorry. So if you think about the screen structure I just showed you, there's, you can imagine building that up out of boxes, right? Every system has a box model, whether it's CSS or tech or Erwid or GTK or whatever, everybody has a box model. The only problem is that everybody's box model is completely different. And everything that you learn about how to work with one doesn't translate to any of the others and Erwid is no exception. But the pile is your typical vertical container. The columns are your sort of horizontal container. The list box is your scrollable vertical container. There is no scrollable horizontal container because I guess people don't want to do that in terminals. Seems reasonable to me. And so forth. So that's sort of how you build things up out of boxes. But again, the behavior of those seems straightforward at first, but when you get into the nitty-gritty, it turns out that you're probably going to have to learn some things about the specific behavior of some of these widgets in Erwid. But fortunately you can, if anything's not to your liking, it's actually pretty easy to change. So in writing GERDI, I ended up subclassing a lot of the widgets in Erwid. So for instance, if I wanted a button that did not expand to fill the width of its container, this is all you have to do to do that. So it's just a button that reports its dimensions differently when asked. So it's pretty easy to do that sort of thing. There's one more consideration that I think is important when you're working on a console application. And that's visual design. So if you think about web design right now, we're kind of coming into this phase where there's lots of boxes and solid colors. Like if you think about the material design guidelines from Google and what those look like, that's sort of what things are looking at right now. In the past there's been boxes with lines around things. That's obviously a perennial favorite in desktop applications. These things sort of change over time. They seem to not change very often in console applications, which is nice and refreshing. Mostly our tools for sort of creating a visual layout to sort of indicate the relationship of information to each other. They're kind of limited to color and space, which is nice. It's part of actually why I enjoy using console applications. So if you look at this, there's no graphic structure here. There are no lines or anything to draw your attention or separate different areas of the screen. What I'm doing there is I'm changing color and using positioning to indicate that. So there's clearly a box of information at the top with key value pairs. There's a table underneath that. There's a list of some files. There are buttons. There's comments from people. All of that's communicated via color and space. Once you use this a couple of times, you pick up on what the patterns are and how to narrow in on what you're looking for fairly quickly. If you look at the same thing without any color, it's rather difficult to navigate. So one of the things I've spent a good deal of time on in GERDI is working on those color and space relationships, and it's fortunately quite enjoyable. So to sort of summarize, console apps are cool. They're fast, efficient, customizable, and modern. That's the link where you can find GERDI, and that's a link to the Erwid project where it actually has great documentation, by the way. So you can find all of that there. And with that, that's all I have. And there are some questions. What are your slides written in? They are written in a program called Presenti, which you can also find on PiPi. You write the text in restructured text. It supports cross-pades, PANs, cut transitions. It'll shell out to figlet and calcay for you. If you use dot-dot-image-colon, which, incidentally, if you're using a restructured text system to do web-based slides, often those use the image command in restructured text to include an image. If you do the same thing in Presenti, it'll shell out to jpeg2a to do the ASCII art conversion for you. Any other questions? So you're obviously an Emacs user because you had good news on a slide. I'm curious if you investigated writing it as Emacs, because it seems like Emacs is the place where console people have gone to hide mostly. Certainly I have. Okay, roughly 50% of the console people have gone to hide in Emacs. And as an Emacs user, did you think about doing this? It was just because we all hate Emacs most, even those who love Emacs? Or was there another reason? This is more of a personal reason. I actually would have loved to spend the time to do it in Emacs's list, because I enjoy writing a list. But I'm not good enough or proficient enough at it to do all of the things that I knew that I would need to do with GERD. Not only the network stuff, but the local data storage things shelling out to get all of... There's a lot there. And I wanted to stop using the web application as soon as possible. I guess one of the really great things about the command line is the key bindings as well. Is that supported in Erwitt? Or is it hard to implement? Yeah, so I made customizable key bindings in Erwitt. In Erwitt itself, it's pretty easy to do. You just... You supply a function, and whenever a key is hit, it gets passed into that function, right? And then what you do with that is up to you. So right now what I have is... I just have a table of key bindings to commands that you can change in the GERD config file. And so you can make your own customizable key bindings for that. I need to make it a little more sophisticated so that you can have two-level things so that you can hit Ctrl-X, Ctrl-something, or colon-something, whatever your preference may be. And actually, it might be good to do that in some sort of generic way that other Erwitt apps can use. I actually had two questions. The first was, at the beginning of your slides, you mentioned being able to work offline, working from airplanes and things like that. And I know, barring the conversations we were having earlier about having Wi-Fi on flights, is there any capability of using this to do some things you might not be able to do with just the Garrett Web Interface? And the second one is... I know we all like black background down white text, but is there also a light option? There is a light option. I think it's dash-L or dash-light or something. I have no idea. I wrote it, but I didn't produce it. And Presentee has that, too, by the way. As for the first question, there... Well, I mean, there's a lot of things that we've done in GERD that you can't use in doing the regular web application. Just as far as the things related to offline use, despite being online, there's the latency issue, right, that you're not waiting for the round-trip HTTP traffic or whatever. And because... If you go back to, like, user interface guidelines, 200 milliseconds is about how long it... If you press a button and something doesn't happen within 200 milliseconds, it starts to feel like you didn't actually press the button. So having... And occasionally, web apps can't actually accommodate that because of all the overhead of doing that back and forth, especially depending on where you are. So what we have is an application that's always responsive, regardless of what the network is like. And so it's actually... It feels a lot better as a user because if I'm reviewing changes all day, the amount of time that I spend waiting for the web app to do something... And I'd work around that by doing things like opening a lot of tabs. Like, here's a list of changes. Okay, open them all in a tab so that they're all loading in the background, which is kind of silly. So here's an application that's always responsive. Or at least if it's not responsive, it's because I can't write SQL queries. When you were thinking about the things that you wanted to accomplish within the console client, was there anything that you couldn't make work with the constraints that you had to significantly change from your original idea? I think all of the big things were in there. Just the usual sort of, wow, this thing ended up being a lot harder than I thought it should be. The hard things ended up being easy and the easy things ended up being hard. The fact that all of the diffs, they show up as one continuous stream, so all of the different files just show up one after another in the diff screen, which has been the most requested feature from the Garrett web interface forever, and it doesn't have that. So apparently that's hard to do in the web interface, but it would have been hard to do in Gritty, and that ended up being easy the way that I did it. So just that sort of thing, and the occasional fighting with, I think that if I put these two widgets into a container, they should behave this way, but it turns out that they don't, and you have to go debug that sort of thing. Any major concessions on the big features. How do you deal with conflicts after being offline for a long period of time? Log them to an error log that nobody reads, and hope it works out. So most of this is append only, actually, so it works out pretty nice, right? You can never delete your comments in Garrett at all, so generally all you're doing is you're leaving comments on things, and you're leaving more comments in addition to that, and all of the comments are directed toward a specific patch set, so if people upload new revisions of it, even if that happens while you're offline, your comments are going to go to the patch set that you actually reviewed, so that works out. The only thing that might substantially change is if you leave a vote on a certain patch set and it's been updated, then your vote won't be counted anymore because it's not relevant. So the good thing about that is that you're never going to approve a change while you're offline, and then accidentally have it approve something other than what you really approved, right? Because that's an invalid condition. It'll end up that you will have left a comment on an old patch set, and you'll just have to go back and look at the new patch set again once it's there. There are some things that we might want to do. We've had folks say that if I leave a positive review on a change and somebody else leaves a negative review while I wasn't looking, I would like to find out about that because maybe they're pointing out something that I missed and I'm going to look stupid, but those are actually conditions that'll be easy to detect, and once we stop doing things like logging important information in a log that nobody reads and instead surfacing that up to the user interface on the screen, that'll actually be pretty easy to do. Very good. Well, that's all we have time for here at the moment, but could you please welcome... Oh, before that, please take this. So thank you from everyone on the team. Thank you. And please thank James for his speak. He's a lovely pair, isn't he?