 I'm Eric Anderson. If you ever see me come across me on GitHub, I meet Joana A6, and I'm a software engineer at Google. Honestly, I'd be OK if someone asked me a question while I go, but it might be a little easier for mics or things like that. Granted, I can repeat back if we wait to the end, but I'm not really too upset one way or the other. But I do have time for questions. And so I'll be talking about modifying GRPC services over time, where basically you've got your service. Apparently, it's not a fixed thing, and you need to extend it. So as far as intended audience, there's actually a couple different ways you can go about this sort of talk. But I hope that you've toyed at least with GRPC and Protobuf. You know what the Protobuf compiler is to give you the generative code. You have messed with a couple different primitive data types in Protobuf. Those things aren't scary to you in any way. That's where I sort of hope that you're in that sort of that place in order to really understand the talk. But it's, I guess, you'll have to dill if you're not. But also, you've started making your own services. You've started playing with it. And then you start getting questions of should I do this or that, or surely there's a better way to do this. And so you're searching for some best practices and idioms. I need to make some assumptions, though, because there's many, many different ways to use the stuff. So I am going to assume that we're caring about cross-platform, cross-language. And we can't do tricks that only work for some people, for some clients, or something like that. Also, I'm assuming this is GRPC native. I know that using REST with GRPC is definitely a thing. I'm completely good with that. But it's not going to be as much of a focus here. There's actually some other documentation on how to build nice, clean REST APIs with GRPC. Although I am aware of it, so it'll seep through some places. And also, I'm assuming that servers are updated before clients. This might seem completely obvious to you, but if you have a back-end scenario that you're using GRPC to communicate between two servers, you can totally have you update one server before the other. It's actually a client, and it sees some new updated protos and fields and stuff before the other. And so it's not as much for those cases. Granted, you've got some things that are harder in that case, because you can upgrade both ways. But you also are in more control, so some things become easier. But I'm not really talking about that. It's also not the case of someone defining a service, and now there's going to be multiple implementations. If you're doing those sorts of services, you have to have a little bit different view of what is supported by the server. So we're talking about changing something over time. There have to be some constraints. Well, that's actually compatibility with all the people who are currently using your service. But there's lots of different types of compatibility. You can have binary and source compatibility, which relate more to the generated code. And that involves particular languages. Then you have wire compatibility, which would be how are things serialized on the wire? It was doing that earlier as well. And then you've got behavioral compatibility, where everything's good on the wire, everything seems fine. But your application is now, let's say, interpreting something differently than it was before, and that ends up breaking a client. So to just start, we'll take a binary and source just at the same time. And there's actually quite a few languages that Protobuf and GRPC support. And so that means basically all bets are off if you're trying to do something fancy. If you rename or remove something, those totally will break someone. That's not surprised anyone, probably. If you change a method type from one name to another, that's going to change method signatures. That'll totally break people in statically compiled languages. But maybe it wouldn't break broken someone if they were using Python or something like that. Primitive type changes, you're like, oh, this was an N32. I'm now going to make an N64. And you're like, oh, that's fine. They're both integers, but lots of languages will not be happy with you on that, because it'll be an implicit cast or something like that. So that basically leaves us with adding services, methods, messages, and fields, which if you've done much like library API maintenance and stuff isn't surprising, you can basically add stuff. That is the limitation. And so as we sort of go through this, it's like, OK, how do we make use of this knowledge? So let's say we just sort of had this conceptual library service, which is used as an example in some places. So we want to be able to create a book, go ahead and get a book, list stuff, delete stuff, and update the book as well. And so I put in list book here as a string. That's going to be some identifier for the book. It's going to be, let's say, the book name, or it's guaranteed to be unique. And over in list books, if you're not familiar with protobuf, the google.protobuf.empty, that is just a message with nothing in it. So it's the equivalent of a void in some languages. So this is sort of what we're starting with. I'm not saying that this actually works, but this is what we're wanting to do. And we're sort of going from here to make it actually maintainable. And so I introduced some of these best practices. We should plan for the replacement of the service ideally. So right now it's suggested to include the version number in the package name itself. So up at the very top that we have the package. That's what I'm talking about here. So you could include a v1. It's actually not that big of a deal if you forget to do that, because you can have a v2 on the v2. But having a version in the package or in the service name itself can be a good way of separating things whenever you have a breaking API change in the service change in the future. Also, GRBC doesn't let you have multiple arguments or some things like that. It relies very, very heavily on messages and you extending messages. So you want to use messages for your extending point, because that's the intention. If you sort of avoid them, you're going to have some trouble. So when in doubt, create a new message for each RPC method. So you've got foo method, and you're going to have foo method request and foo method response. And you can just go about that just all day long. And that allows you to have a very, very fine scope for adding things in the future. And then you can also feel free to add new services and methods as necessary. You don't have to limit yourself to just a very, very few number of methods. You can add them as they're useful, but at the same time don't have an explosion for every single different combination of possible arguments. This is probably nothing very special. Methods are great, but if you have tons of them, you start drowning in just the accounts that they are. And so here are some of the changes that I made into that earlier sort of conceptual model. I've gone ahead and put a v1 in the package. And then I've created a request arguments for each of these, or request messages for each of these methods. And that means that if in the future list books, now all of a sudden we want to, let's say, have a filter for you to be able to, say, only certain books am I interested in, you have a place to put that. If you didn't do this, it's not the worst thing in the world. You would just need a new method that then has some message where you could specify those things. But adding extra parameters to an RPC just happens all the time. And so you can plan for it a little bit, and then you avoid having tons and tons of methods. That one was sort of rusty. Let's say that we've got something more functional, like computational. And so let's say we've got this nice infrastructure. We're in the infrastructure, and we're providing a clock service. It's really, really advanced. You call it with no arguments. That's the empty. And then you get back a timestamp. This is in some ways fine, but you might instead choose to go ahead and make the request, make the response, message types, and there they would just be on the end. So this seems boilerplate plating. Again, it would have been OK if we forgot. We would have just ended up having a get better time method later. But doing this ahead of time does save us that cost. And really, for the users, it's not that much harder to use this API versus the one we had before. So it's not with too much cost. Oh, hey, hello, people chatting with me. All right, so that gets us binary and source. Now we're going to go ahead and go on to wire compatibility. This one's a little bit more in-depth. So RBCs and gRPC are only distinguished by their name. So that is the package they're in, the service name itself, and then the method name. Anything else than that is implicit. And so that means request and response types are implicit, those names that are used. The cardinality, whether it's a streaming request or a streaming response or just a single request, those things are implicit. That means you could go ahead and add new services and methods because those have a different name and such they won't collide with anything before. It also means you could change a message from one type to another if you were really wanting to. And you could go ahead and add a stream keyword to an existing method. But because of where we were before with APIs, doing those last two will totally break people at compilation time. Now granted, if you can force people to get a compilation error, and then they'll update, but it doesn't work as well. Wire compatibility for protobuf is even a little bit more granted. A lot of people have had familiar, are already familiar with protobuf, so they may need to be aware of some of this. And some of this is slightly different in proto3 than proto2. But message names are implicit. So if you change the message from one name to another, it can act like the same thing, except when that's not the case, like any. And except, I guess metadata would also be a common case where it sort of leaks through. Field names are also implicit. That's why we've got the tags. So you've got the equals of one at the end of your field. Those are also implicit, except when they're not. Converting to and from JSON is pretty common. And FieldMask is maybe something that you might want to use. I'll talk about it later. So I'm saying it is, but then it's also not. We do know that field tags are explicit. You cannot change the tag where it says equals one, equals two, equals three at the end. If you change that, yes. You're totally going to be breaking someone. That should not be a surprise. One interesting thing is some tags are the same on the wire. Sorry, some types are the same on the wire. Int32 is the same as int64. So that means you could actually say, ah, I need some more bits. I'm going to upgrade this to an int64. As far as Protobuf is concerned, that would be fine. But it's risky because of the API and the ABI stuff earlier. So really nothing new. Our limitations were pretty existing. We're really just limited by binary and source compatibility. So I don't know what that same practice is. Oh, yes, we follow the same practices we were talking about before with the binary and source. Now, yes, you've got more options here. If you don't care about that binary and source, you could do more fiddly bits here. You think, oh, I control both the server and the client. That means I can control things. However, in a larger application, even if you control both sides, it's actually really hard to make a breaking change all at once to the entire code base. And so you start having to abide by the earlier rules. You really only get to break stuff whenever it's a small enough project and you control both sides. And then that takes us to behavior compatibility, which is actually really, really wide scope. Most of your effort I think will actually be spent here. The other stuff, you can add stuff. That's easy. We move on with life. Here is the application-specific pieces, which means I can't necessarily solve all of your problems. Everyone's gonna have a different problem. But there are some common patterns that you might actually want to know about. So here's some of the best practices I want to talk about. We'll go ahead and start with the new primitive fields default to zero or empty string or each primitive sort of has its default. Now, that is actually a statement that is just true. That is how new default fields work in Proto 3. The point of this is to actually accept that. That's how Proto 3 works. Don't try to avoid it just because you think that that's silly. There are ways to avoid it. There are ways to get around it. But if you just let the zero value be the I don't care value or the default value, you'll probably have a better time. If you need to, if you need to know field presence because that was removed in Proto 2 to Proto 3. There are these wrappers and in fact, it's called wrappers.proto. So every primitive field has one of these wrappers. And so it's really, really complex as in you've got a string value and it contains a value. That's, it contains a string value. That's all we're talking about. These are not anything that are really special. It's just that we need them. And so there's some well-defined ones that are already made for you. So you're free to use those. Also, if there's a couple different fields that you need to sort of know about whether they're there or not together, you can go ahead and make one of your message that's custom that has those specific fields and that message overall is either present or not. But all of this relies on the fact that while we lost field presence for primitives, it's still there for messages. And this is actually not really all that strange. Most programming languages actually deal this way. You have an int in your Java or C, and there is no, oh, this int isn't there. You're sort of forced to make it zero or negative one or something like that. And if you need it, if you need to know whether it's there or not, you end up making a pointer to it and then having it be null or something like that. That's the same thing as this. This is just normal boxing that takes place in languages. So moving on, there's updates. Whenever you add a new field, if you're not a little careful, you could actually have some client that's doing an update clearing that field. So let's say that we've got our library service before, so I just took out the important parts. We've got update book and I ask, all right, this is the book I want the new contents to be. That's in the request. And then in the book response is the name, the author, and the title. And that's really all we have right now. Well, and I guess, let's say that I'm actually, there's a client that's preexisting today and it looks at the book request and it goes ahead and updates the author name because it was misspelled or something like that. So it would commonly get the current book. It would modify the message and then it would upload it again via update book. If we end up having a new field because apparently users were forced to keep on reading the same book over and over, we're now going to introduce this field called red. And if it's red, now then we have a little icon or something on the UI to tell, hey, user, you might not be as interested in this book. So if there's an existing client that was modifying the author, it downloaded it. The red bit was let's say on, whenever it processed that, that client which was older would throw it away because it wouldn't know about red. And so then whenever the client then modifies author and then uploads the changed version, the server now thinks that the client was clearing red because it looks at the value of red and it's now zero. So it's now accidentally performing a change. This was not as much of a problem in Proto 2 because you had field presence. So you'd see, oh, red isn't there. It must not be a set. So, Fieldmask is actually a partial solution to this. You might not have looked at Fieldmask before. It's there, but it's not necessarily advertised in a lot of places. And so basically all it is, is you say which fields you care about. This is useful for both querying, where you say I only care about these three fields, don't bother looking up the rest from the database. But this can also be useful for updating, where you say these are the fields that are actually selected. And so it can have basically a repeated field of pads where you specify. And it works fine with sub-messages. If you have messages nested inside of other messages, you can say just the fields you're interested in. And Fieldmask defaults to all fields because this works pretty well for querying, where you say just get me everything and you don't need to fill in the Fieldmask. The reason I say that this is a partial solution is while it would solve our problem from before because the client would say I only updated author. And so the server would know to only update author. It's really annoying to use all the time. It's fine to use some places, this doesn't hurt, but if like every single API you started throwing this in, you'd get it probably pretty annoyed with specifying these fields. But this is how you would use it. So we had our update book request from earlier and you just go ahead and specify a mask in that request. And then this is what Java would look like, but each language has some helper utility in order to use this Fieldmask. And so we go ahead and get the name of the book and we load the book from the database with its current values as they are right now. Then we do this utility function. What this does is it copies fields, I can even point with this, eh, no. It copies fields from the request book into the destination book, into the current version that we just grabbed from the database, but it filters based on the mask. So it only copies the ones that are listed in the Fieldmask. And at the end of it all, we'll go ahead and push the book back into the database at the very, very end. So that's basically the flow. There are some options on the Fieldmask we tell, if you're doing this yourself, you should probably take a look at those. But it's sort of not important for this. So going a little bit more, I mentioned it earlier, there's actually some other API guides. The rest one suggests not to add new fields to things that are updated in this fashion. Not all of your messages are used with that sort of update request. That's actually a little bit hard in my mind. Granted, there's reasons for it. I think that this is still slightly an unsolved problem, except that Protobuf 3.5 re-ads unknown field support. So the initial problem was that the client threw away the request, sorry, the red field. With unknown fields, the client would go ahead and save that. And it has no clue what it is, but it just saves it. And if that message gets serialized again, so it didn't know it whenever it read it, whenever it serialized is it again, it'll go ahead and re-add that field. And so it's just a pass-through mechanism. And that's useful if you end up persisting stuff in a database or something like that, like a local data cache or something like that. Whenever you do understand what these fields are, you would be able to read them. But also in this case, it allows pass-through. And so the unknown fields are no longer cleared. So in a couple more months as Protobuf 3.5 sees more and more clients using it, this may be a problem that is just not an issue anymore. It just sort of evaporates if you can require Protobuf 3.5. So some people don't necessarily know how to use air details with your RPC. That's completely fair. So what the suggestion is, we've got the status code. And that's good, but it's very, very coarse. For additional information, you should make a message for it. And you make that however you want. It's useful for your application to say, OK, this failed. Client, whenever you need to know what went wrong, here's what you look at. And you can use a message for each different type of failure or for a small class. There's actually some predefined ones in airdetails.proto. I just took a sampling here. There's some more. A debug info is just like a list of stacks, just as strings. Quota failure has a little bit of information to say, oh, was this a failure because that particular client was doing too many QPS? Or is this a failure because the user ran out of disk space? So it can provide a little more information there. Help just provides a link to a documentation so that whenever someone sees this, they can actually go and look up what the actual more words than can fit in the description. They can look at what went wrong. There's also localized message, which is useful because the normal description will be in English. Localized message can be localized string for whoever the client is. And so then they can read it. So the recommended practice is to use these messages and they need to put them somewhere. That really is put them in metadata, which is what we used to recommend. It turned out to be hard to pass that around in some languages. And it was unclear. You've got this metadata and you've got this air. It was a little unclear how they related exactly what. So was the particular metadata key because of the air? Or was that just extra information, let's say tracing information or something like that? And so Google RPC status is pretty simple. It's the code in the message, which exists in GRPC today, and then an ENI for details. GRPC actually used to use this exact message type for its response status. And there was a point that we decided to remove the dependency on protobuf. And so we went ahead and removed this message. But we decided not to put details anywhere other than just the metadata because the metadata would work well for that. It turns out that it does work. It's just a little bit hard in APIs to propagate all these things together. So languages are receiving utilities to work nicely with Google RPC status. Not every language has it yet, but it's coming. And so for Java, here's the difference of before and after. You end up getting a status, this Google RPC, com-Google RPC status, as opposed to the normal status. And this is what the client code would look like. But then you have that ENI field where you can put associated information if you're propagating it with, I guess, exceptions in Java was a little handled already. Java was able to do it. Some other languages, it will propagate more nicely with error paths. And then the last thing, because we like streams and we like long-lived RPCs, that's one of the things that are really nice to do because you can issue the request and just wait for an event to happen, I highly suggest that you break that occasionally because clients are really easy at avoiding work that they don't have to do. So whether on purpose or not. And so if a client doesn't realize they need to handle the fact that, oh, this RPC might fail before it completes, then they very likely will have some bug in that code path. And so go ahead and if there's something that takes a while, go ahead and sometimes purposefully complete the RPC prematurely. That could just be after a fixed age or randomly just to introduce some noise. And then the client will see that it completed early and then go ahead and re-issue the request and then it'll be able to wait a while longer. There are some cases that this can free up connections, but as far as this goes, I'm mainly talking about it for the client code health. And now we're to Q&A. Thank you. Thank you. He's in the room right here. Oh, none. It seems, oh, look. Oh, oh. So does anyone have any questions? Like this is, I guess, free for all because people have probably dealt with their own problems. Were people following what was happening with the unknown fields? You all might not have even noticed that unknown fields were being added in program 3.5. A lot of people, that probably makes them pretty happy because that was a feature they sorely missed from protobuf 2. So the question was how do the versions work or how do they interact? Are they compatible? So to begin with, most languages, if it's supported protobuf 2 originally, protobuf 3 runtime also supports protobuf 2. So you can mix and match however you want in that case. Now, granted, protobuf 3 supports more languages. It was a simplification in part to allow more languages to be supported. And so you may not have as many client languages that can support it. But if you like protobuf 2, you can keep protobuf 2 when GRPC works, find the protobuf 2. It is, as far as wire compatibility, protobuf 3 and protobuf 2 use the same wire format. There is some compatibility there. The intended way of dealing with it is, let's say you've got a protobuf 3 client and then you've got a protobuf 2 server. The protobuf 2 server sees a little bit more information than the protobuf 3. And so if you're going to have them differ, you would probably have the consuming side be protobuf 2 because you can then see a little bit more information. But it can work a little bit both ways if you're careful. As far as some of the features and stuff that were added in protobuf 3, there's all protobuf 2 equivalents to them and things like that. So it was understood that you might need to mix and match a little bit. And as far as the protobuf 2 and protobuf 3 compatibility, as far as which type you use, those you can actually have protobuf 3 messages include protobuf 2 messages, and even that level. It's not like you have to have a particular tree of messages that you don't cross the streams. You can actually mix and match a little bit more than that. So the question is, when do you know when you break compatibility between systems between protobuf 2 and protobuf 3? Is that what you're saying? So that is basically getting here, add stuff. Don't remove stuff. Don't change names. If you aren't wanting to support JSON or FieldMasks, you can change names. That was classically completely fine to do in protobuf. But generally, nowadays, it's a little bit closer to just add. There was a question over here. So the question is, was there any request from the community for JSON RPC compatibility or translation? I've not seen any for specifically JSON RPC. There is quite a bit of translation between protobuf and JSON. Now, granted, that may not. I think some people will get a little confused on that. That is, if you have a protobuf message, there is a JSON encoding for it, and you can go back and forth between it. I have some arbitrary JSON, and I want a protobuf message for it. It's not quite as useful for that. You may get lucky, but you may not. I've not seen anything for JSON RPC specifically. It seems very possible. I think most of the tools are already there if you were so inclined. But I don't know of any movement in that front. Oh, so not many questions. Do y'all have had no trouble making APIs and letting them survive the test of time? Granted, G-RPC is only so old, but this must be really, really easy. Your problem now. OK, so the question is, is there any way to add Gzip compression after the server is already deployed and the clients are using it? Actually, so the server doesn't support Gzip? And so you're worried that some clients may not support Gzip. OK, so this is dealing some with the handshaking that goes in with Gzip. In general, you should be fine if you are wanting to respond with Gzip. Basically, you should be able to generally just say, I want to use Gzip, and then the server says, the client doesn't support that. And then just it'll send back uncompressed. And APIs may vary a little bit on that per language, but basically, there should be no problem for the server to send back Gzip whenever it wants to do Gzip, because the client informs the server what it supports. The much harder case is whenever the client wants to send Gzip, because it needs to know what the server supports, but the server, there's not a handshake before you do the message. And so the main thing we did there was to make sure that servers supported Gzip before 1.0. So you've got that going for you. You know that you can make a client that goes against something that's at least Jrpc1.0 server, and things should work. But if there's, let's say, a new compression protocol or compression type, you're still going to have that problem. I think that that part is unsolved. I think that we might be able to solve it some with a client config in order to notify the client ahead of time. But if you're just wanting to send back, were you wanting to respond with Gzip? Yeah, so we'll talk about that. OK, so it sounds like you were maybe hitting a bug or an API gotcha. There should be a solution for that. I'm not sure exactly which one you had, but we can maybe talk about that and figure out which language you're in to figure out if there isn't a step. I think it may be possible for some of the APIs to let you choose a compression format that the client doesn't support because you know better. And maybe the client is hiding it from you. But I'm not sure that that was necessarily what you were using. How does the binary payload play into using compression? So the question is, why would you use compression with Jrpc? So there's a couple of different reasons. One, you can compress just one message worth of content. And so let's say that you're transporting a bunch of bytes from a file. Well, those bytes might be compressible. And so you're like, well, let's compress them. Also, there's plenty of strings in ProtoBuff Messages. Apparently, they only use so many bits because they're ASCII and things like that. And so you can also get some savings there. There is a further thing that we're working on, although it's in various forms and it may have stalled a little bit because of push. But we were working some on full stream compression where you could actually send a stream. And multiple messages would be able to be compressed with one compression context. That's most similar, I guess, normally to if you have a repeated field. And you've got very similar data. One message to do the next will be similar because lots of data happen to be similar. It's the same name over and over. You've got a prefix as part of your ID name. And so those could all be stripped out. And the full stream compression was also assuming that fact. Also, some people will send HTML responses back sometimes or things like that. And there's plenty of payloads that compress well. Exact time this ends. OK, so it's technically ended one minute ago. Y'all are free. If y'all have any more questions, I'm going to be around. You all can catch me.