 Welcome to the third evening of GPN in 2022. Our next speaker is Eric. He is a software developer who wants things to work and has developed a framework to handle errors and error codes which happen during software development and he wants to enlighten us how to do this better and serializable and all other cool features. So give a warm welcome to Eric. Thank you so much for being here and doing this talk and for all of you, enjoy it. Thank you. So I'm going to introduce a couple of things in this talk. One is going to be a suggestion for how to handle errors serializably if you have message passing services need to log them, etc. And there's also going to be a second half in which I'm a go programmer and error handling in Golang is painful and I'd like to improve that by having some static analysis tools. So the second half of the talk is going to introduce that tooling which is based around the standard. I go by warp fork on the internet mostly. This is relevant because if you want to find these slides or bug me via either GitHub or Twitter, there's the name. So, Serum, the serializability part is please make your errors JSON. That's controversial. But there's a lot of different ways you could turn errors into JSON. So what I would like to propose is that simplicity is your friend and if there's one thing that you should do if you are standardizing your error handling is to pick simple codes, pick a string, pick something that is reasonably clear for a machine. You can attach human readable messages, you can attach as many details as you want but if you're going to make any choices at all try to pick unique error codes. Because structure is good. You want to log things almost certainly and you also want to help programs handle your errors gracefully and in order to do that you need some recognizable structure. If the only thing you give is human readable text, that's not very helpful. The only thing you give is stack traces. Well, that's helpful to a human in some cases but you don't usually want to have your logical error handling be based on that, right? So some sort of a string code is a good idea. The Serum standard suggests that you use some string code and cares almost not at all beyond that but having a code just helps. One of the inspirations I'm running on in Serum is experiencing SQL databases. They almost always have a code and a human readable piece of prose, right? And having that code be something you can copy, dump into a search engine and get results is just valuable. I don't know why we don't do that in all software and I want to suggest that we should. Everything else is really icing on the cake. Having something you can plug into a search engine is good. Later, when we start looking at how I would like to improve my life in Golang, having these error codes will also help because they're easy to enumerate. So if you recognize Golang syntax, indeed, this here is Golang syntax. Having strings for error codes means I can put them in documentation and then maybe we'll do some tooling leverage on that in a little bit. Having free text is also good. Having something that you can print to humans is good but it's useful, I think, to keep that separate from what you would parse with the machine logical code. So separate these things, easy. And if you want more details, I would propose putting them on map. The thing that's handy about doing all of these things in such simple ways is you can write one... If you have a strongly typed language, you can write one structure that says code, message, details, map. And that will be sufficient to carry any error. If you're building large, collaborating suites of programs and they need to message pass between each other and you don't want your strong typing system to make things unnecessarily fragile and error handling, fragility and error handling is bad. Side note, it's way worse than fragility in any part of your program other than that, any other part of your protocol. If you have a type error while you are handling errors, it's bad, right? So simplicity. If you're wondering where stack traces fit in this picture, I would also say stack traces are pros. Trying to over structure those doesn't produce a ton of utility. Things that are looking at stack traces are humans or if it's a program looking at a stack trace, it's a bad program. So treat them as free text. The standard is so simple that if you want to start using it, do it. You don't need to do anything in particular. You don't need libraries. You just write a structure with about three fields. If you think I'm describing something too simple, okay, fine, add fields. That's not generally considered a problem in JSON. Mostly I would even say use the details map. That is meant to be extendable. As I was mentioning in inspirations earlier, SQL databases were a really strong hint to me in this. Having a code that you can plug into a search engine is just so good. The other thing that strongly suggested this simplicity to me is working with GraphQL APIs. I don't know if anyone in the room has used GraphQL, but it has a concept of unions. It provides some type structure and you can say, what is a type that should contain one of these things? The best way that I found to do error handling in GraphQL APIs was at roughly the top level of most requests to my APIs, there would be a big union, and it would be the result type for success, usually. Maybe there's more than one, but usually one. Then there would be a union of all the different possible error codes. What that code shook out to look like is in GraphQL, you have this magic field called underscore underscore type underscore underscore. That contains the string name of the type in the GraphQL schema for what is in that field when you've got a union. My error handling would always just switch on that. This produced some of the easiest, smoothest, best error handling APIs at the lowest cost of anything I've ever worked on. My suggestion is do that, do that everywhere. In Serum, I call that string code in the JSON map instead of underscore underscore type underscore underscore, it's the same concept. In the other direction, a negative influence that inspired Serum to be as simple as it is is I've tried to write a series of programs in which I had JSON APIs through and through and error handling that even had the concept of this union sort of thing, where you'd have like one code string and then more structure. But I made too much structure in the rest of the thing I wrote in Golang types, like you would have the code field that would have different field types that were recursively complex as well for all the other details. And in trying to maintain that suite of programs, there was like three of them that were connected with like a serial pipeline API. The number of times that those programs broke each other by changing their error detail fields was, I mean, more than never is the wrong number, right? So simple detailed maps are a good idea. If you want a specification that just says keep it simple, I have written one. There is a read me available on GitHub, which will detail this little bit, provide some suggestions for how I would tend to structure my strings and error codes. All of it is, of course, simple suggestions and easy to implement in JSON. So you can use these things or you can be inspired by them. Both would be productive, I think. Okay, so that is just the suggestion of use simple errors, use JSON. Now I'd also like to show you how this can turn into something much higher leverage if you're developing. It also becomes a little bit specific to go lang from here on out, but I hope you will forgive that and appreciate it. So error codes are innumerable, as I was belaboring earlier. That's good because it is innumerable for a human. It gives you a limited amount of things to reason about in your brain. It's innumerable for the purpose of dropping it into a search engine. That's also fantastic. But innumerability also means you can have a tool reason about it and do exhaustive checks for error handling. It's not just about the human brain having a limited amount to think about. It means a tool can make a checklist of all of the possible error codes that it encounters and say, yep, that one was handled, that one was handled. This one was not worn, the user. And in going, I get a little frustrated at the error handling pretty quickly as I write honestly even really small programs. The first thing that is likely to give me grief and frustration in writing a go lang program of any size is the error handling gets out of control. If you haven't written go lang, you might not be familiar with this, but the usual way to handle go lang errors, the easiest way to do something is return errors are values and going. I like that. But it's return fumped.errorf and then a string. And you can put formatting codes in that string, but it's a string. It's easy to go nuts with that, like have no control whatsoever and it will compile. You're just not going to have a fun time dealing with it over time as your program goes, especially if you have more contributors, especially if you need to start reacting to specific errors elsewhere in the code base. It does not go well for me, I find. You can write typed errors and use like strong typing and go lang and do type switches, but it's not encouraged and the compiler doesn't help you do that. So I'd like to try to fix it by using simple string error codes everywhere in my program and then using a static analysis tool to verify them. And such a tool has been implemented. It is called the Go Serum Analyzer and it uses pretty standard mechanisms in go lang for other go lang tools like GoVet. There's a framework for static analysis tools and this uses it, so it's quite easy to introduce. This is an example of some go lang code that is subject to that kind of analysis and the documentation strings that we will match the analyzed flow against. So here we have some function to do something and it returns some error types. These strings happen to be the error codes. The analyzer discovers this in a way that I'll cover a little bit more in a moment. And these pieces of documentation, everywhere after it says errors, colon, line break, then this dash, treat it like a list, here's the string, double dash, description. If you're thinking this feels a bit like static errors in other languages like, say, Java where you have to write a big list of the throws clause, it is, it's exactly like that. We're just doing it in Go language, it doesn't have those features, so we're doing it in the documentation strings. So writing the simplest possible format that both a human and a machine can recognize and plug into the middle of this documentation is our solution to attaching this information. So the way the rest of this analysis will work to help you make sure that your code does what you think it does, is we're using a very simple taint model. So as the logic evaluates in your program, any place that an error value is assigned to or returned, we're going to look at the total set of codes that could appear there and grow that set until we've reached the end of a function. And then we're going to check that that function does those things. So we'll examine any function that returns an error interface. Then we'll actually look a little bit more at that error value, any place that there's a return, and see if that interface also includes another function called just code that returns a string. We're going to assume that this function then returns one of these serum style strings. This will be the subject to our analysis. We'll look inside that code function to see what strings it can return and do a little bit of analysis of there. And then we'll add that to the set of things that the overall function, the one that returns this error interface, is expected to return. And like I said, simple taint model. So that set just grows for the scope of that function. And then we look at that doc string with the simple parts of like dash, name, dash, dash, explanation. If those sets are different, report it. It's pretty straightforward. I mean, implementing that takes some work, but we've done it, there's a tool for it now. This can analyze a wide variety of different kinds of go line code. So for example, here we see some function that has many different error values being returned, many different return sites, there's branches here. This works. The analyzer can trace all of the branches and it accumulates the total set of codes into our expectations. If instead you want to assign to error values in the intermediate, the analyzer traces that too, it works. You can write code very freely. When we're analyzing an error type to see what kind of codes it can return, here you see this is the magic function signature which we will look for the code in. Here we're seeing more than one assignment. This can be followed as well. We can even allow you to compose strings as long as it's constant composition. We're not like doing total, what's it called, function, quiet evaluation. There's a word for when you're doing too much analysis and we're not doing that. I forget the term right now. There are some things that this analysis tool cannot handle. For example, here we're plugging some errors back out of an array indices and like this is... code you never write in practice and so accordingly we also haven't taught the analyzer how to deal with it. But if you do have code like this for some reason, then you can use this particular kind of comment here, error codes equals. Overrides the analysis and says I swear. Now of course if you swear something that's wrong, the value of the analysis tool is going down rapidly, but don't do it wrong, of course. You can also use overrides in a more partial way. So if you've looked at error analysis systems before, you've probably been looking at me a little funny ever since I said simple taint model because the problem with simple taint models is they grow constantly. If you have a very large program, the list of errors that you'll have from the main method will get very large indeed. You probably don't want that. So you need some way of reducing taint as the program gets larger. So a very common pattern in code that I've now written with this framework is you use a switch statement to look at the code and then pluck out the ones that you recognize and want to handle explicitly in some way or just say I swear this is an error, I'll just throw it away. Whatever handling you want to do, pluck those out. And then in the default case you can say, yeah, I swear I've removed this from the set. It's difficult for the static analysis tool to really tell that you've removed it from the set, so you have to make this claim still, but it's fairly easy to make these annotations yourself and get a lot of value out of it. So this is how you reduce taint spread. This analysis tool also knows what interfaces are. So if you want to write an interface, say what errors it contains and then anything that implements this interface, it will be checked. So in goal-line interfaces, you may also have to make sure there's some place in the program where it is clear that the implementation is being assigned to an interface value, but this is the nature of goal-line. If you're a goal programmer, this does not surprise you. If the implementation of the interface returns fewer errors than are specified, no problem. If it returns more errors, then this will be flagged. The analysis tool will tell you that you have made a mistake. Line numbers will be attached in all of these reports, incidentally. So hopefully this is enough examples to convince you that this analysis tool will let you write a lot of goal-line code in a very free-form way. It does not require you to change the way you write code. It just lets you document your error codes. The system is overall also designed to be easy to adopt. I know everybody has lots of stuff to get done on their day, and they don't want something that slows them down a lot. So this tool is designed to let you. If you annotate functions with those documentation styles, then this analysis tool will check them. If you don't, it won't. So you can bring it into the code base in the areas you want, at the pace you want to do so. Unless you really want to get strict, and you want to say, I am doing everything with annotations, then the CLI tool allows you to add a strict flag, and then anywhere there's a public function in your whole code base, if there aren't annotations, it'll say you goofed and give you a line number. In order to keep the amount of annotation work that you want to do down, you can also, especially on non-public functions, skip the annotations. The evaluation system will still recurse through those functions, as it is doing the analysis. So in the analyzer's mind, it will still be building that set of error codes that it knows this function can return. But you don't have to annotate them. So other public functions that call through a series of internal private functions can do so, and analysis is still operating. You can just save some verbosity for yourself. The system can be applied to one package at a time or many packages at a time. If you've used other Go tools, it's normal to write GoTest.slash. And that means all the packages in this directory. Or you can say dot slash just this. Same with this tool. The tool will also recursively scan other packages as necessary to figure out what's going on. But at each step of that recursion, it will trust the public functions of the next package and their annotations to be telling the truth. So if you're analyzing one package, all of the errors within that package will be found. Or all of the desyncs in your error documentation will be found. But if the recursion of checking all these things has crossed packages, and then there were errors of the error handling internal to that package, it will not be reported twice. It will only be reported when you analyze the inside of that package. So the reports are meaningful. And you can avoid noise if you need to zero in on some parts. In practice, this has not been deployed in a ton of code bases so far, but it has been deployed in a few code bases. I've done some analysis on what indeed the analyzer does. It is really incredible how many manual error checks get replaced, even in very small code bases, like small number of thousands of lines. We've found that deploying this analyzer tool was replacing thousands of manual checks. I put some debug statements in the analyzer. You'd be like, how many sites did you check this on? And it was like, tons. Holistically, people who were developing in a code base where we used this also said that the quality felt better. One person had a quote of, I don't like that this tool has to exist, but I'm glad that it does, because we were less happy with not having these verifications. For some numbers, in this medium-sized deployment we made, we found 900 and some error code sites were verified by the automation. After only like 60 functions, that number goes up really fast. In total, there were closer to 200 functions that were covered, because remember this search will recurse through private functions even if they're not annotated. So many more functions were analyzed than were actually manually annotated. It was about a third that were fully annotated. And that gave us the full effect and the full confidence in the public API of the packages in which we applied this, with a lot less work than you might expect. Arguably less work than doing these throws annotations in Java, because you have to really be quite clear with those at every level of detail. Here you can pick and choose. So this is a result that I'm quite happy with. We improved the quality of code without a ton of work. Our documentation also got better because the docs of these functions say what the error codes are. It's a system that I'm happy with, and I would recommend that people try out if they're interested in improving their error handling. That's about it for the body of the talk. Here are some links. The spec is in this first link. The analysis tool is in the second. The analysis tool is also staggeringly well documented. If you jump into that repo and go to the user guide, almost all of these slides were just screenshots of the sections of the user guide. The full user guide is, in fact, even better. I should say also that there is somebody in the audience who has written a huge amount of the code behind this, as well as all of the excellent documentation. Janne's pair over there. So if anybody has more questions and wants to bother an expert, you know where to go. Any questions? I will come to you with the microphone. Quick question. Is there any project in the wild where we can look at it and basically see how it works or how you should implement it? What's the best practices per se? You will find some documentation of best practices in the user guide. Otherwise, I don't have one in Candy, but if you go to Warp Fork in GitHub and then Warp Forge, this is the first project in which we deployed it, so you'll find some good examples there, and I can try to pass you that link later as well. That helps. Thanks for the talk. I'm curious if you have any experience combining this or mixing this with some logging. And if so, is that just plainly logging the error codes and the error message, or are there any tricks that you could use with this? I serialize the whole JSON of the whole thing. I think... Well, it depends on what kind of logging means. If you mean DevOps logging, put the JSON out entirely. For the usage of humans, usually the printout function that I'll use is error code colon message, and then I'll skip the details field. Usually I use the details map for more mechanical things, and my recommendation is that if the details are reasonable for a human to see, then you should template them into the message string immediately. This is a good idea, in part because if you have a chain of microservices, in who knows how many different programming languages, you don't want to assume the same templating functions exist in every language and so on, so just templating the string as soon as possible into a single string seems to be the most like resilience choice. Of course, I can't enforce any of this. It's a rough series of suggestions, right? But that would be what I would do. Okay, cool. Any further questions? Thank you for the talk. And I wanted to know if your analyzer also inspects the error codes from the standard library. We can use your tool when writing our own application, but what about the standard lab? There's no fancy features for that yet. I guess that would be an awfully nice future work topic. As a rule of thumb for how to work with Golang productively, I find that usually I want to do all of my error handling enforcement and like big logical rules on the entry and the exit of each package. And so usually I'll write some functions on the bottom, the incoming side of a package that look at everything that they're calling out to and they sort of normalize it into something I understand as close as possible to that site. Handle errors early. And then these documentation level checks are the most important on the public surface that's going further out. And that doesn't help at all with the standard library. No. I tend to handle conversion of standard library things at the bottom of the interface where I'm touching them, I guess. It might make sense in the future to add some more special heuristics to this tool which like auto course the EOF error would be like a super common one you'd probably think of, right? We could add cases for that. It hasn't happened yet because I didn't want to get that opinionated. But I guess that PRs in that direction would probably be welcome. It seems like it would probably raise productivity. So good question. Good idea. Do we have any further questions from the audience? If not, I would like to thank you again for coming to GPN this year and having your talk. Thank you so much for sharing with us. Let's have another round of applause for our speaker. And I wish all of you a nice evening on GPN.