 Welcome to Debugging with PostgresQL, a strategic approach. I'm joined by John Ashmead, database developer and DBA at Nistica and principal at Ashmead Software. John's going to discuss strategies for debugging Postgres and take a look at root causes, technical tricks and scientific strategies. My name is Lindsay Hooper and I'm one of the Postgres conference organizers and I'll be your moderator for this webinar. Let me tell you a little bit about your speaker. John's been working with relational databases since the 1980s, building and enhancing databases in SQL Server, Oracle and Formex, Ingress and Postgres. For the last five years, he's been the DBA and database developer at a leading manufacturer of optical switches where he's in charge of the care and feeding of the manufacturing database. For the last six months, he's been porting the manufacturing database to a factory in Zuhai, China and John gives frequent talks at database and other programming users groups. He's working on a book called Zen and the Art of Debugging about the most underappreciated but also important aspects of software. With that, I'm going to hand it off to John. Take it away. Hi, thanks, Lindsay, for that wonderful introduction. And hello, guys. Good to see you guys here. And this is one of my favorite subjects. One of the most important parts of our lives as programmers and something we're always all trying to do better, not something we always talk about as much as we should. We know that debugging takes from 50 to 90% of the time we spend as programmers. And yet it gets only 2% of the attention. I can quantify that. On Amazon, there are 90,000 books on programming. There are about 1,000 with debugging in the title someplace. So about a percent or two on debugging absurd ratio. Postgres SQL has some great tools for debugging. It's very strong in that area. We need to know about the tools and we need to know how to use them as part of a tool set as part of systematic strategies to debugging, which is our subject today. So we're going to go over ways that you can be more effective as a debugger, build better code faster. So I got my start in this field when I was in the work study program at Harvard University and I showed up the first day at the Harvard physics lab. My assistant professor asked me to crimp some cables for use in an accelerator experiment. I crimped with more enthusiasm than judgment and destroyed the cables and quite a bit of equipment. About two days later, I came back for my next visit. My assistant professor had calmed down and he gave me a copy of the Kraken on Fortran and said, see if you can destroy this. He also gave me three pieces of really good advice. Write the code back in a code deck that goes from top to bottom. Don't jump around a lot. Use meaningful variable names, basically have the code explain what it's doing itself and as best single piece of advice, throw the first one out. Typically, your first piece of code to solve some problem is where you're figuring out what is the problem, what are the critical issues, how does the language work, what are special things about the environment or the compilers. And by the time you've sorted this all out, you're likely to want to take a different view of the design. Throw the first one out. You'll be happier for it. The first one that I had to write was a program to track particles going through a magnetic beam. That's particles and magnetic beam on the left. I found that I was having a lot of trouble with bugs. So to help sort this out, I had my Fortran deck inject random starting points into the field and I would then track them through. The randomizer had some particles start in the middle of a magnetic field. As a programmer, I would never have done that because we know as humans that we don't start particles in the middle of a magnetic field, but it was valid. I double checked those calculations and it gave me a lot more confidence. This turns out to be a very powerful technique. Postgres has wonderful tools to support this. You can use generate series to get a set of data to just probe your database or your code thoroughly. You can use the random function to select typical rows to blast through your code and try everything. Basically, bugs have a kind of exponential falloff. If you try 100 cases, you might see 10. The next 100 might show you two or three cases and so forth. You never quite run out of bugs. This is such a good idea. It's become a professional technique called fuzzing. There's a consulting group which uses firing random data at websites to verify that they are secure. You can never outwit the black hats. They're thinking all the time, how do I make your life miserable? But if you try random data, you can make it harder and harder for them to find holes. And they'll give up and they'll pursue some other victim. Netflix has turned this into a whole methodology. Netflix, which depends on the reliability of its servers, which have widely varying load, had a problem with the servers going offline at random and its systems just collapsing. Its programmers were intuitively assuming that everybody was going to stay up and their only problem was their own code. So Netflix created something called the Chaos Monkey, which would randomly turn servers off. And the Netflix programmers were told, your code is to survive random hits. If one server went down, you ought to fail over instantly to the backup. This, in a certain sense, made Netflix possible no matter how spiky the load, their code is geared up to track it, to deal with it. They now have a whole family of products based on the Chaos Monkey. And there's even an engineering discipline called Chaos Engineering, a bit of a paradox but a fun idea. Okay. And what we're really dealing with here is the root cause of all errors. And typically, our bugs are self-inflicted. We have some habit of mind, some misconception. It's a little late in the day. We shouldn't really have been writing critical code. We're overconfident of our abilities. What have you? We're the problem. We have to get over that. Of course, bugs also come from BitRot, where the environment has changed versus I just got hit this week with some bugs that came from an upgrade from an earlier version of JavaScript to the ES6 version. The syntax for the apostrophes and string interpolation is slightly and subtly and fatally different. Okay. This happens. BitRot happens. There could be new requirements and the old code isn't up to it. And your users might say, well, your rival, your competitor is doing this. Why aren't you doing this? But mostly, it's us. That's our problem. Okay. So we know the problem. What do we do about this? I'll start off with a couple of really powerful techniques that relational databases offer to you that can produce performance improvements of 1,000 to one. It's just amazing. Foreign keys are probably my most favorite single technique for cleaning up a database. We can see here a very simple sample of cities and the weather in each city, each day. And by using foreign keys between the city and the weather table and the city and the city's table, we can verify that there's no weather for cities that don't exist. We make it faster because our foreign keys usually will have indexes and we can go up and down the index. And now we have to think about what indexes we're putting on the table. They serve as documentation. When you see a references statement, you know that two tables are linked. This information is tracked in the Postgres system catalog, a PG catalog in the information schema. And if you're not already used to looking at them, they're a treasure trove of wonderful stuff to find out about your database. However, even the simplest piece of code can have bugs in it. The first time I showed this slide, I said to the audience, there are two bugs in this code, find them. My audience who are always up to a challenge said, oh, and here's a third bug. So there are now at least three bugs on this slide. And I'm going to give you guys two slides in which to find them. Okay. Clock starts now. The second favorite of my techniques for finding bugs without by proper prior preparation, as they say, is to put timestamps everywhere. Usually when you're tracking down some difficult bug, the most important single thing is when what happened. This is a Ruby on Rails trek that Ruby on Rails will put in for you in every table unless you stop at two fields called created at and up and down. Two fields called created at and updated at. And Ruby on Rails will update these for you so you can figure out when things happened. And usually the most recent change is where you start your search. And that's great. Except for one problem. Ruby on Rails spends a lot of time updating them for you. Okay. But it also doesn't know about other tools that are also updating that table. So your timestamps are only part of the story. You can eliminate the wire traffic between Ruby and the database and get more reliable timestamps by using Postgres and by using in particular timestamp triggers. If you create a little function called I like timestamp trigger because that's really clear. You can have it take charge of fixing your timestamps. Now you know they're always correct. And you just always put two fields called created at and updated at in your records. And then turn on attach the timestamp trigger. If you don't change the field names, then you only have to write the code once. And I've turned this on in every table I've built in the last five or six years. It's just a godsend. Even very simple tables. It helps back to our three bugs. The title of this slide is actually bug number one. That is the name of Bangkok capital of Thailand in the local language. That's a hundred and seventy seven characters long. It blew out of our Char 80. Okay. You could check the entire planet for the longest city name and make your field type one character longer. Or you can just make it a text, which is just as fast. So the first fix is to create a type. I like to call mine underscore T city underscore T and just as a text and not know because you're supposed to have a city this way. If you did need to change the type, you would change it in one spot. And it would change everywhere in the database. And it also helps to document the code. You can see that cities and weather are using the same type. Furthermore, this is a really good idea. The typing is because it tells us what we're doing with the other fields. When you looked at the two temperature fields, two slides ago, you didn't know whether we're centigrade or Fahrenheit. Furthermore, people could put in all kinds of ridiculous garbage and it would be accepted. So that's the second bug in the previous two slides. The third, and this is the one that the audience picked up on, is that you don't call fields with reserved names in Postgres. Postgres lets you do this, but it shouldn't. So that shouldn't be a date field. It should be a weather underscore date field or what have you. Okay. And so you can contribute in the chat room. If you see any more bugs here, there probably are some more, but that's three bugs in a 12 line program. And it looked fine on the page when it was first written. That tells you what we're up against. Okay. Another thing which we're up against are habits of mind or our innate prejudices. My first contract when I went independent was to fix, was to maintain a small medical database for a laboratory where the original programmer was a moonlighting IRS agent who had died of a heart attack. There was no possibility of an exchange of code. There was no documentation. And being an IRS agent, he naturally thought about patients in terms of how they paid. So he divided the patients into patients whose doctor paid and whose insurer paid. And every patient was in one of the two tables. This meant that all the reports had to have two copies for each type of patient, even though the medical question was the same. This way of thinking had been carried out in other areas. Each of the test types had different code and so forth. To the point where when I got there, the main lab report had 12 copies. And I could tell by the timestamps that the previous programmer had been making each change by editing all 12 copies in succession because the timestamp would be a couple of minutes after the previous timestamp. And he would usually get 11 of the 12 right. So I said to the lab director, look, they can save us a lot of trouble going forward if I consolidate the code into a single effective table and just have one lab report. Back then I had to use the make command and macros to do this. Today with Postgres, I would use an updateable view. So here's a sample of what you might do with this thing. You create a view called patients. You pull in the two tables. You have a little flag that says which table you pulled it in. And you think, well, with views in most databases, all you can do is a select. But with Postgres, you can shift to an updateable view and have an update on the patients table. Say, okay, I've got to go to the patients who doctor pays or the insurer form of the patients and make the appropriate edits there. In fact, in Postgres, a view is really almost a kind of function definition. You can arrange the view so when you edit the fields in the views, you're really doing a function call on each row and turn. This is an extremely powerful technique. And in fact, I was just using it last week on a project where we had to add a new factory and we had to combine code, old code and new code to make this happen. Okay, so one of the approaches, we're always trying to find new ways to avoid bugs in the first place. One of the pioneers in this is Bjorn Streusrup, inventor of C++. And he's one of the great languages. And I was trying to pick it up in its early days, but was having a lot of trouble because the debuggers of that time couldn't cope with all the macros. And the code was just too much stuff there. So at a conference in Philadelphia, I asked Bjorn, how do I debug C++? What debuggers and linters did he recommend? I never forget his response. Okay, well, that's one approach to the problem. And I was, of course, annoyed. But then when I pulled back from this, I thought about it some more, and I realized that friend Bjorn was on to something, that even though we can't normally write perfect code, we can try to. And if we're looking at a module, we're not sure it's going to compile, maybe we need to simplify it, break it up into pieces or do something so we can be confident that it will compile. Then when we actually run out and we see mistakes come back, it tells us it's too complicated, we don't understand the language. The compiler is helping us debug bugs in our own head. This is an extremely powerful concept. So you have a little bit of discipline, you try to make your code really clean, and you save the compiler as a last ditch defense against your errors. And what we have to remember here is that every piece of code has two readers. One is the compiler, the other is you, and you are just as important as the compiler. And it could be you three months from now, when you can't remember what you were doing or a junior program you've brought in or what have you. Okay. The Postgres man pages, I think has some really wonderful stuff in there. They don't call attention to themselves, they just try to lay it out clearly. They have a wonderful note at the end that says, if you see a problem here, let us know. They are written in a uniform style. The naming conventions in Postgres are pretty consistent. I myself like to put the noun first and then the verb afterwards because that causes related code to show up near each other in an alphabetical sort. A cheap trick, but a nice one. You want the code, again, to flow from top to bottom, as in my physics lab. And you can see in the documentation that it flows in a nicely hierarchical way. So again, you can see where you are and how to get there. That's good design. Use logical consistent indenting. This is one of the strong suits of Python, perhaps the best single language for communicating with Postgres. I use appropriate comments. I've had code with no comments in it at all. What's going on here? And then you have code where some of the comments are like small novels. As Kurtigan and Richie said, I think it was Kurtigan and Richie and elements of programming style. Don't comment bad code, fix it. And use the house style. Even though you think tabs ought to be two spaces if the locals are using four, let it go. They're more important battles. Okay. So there are three main styles of debugging the code itself. Lots of specific styles, but three important ones. The first I call a cat in a box debugging. This name I get from when I was a starving grad student and I couldn't afford a cat carrier. So I got a cardboard box and I folded over the four flaps and stuck the cat on the inside. Cat was not happy. Clawed in every direction until it's got a little head out sticking like a periscope through the center of the box. And then it was okay as long as they could see where we were going. So clawed in every direction until you see daylight. Sometimes you actually have to do this. You have no clue why there's a problem here. You're just going to keep trying everything until you find some green case and a red case that you can leverage to actually solve the problem. This in a certain sense is the heart of techniques like fuzzing and the chaos monkey. But it's not usually the sign of a discipline programmer. Better is to get some sleep. That solves an enormous number of debugging problems and then get a second pair of eyes on the problem. I remember one thousand line program I'd spent the day writing. There was only one line that wasn't working. I couldn't make any sense of it. It was like late at night. So I called a day and the next morning I go to Eric, a senior programmer at Bellcore and say this line is not working. It makes it looks good to me. And Eric says after a moment's thought, you know, you need an extra pair of parentheses here. Yes, he was right. And it took him 30 seconds to find a problem I couldn't have found in three hours. The second and most common kind of debugging strategies is linear debugging. Get out the steamroller and just flatten the code. You can put in a raise notice. You can put in print statements. There's actually a debugger for the Postgres procedural language. Not much used, but it exists. This is a linear process. So it's slow. If the bug is at the end of your module, it's even slower. It should usually be the last resort. My first resort when I see a module with a problem is to stare at the module and ask, can I see it by inspection? But you do have to know how to do linear program. Sometimes that's what you're stuck with. Or maybe you just don't understand the particular language well enough and you're learning the language at the same time that you're learning the solving the problem. Now a linear debugger is not as effective as instrumentation. A linear debugger shows you every line, but in most code there are a couple of twisty points which merit the attention and the rest of it not so valuable. There's a lot to be said for putting a print statement or debugging check at the start of a function and at the important turning points in the function itself. And I've come up with a lot of tricks for doing this. You can even create temporary tables or on log tables to select specifically useful values from intermediate things. This is just a sample of that. Notice the clever trick where you preserve the local time at the instant that the code is executing. One of the characteristics of Postgres is that the timestamp is the time of the start of a session. If you want the wall clock time, you have to do a trick like the one you can see on the slide. I'd like to thank Bruce Moomjian for an explanation of how that trick works. Okay, so that's something to bear in mind. Don't rely on the debugger. Put print statements where you know they're needed. Plus once you comment out the print statements, you'll be able to reuse them on your next debugging run. And a commented out print statement tells the next programmer something about the code. Here's a weak spot. Okay, we found our bug and now we have to fix it. Eager programmers that we are, we have a tendency to just get in there and fix it. First off, statistically half of all bug fixes either don't work or create a second bug. I don't know how you do the stats on that, but you know I'm going to believe that's ballpark right. So you don't want to fix the bug right away if you can avoid that. Instead write an error report or some trappy stuff or what have you to catch the bug, verify you are now catching it. Then fix the bug and verify that the fix worked and that your error code is not reporting an error. And in fact one of the ways to fix bugs is to start from the last one, the last place where the bug showed up, make sure you've trapped it there, then walk through the places where the bug hit to where the initial mistake was made trapping at each point which might be four or five successive points in the code. So you can from one bug get five or six checks out. Furthermore, bugs travel in packs. Look for similar problems elsewhere in your code. For instance, I had one bug last week where I spelled air count with an underscore and one piece of code and without the underscore in another. I gripped out every spot I was using air count and fixed another bug or two just from that. Furthermore, you can look for the root cause and in the case of my very simple air count example, the root cause was not having a consistent policy for picking out the names and I had to remind myself that when it's more than eight letters, I put an underscore out. But it might be a design bug or anything which can lead to a lot of bugs all over the place. So you can for one bug report, get a leverage of 10 or 100 on bugs fixed. And of course, you'll ultimately want to see if you can prevent it entirely. Maybe that module like our friend on the Ariane 5 doesn't need to be run. Ah, yes, I see a question in the chat about how to debug, how to have statements run and debug and not in release. We'll actually get to that in a moment. Excellent. Now when you see enough of these annoying little problems, you have what the Ruby community calls technical debt. The symptoms are it's hard to read. The code stinks, cut and paste. It's the comments don't make any sense. There are lots of debugging statements in it. And the most characteristic thing is that small changes tend to blow the code up. You can't touch it, it just destroys something. And sometimes you don't find out about it for days later. After a little bit of frustration, and my rule is Goldfinger's rule, once is happenstance, twice is coincidence, third time is enemy action, Mr. Bond, is that if the same module annoys me three times in a row, it's time to think about getting rid of it. So we need to repair, refactor, rebuild. Rule one, and I gave a copy of this book to my niece who just passed her medical exam last week, don't kill the patient. When you're fixing code, you want to fix one module in a way with as few side effects as possible, and your users will thank you. It's incredibly easy, particularly with bad code, to have unintended side effects. So testing is important and warning the users is important and backup copies are important. Then this is too large a subject to deal with in our time, but I'll recommend two texts which are particularly strong on this. The Mikado method is a reference to the game of pickup sticks where you want to pull out one stick without disturbing the rest of the pile, a very good metaphor for what you're trying to do. Now this is probably the most expensive single bug in history. A pause to admire the picture of fireworks, those fireworks cost the European community half a billion dollars. That's 43 seconds into the first launch of the Ariane 5 rocket. That's the rocket blowing itself up because it's having a bad day. What happened here is that the Ariane 5 was a lot bigger than the Ariane 4. They had a lot of trouble with the boosters on the Ariane 4, so they were worried about the boosters on the 5. The code in the 5 had bigger variables coming in and they promoted it to 64-bit floating point off 16-bit int. Okay, they had a module in the old code which lets you do a restart if you needed to pause the countdown and continue it. The Ariane 5 was too big for a restart, the code was dead. Because the code was dead or at least irrelevant, they didn't test it. But because it wasn't breaking anything, they didn't remove it from the software code, from the controller code. When the flight controller, when the launch started, the flight controller is checking out its modules, sends 64-bit data to a 16-bit int module, module throws an exception. What else is going to happen? The flight controller, nothing daunted, sends it to the backup copy of the module. Same software, same exception. Flight controller goes, oh, this is not acceptable. I'm going to blow myself up. 43 seconds into the flight, the flight controller blows itself up. The next three days are spent by the French foreign legion picking up pieces of dead rocket from the tropical island where all this happened and the French manager of the project said he was on the whole pleased that the French government had abandoned the use of the guillotine as a management reinforcement tool. Okay, the full report, which is worth a read, is in the slide. But what we take away from this, just looking at is dead code can kill you, always run a system test, always fix up your types like your domains and so forth, and so forth, standard lessons. Well, one of the most important parts of our code is the users. We need to work with them. There's essential part of the system is the code itself. And I got back a set of screens from one of my users where it was like watching storyboards. He showed me the starting, middle, the endpoint. He circled the bad stuff. It was great. It took me half an hour to find the problem rather than a day. I congratulated him aggressively with that. Your screen should tell you not just what the user needs, but enough breadcrumbs and locations so you can spot where the problem is. And in fact, you can actually have the users read the code over your shoulder. This is some code that I showed my doctor in the medical lab. And he could read this code. It was a different language. But the same logic. And then agree that my code correctly implemented the diagnoses. Okay. So obviously the best way to not have bugs is to eliminate the code. Code that doesn't exist can't break, except of course for launch modules. So one thing is to write simple little tools like the small shell script I use as a wrapper for PSQL. You can use the make command. You can use function calls to consolidate code. You can use the third normal form. Make sure your database is set up properly rationalized. Third normal form is a mathematician's way of saying if you have one change to make it, you should only have to make it in one spot. As you are working on the code, keep looking for ways to automate stuff. If you find yourself doing the same thing mechanically, make the machine do it. You're more important and more valuable than the machine is. One of the things we can automate a lot of are reports. We can create a table of reports and of report types and half the code will be uncommon across the reports. And so we can have our delivery time on a new report and maybe even suggest some reports which wouldn't have been possible without this trick. Okay. So there are a lot of ways in any shop where you can automate stuff, factor out bits, use the database not only to track the user information but also to help you serve whatever the project requirements are. Well, one of my biggest single successes with this was at a portfolio management company where I was called in to write some tools to help their clients review their portfolios. These had high security requirements. Their clients all had $5 million or more portfolios. That's something I really cared about, who could see what. But I noticed that the specifications fell into a very common pattern, usually master detail joins. So I could go into PG catalog, pull out the data needed to build the stored procedures. And most of the time, 80% of the time, the code generator built the right code. Most of the rest of the time, it got close enough that I could just write a couple of little squiggles of additional code. And there were a few bits of code where honest work had to be done. In 80 minutes, the code generator had done 100,000 lines of code and a couple of more days and I'd finished the project. Okay. And I then have a wildly coyote moment. I realize I've just done a two-week project, three-month project in two weeks. And what is the bank going to say about the mortgage? Fortunately, the guys had a sense of humor about this. And they said, okay, that's great. This project is under control, but you're already budgeted and we have some other things we'd like you to look at. Great. One of my favorites is foreign data wrappers, Big Jim Lodgensky, wrote the first cut on this thing in a very powerful tool. I think of Postgres as a kind of friendly octopus. It sits in the center of things, reaches out to other databases on the left and on the right. Turns them into foreign data wrappers so that they look like they're part of the Postgres database. And then you can do joins and so forth as if we were all friendly tables in the same database. A very powerful concept. You have to watch out for some rules about, some things about performance and so forth, but still very effective. And when you're deploying it, you can use foreign data wrappers again to start off deploying into your test database. And then just switch a few names and deploy into your target. Very rapid deployment. So now we think about the strategic level on programming. We've been looking at tactical debugging, but let's think about our overall project. Well, the way you solve a project is to start with the problem, break it up into pieces, and then assign the components to individual programmers or teams or just yourself on different days of the week, depending. And then use messages or function calls or whatever to communicate among the parts. I prefer messages because then you can test the components in isolation easily. And you can also do something very clever, which is to stub out all the components and test the overall system design before you get near the end of the project. And you pull them all back together and you have the solution. If a component turns out to be a problem, you pull it out and replace it. And this is, I think, at the heart of an awful lot of rapid development efforts. Now, you want to plan out not only the overall project, but what order you're going to build it in. It's just like when you're driving, you know your destination, and you also know what turn you're taking next. One of the rules here is see if you can get something up quickly. Stub out other bits for later, but the moment you can share the code with the users is the moment when they can tell you your head is pointed in the wrong spot, in the wrong place. That's much better to hear at the beginning of a project than near the end. This is basically agile programming. Even if you can't do full frontal agile, it's still a great idea. The most complicated sort of code that I've had to work on involves client server. You got your browser, you got your web server, you got your database. And this is one of my tricks for making this go up quickly. On the browser, validate the Ajax request before you send it. On the web server, check it when it comes in. On the database, check it, return standardized response. Don't invent the returns for each function. Validate the return on the web server, validate on the client. And you're going, well, that's like five or six validations. Yes, but this means you're going to catch the bug the first time it hits with high probability. The additional effort of writing this validation, particularly when you use library calls and classes to do it, is maybe 10 or 20%. One debugging session will pay all that time back. One nightmare debugging with the users are clawing down the cube walls. And this is actually how the FAA provides for airplane safety. There are a lot of systems that go into making a flight safe. No system is perfect, but if you have enough controls, automated pilot, cross checks, reports, it's unlikely the plane as a whole will crash. So I recommend this kind of thinking every step defends itself. In fact, each piece of code should defend itself. This gets back to a question in the chat thing. Here is a typical piece of code. Obviously, we're setting the social security number. We don't need a comment to say that. We know exactly what that first argument is doing. We've cleverly used the postgres typing mechanism so that we're sure to be in sync with the base type, whether it's an int or a big int. And we have a debugging flag wired into the program. So we can just turn on the debugging flag at a higher level. It'll default to false, but when set true, we're going to fire off like a notice right at the top of the function to say this is how we're called. That'll tell us this is where the flow is right now. If we raise an exception, we have each exception report where it came from. There might be a hundred functions in our code tree. And what value killed it? Make the code do the work. Well, I had one amusing problem with a user at Bellcore who didn't want my form to track his social security number because he felt Bellcore didn't have a right to that information. And I said, that's fine. And he said, well, that's why I'm putting in two different social security numbers or five different ones. And I said, don't put in any social security number you like. Just lie consistently. And we agree that that was acceptable. Oh, it's good. And this sort of trickery is especially valuable when you're under time pressure because it's easier to just type in the code. And under time pressure, most of us make a lot more mistakes. So now we get to the third kind of debugging, the most valuable kind, scientific debugging. You've got a piece, a code path, a data path, which is not working correct. Correctly, it looks like good stuff's coming in, bad stuff's coming out. Check the good stuff. Take notes. That's part of the science thing. There's a Chinese saying, pale ink is better than the best membrane. It's better to write down quick notes to yourself than to rely on remembering what you saw three hours ago. Have a guess as to where it's coming from. Just don't fall in love with your guess and put the instrumentation at spots where there's a 50-50 chance of the problem being before the instrument and after the instrument. Prefer binary search to linear. Cut the problem in half with each run. Don't start from the beginning and work through to the end. Okay, so we are now in the wrap up phase of this talk. What I have been recommending here is systematic debugging. Think about debugging when you get started, as you run the project and then after the project is done and you're coming back for maintenance. It is more work to wire debugging in upfront. No question about that. And there is no silver bullet. No amount of forethought on debugging will keep bugs from happening. There might be fewer stupid bugs and more intelligent bugs but you're still going to have them. However, the higher the quality of your code, the fewer bugs in the code, the more reliable the resulting end product. This is one of my favorite examples of an engineering screw-up. This is the bent pyramid. It's the second pyramid ever built. And the first pyramid had worked for the first ferrule. Great, wonderful monument through all times. Still a tourist visiting spot. And they figured, well, it was a lot of work on that first pyramid to dig all the way down to the basal underneath the sand. Let's build on sand. This is where the expression don't build on sand comes from. As they were building the pyramid on top of the sand, they discovered that when you put a lot of rock on top of sand, the sand turns into a slow liquid and the pyramid was beginning to drift. Negative comments came back from the Pharaoh's review team and they had to make the angle of attack of the pyramid shallower to avoid the liquid sand problem. If you have properly debug-thought-out code, you're building on basal. What can I say? If you don't, make sure your subscription to Too Much Coffee Man is current. Great strip because you're going to be needing it. So sleep versus coffee, the two ideal debugging techniques. And if your code has a good angle of attack, if it's solid, if you can go to a the latest bug very quickly, that's always a good sign. If you can say, hi, I know where it's coming from or just turn on the debug flag and see where the problem is. Then it's like you're building a coral reef. Coral reefs are incredibly beautiful, but not only good for the coral, but they support a wide variety of other plant life and animal life and tortoise and all sorts of wonderful things. And by having a good underlying structure, you can build an absolutely wonderful ecosystem on top of it. That picture, not by chance, is a brain coral. Okay, so last slide. Some of my favorite references on how to do this, St. Jerome, reading the Puskress manuals. You didn't know it was a database programmer, Jim's book on the Puskress serving language. And then a couple of books which I found helpful. You'll be able to see this in the recording of the talk. And you'll see what I've been saying, but in much more detail and more carefully worked out. Thank you very much for your attention. Okay, we don't have any more questions coming in. What I will say is that two more bugs were found on your slide. Great. Mohan says the city, different cases for the same city. And then Justin said, weather date should be timestamped with time zone. Weather changes throughout the date. You're both are absolutely right. And now I'm going to have to fit five bugs into my slide. Thank you. That kind of problem I like to have. Yeah, clearly a very astute audience here. So with that, I'm ready to shut it down if you are, John. Yeah, absolutely. I encourage you to send me an email with any follow-ups on this. I hope if you get a chance to review the talk or pass along the link to the talk to your friends. And as Lindsay said, I'm hoping to turn this into a book. So further suggestions of topics to cover and so forth. We'd be extremely welcome. Again, thank you for your time. And thank you for taking the time with us today, John. Thank you to our audience for being so engaged today. And as a reminder, we are taking the month of August off of doing webinars. We want everyone to go enjoy the little bit of summer that they can. And then we'll be back at it come September. So thank you all for everything. And I hope you have a wonderful rest of your day.