 I think it's about time to get started. And if people want to keep filing in, there are plenty of seats to go ahead and sit down wherever. No need to say in the back. So my name is Colin Fulton. I'm a front-end developer with Duo Security. If you want to reach me on Twitter, you can hit me up at peterquines. I don't really use Twitter, so if you actually want to reach me personally, just email me at justcolon at gmail.com. If I don't respond, just send an email again. Sometimes I forget to respond to things. And all the slides, as well as a number of other resources, are going to be up on github.com. They're all available now with my GitHub account, just Colin. And you can go browse those at your wish. Alrighty, so many of you may remember the time that NASA sent a rocket to Mars, and it crashed into Mars. And the reason why it crashed is because they had one piece of software that used pounds per second, while the rest of their software used newtons per second. And this was very embarrassing for NASA. But how many of you have ever done a multi-corporation project which takes years, if not decades, to complete with many people working on specs that are hundreds of pages long and not written a single bug in it? In this case, the bug was pretty bad because they lost a multimillion-dollar satellite. But still, I think we all can relate to this kind of problem. All of you may also remember when we initially launched Hubble into space, the images that it took initially were really blurry. They couldn't do anything with them. And that's because they had actually gotten the geometry of the mirror wrong. So they had to send up a number of astronauts in the space shuttle to go install, effectively, a pair of glasses to correct it. What a lot of people don't know is that they were actually able to measure how blurry the images were. And the glasses not only got them just as good as it would have been if they had initially gotten it right, they actually were able to correct for a number of other things and get sharper images than they would have gotten if they hadn't had this bug. This is an expensive way to discover a way to get a sharper image, but still, it worked. So today I want to tell you four stories about bugs in space programs because there's a lot that we can learn from bugs, and space programs are really cool, so the bugs are more exciting. I'm going to tell you right up front. I'm not some guru on top of a mountain who can pass out sage advice. I'm going to be editorializing a little bit, but I'm not going to be able to give you grand lessons in this talk. I'm just a workaday developer, like all of you, in fact. Many of you here, if not most of you, are probably smarter than me and know more about some of these subjects than me. But I'm going to tell you these stories that you can kind of reflect on your own experience and what you've done, and see what you can get out of them. And I'll provide a couple of little editorializing bits to kind of give you pointers and where I think that's interesting to think about these stories. So many of you have probably heard the phrase, we can send a man to the moon, which is usually said to be like, we can send a man to the moon, but we can't get a printer to work well. I don't know if any of you have ever done anything with printer firmware, but it's actually more complicated than you think it would be. And going to the moon was not something that we did perfectly. There were a number of bugs along the way, and this is just one of them. Specifically Apollo 14. And no, this isn't a typo, this isn't Apollo 13, the famous one that had the explosion in the exciting movies. No, no, this was a much more subtle, but still almost dangerous bug. Now, in Apollo 14, they managed to get to the moon and they were orbiting the moon. And it was less than four hours before they were about to land. And the crew on the ground noticed that the abort bit was set in the computer. That's the bit that gets set when you push the abort button, which the computer then checks and says, oh God, the abort bit is set. We have to abort whatever we're doing and get back into lunar orbit. Now, luckily they weren't in one of the procedures that would actually check the abort bit yet, and so was ignoring it. But what they ended up discovering was that there was probably a little bit of metal or a little bit of solder inside the abort button. There was just floating around and it occasionally randomly shorted, which meant the abort button was acting as if it was randomly being pushed every so often. This is fine while you're orbiting the moon, but while you're landing, it means that there is a certain percentage chance that you're just gonna have the rocket thruster turn on, whip you around the other way and send you back into orbit. And you only have enough fuel to land on the moon once because it's really expensive to get all that fuel into space. So this was a major problem, a really serious problem. So they contacted the guys who knew about the Apollo Guidance Computer, the people that actually written the code for it and said, how can you help us? And so they had about three to four hours to figure out a way to patch the solution using the Apollo Guidance Computer so that it would ignore this particular abort button. They also needed to make sure that there was another way to do an abort. They only had three to four hours to create that patch, test it, send it over to Houston where they would then test it again and then relay that patch up to the astronauts and manually punch in. Now, the problem is the Apollo Guidance Computer really didn't want to ignore that abort button for a number of different reasons. So this is some Apollo Guidance Computer source code. If you've never looked at assembly, this may look a little bit weird to you, but it's actually very, very similar to the assembly programs that are written today. In the first column optionally, there's a label which you can jump to. So you can think of this as just like a name that you can put there either as documentation or to say elsewhere in your code, hey, once you get to this point, jump over to this other part of the code. Then the next bit is what's called the operator or sometimes these are known as mnemonics. These are little tiny short strings which represent different commands. In this case, the CA there means clear register and then add on to it. Finally, we have the operator. This is the thing that you're gonna operate on. So in this case, we are going to clear register and then add in this particular flag to it which is stored. Then finally, there are comments which, hey, look, they use hash comments just like Ruby, that's pretty cool. So what does this line of code do? The first line clears out a register and then sets it to a word that's full of a bunch of different flag bits. Next, they use the mask command to mask out all but the let abort bit, the bit that actually gets used. So now all the other bits will be zero except if that let abort bit is set, that'll be one. Then there's a little bit of magic, they didn't have enough space to get all the commands they have to call this magic extend command but we don't have time to go into what that does. And then finally, there's a branch if zero command and so what that'll do is it'll check is the number that's currently in the register zero? If so, it will go ahead and jump to the land disk there which is another label in the program and if not, it'll continue on to do the abort sequence. So if the bit is set, it'll go ahead and continue with the abort sequence. If the bit isn't set, then they'll go ahead and do it. And so there's this let abort flag stored in memory. So let's just set the let abort flag. This'll be easy, we'll just go ahead and set it and then if we set it to zero, then this code will run and an abort isn't gonna happen. The problem is it isn't that simple. There are a variety of programs that get run that also change that let abort flag because they wanna set that flag during the landing to say hey, we're in the middle of a landing and that means we may want to abort. And so they would have to set the let abort flag immediately after the computer set it and that would mean that there was a gap of time between when the computer set it and when they manually set it that theoretically that abort button could short out and it would be a problem. Some of you also may be thinking that maybe we can just send up a patch for the code that will ignore the abort button but in this case, the code itself is woven with a bunch of metal wires by a bunch of women guided by a computer in Massachusetts. So if we want to patch the code, we would have to actually weave a bunch of wires together, package it up in epoxy, then send it up in a rocket and send it back to the moon and you can't really do that in four hours. So that's why we can't edit this code. It's very, very much read only, more so than we have to deal with today. So the solution that they came up with has to do with the next line of code that we have here. You'll see that immediately after we check the abort flag, we check that modreg there and that checks if we are in program 71. Program 71 is one of the two abort programs. The other is program 70. If the computer senses that it's already an abort sequence, it's not gonna go into another abort sequence because why would you abort and abort? So instead of setting the abort flag, we can set modreg. If we do that, then we're gonna go ahead and trick the computer that we're already in an abort but it'll continue doing whatever it's doing. This will solve the problem, except it's a little bit more complicated than that because there are other programs that check this particular register to see what program they're in in order to figure out what to do during the landing. So the sequence that they came up with was this. First you set the modreg to program 71. That tells the computer that you're already in an abort sequence, so it's gonna ignore abort after you set that. Then you have to wait until the ignition starts which is gonna automatically set the let abort flag but we don't care about that because we already changed the modreg register. Then you're gonna push the throttle up to full because the computer isn't gonna do that because it thinks it's in the middle of an abort sequence. Then you're gonna set the zoom flag which lets the computer know that hey in the future you can go ahead and throttle up the rocket because again it's checking these flags and we're kind of really screwing with the way the program is supposed to work. After that you can reset the let abort flag now that it's safe to go and set it back to zero because it ain't gonna be set again. That'll again cause it to ignore the abort button. Then we can go ahead and set the mode register back to program 63, the program that it thinks that it really should be in and that way it'll go ahead and check the zoom flag and throttle up the engine as it should for a landing. Then they can go ahead and turn the throttle back down and let the computer take control. If you're confused, don't worry. It's a little bit confusing and convoluted but all this put together meant that they were able to safely ignore that abort button and in future flights they actually set up additional routines to allow them to do this in a slightly less convoluted way in case this happened again in the future. Not about you but I think monkey patching is really great when you need it. You can fix amazing problems like this. If they didn't have the ability to manually go in and edit all the registers and memory while the program is running, they would have been screwed. They may have aborted midway through the flight. However, monkey patching will save you in situations like this but you only should really use it when you need it which is almost never. I don't know about you but I'm not landing people on the moon terribly often. Most of the time I have more than four hours to fix a bug so take it as you will. All right, the next thing I wanna talk about is how much damage one line of code can do. And the line of code was in the Ariane 5 rocket which is one of three heavy lifting rockets in the world. These are rockets which can lift tens of thousands of pounds into space. They are big, they are expensive, they are complicated and we don't have that many of them because it's not that often that we're throwing things that big into space but every so often they're needed. And Flight 501 was to be the first flight of the rocket. Now the European Space Union had spent pretty much about a decade developing this rocket in total. They had spent a lot of time carefully testing all their systems and making sure that this would work because this project in total cost them about $7 billion. So if something went wrong, that's a big loss. These rockets are disposable, they don't reuse them every single time but their first launch was gonna be an actual mission launching satellites into space. So if this mission failed, not only would they lose the rocket, they would also lose all the satellites that they were trying to launch into space. So Flight 501, initially the launch is going okay. Everything is relatively nominal. There are a couple little hiccups that happened but this is normal with a new rocket. Now less than a minute into the launch, the thrusters on the rocket suddenly gimbled to one direction and that caused the rocket to turn really suddenly. Now, rockets have a pointy end and you wanna keep that pointy end going into the wind because that's good because then the wind will just go around it. If the wind starts beating against to the side, the rocket is gonna fall apart. And so the rocket did start to disintegrate and there's a video of this. You can see bits of the rocket coming off from all the wind hitting it. And at this point, the rocket exploded which generally is a bad thing. However, in this case, it's a good thing because when a rocket like this is in a dangerous situation, it has a self-destruct sequence so it can just blow up. In this case, the rocket detected that it was falling apart and so it intentionally exploded to prevent even worse things from happening. It's bad when a bunch of bits of rockets fall to the ground from an exploded rocket. It's even worse when a rocket that is flying flies into the ground. Now the line of code that failed is this one. Now, there are a couple of problems here. I don't know Ada code and this is written in Ada and even though this is relatively easy to read, they didn't use the most intuitive variable or function names. I assume that if you worked on this rocket, this makes perfect sense to you. But the line that went wrong is this bit right here. And in this bit, they are taking a 64-bit floating point number and converting it into a 16-bit integer. 64 is bigger than 16. And they weren't checking whether or not overflowed. This isn't like Ruby where you can just add arbitrary large numbers together and it'll magically just figure out how to get those big numbers for you. Now, luckily, they had actually detected all of the possible overflows in their system because they're rocket engineers. They're doing actual rocket science. So they knew that overflows were possible and they had fixed almost all of the overflows. And the reason why they couldn't fix all of them is because they had made a rule that their computer couldn't work harder than 80% of its capability because this is a relatively resource constrained system so if they work it too hard, it's going to cause problems and then crash. Now, the reason why they didn't put the checks around this bit of code is pretty interesting. They had actually tested it with real data which is a good thing. Except they tested with real data from the Ariane 4 rocket which was a different rocket which had a different flight profile. So in the Ariane 4 rocket, this particular bit of code would never overflow. Except in the Ariane 5 rocket, the horizontal velocity which is what this particular number represent was much larger and thus it overflowed. Now, luckily for them, they had two copies of the computer running this and it was only the backup that failed first with this overflow. The problem is they're both getting the same data so immediately after, just milliseconds after the backup failed, the other system failed and so they had both redundant systems go down. Now, a team was formed to investigate exactly what happened and what went wrong and they got a number of different conclusions out of this. One is that the computer itself was assuming, or I'm sorry, one is that they were protecting against random faults. That's why you have two computers. If one happens to randomly fail due to like a bad capacitor, a bad solder joint, you still have the other computer and it can take over. They weren't protecting against design faults. They weren't protecting against the crazy idea that maybe there's a bug in their code and something went wrong. The committee that was looking at this instructed that they should really just assume that all falseware is faulty until it is correct. If you do test-driven development, this is kind of what you do. You assume that your code is bad until you have a test that can prove that it's right. So the first thing you do is create a test that can prove that that code is right. So then as you write it, you can check that. The other issue is that the computer had assumed that everything would work and thus when it had this error, it just stopped working and sent out an error message. And that error message was sent to the rocket gimbals which interpreted it as a movement command and thus it went over and blew up. The next thing that they said is that you should focus not only on developing good code but good documentation because a lot of the investigation that they did and actually figure out all these bugs and exactly what happened wouldn't be possible if it wasn't for all the documentation about their process that they provided. The final recommendation was to remove dead code. This is kind of the zinger with this story. That code was to calibrate the gyroscopes inside the rocket to make sure that they knew which direction was up. And so that program would run when the rocket was sitting on the launch pad so it was constantly calibrating those gyroscopes so when it launched, it knew what direction up was. But it was still running after the launch happened. Now, this doesn't quite make sense because after the launch happens, the rocket's gonna be turning so you don't wanna calibrate which way is up while you're actually turning the rocket. The reason why they kept it running in the Ariane 4 rocket, the previous rocket, was a very clever hack. They wanted to make sure that if the launch was extended for some reason, they had to delay when the launch was, they were still running that calibration sequence. They'd get it up and running faster. But in this case, this dead code caused a rocket to blow up. And so they said, okay, clever hacks are all fine and good, but remove dead code. It isn't doing anything. Your clever hack is very nice, but it caused a rocket to explode. And as a little post-script, the first launch of the Ariane 5 rocket was obviously not so good. But a lot of the subsequent launches were quite good. In fact, even through to this year, they are currently at a record of 81 successful launches in a row. They've had about four launches total in a history of almost 100 launches. This is a really successful rocket, and they did a really good job with it. And after doing this investigation, they fixed a lot of the problems in their systems. So the next thing I want to talk to you about is agile development, and agile development in the Soviet Union. Now we all know that the US won the space race, and we know the US won the space race, won because we're Americans, at least most of us are, and we know that America's the best. We put the first people on the moon, and we have the first people orbit the moon before that. Therefore, we won the space race. The problem with this is that it ignores the rest of the space race. Well, we did accomplish that, we definitely did do that. The USSR did a couple of things before us. They actually did like all of the things before us. They beat us at every step of the space race, which is why we didn't feel so good throughout it. So the question is, why is it that the USSR never managed to send a person to the moon? We got to the moon, and they beat us every step of the way, so surely they were better than us. The answer is that getting to the moon is hard, really hard. One of the hard problems is you need to launch a tremendously large rocket, because not only do you need to get into space, you need to get to the moon. So you need a rocket big enough to get from low Earth orbit to the moon, and you need to get that rocket into space, which means you need the largest rocket that humanity has ever constructed. Now the engine powering that rocket for the US was the F1 engine. This thing is an amazing feat of engineering. To give you an idea of how big and powerful this rocket was, it took in fuel at a rate such that the pumps that it used could drain an average size bathtub in about 200 milliseconds. That's a lot of fuel, and not only did they have this engine that was sucking in all this fuel, they had five of them on the first stage, five of these engines. Look at how big these things are. That's actually a relatively tall man standing next to them, and look at just how ginormous this thing is. This is an amazing feat of engineering, and the USSR lacked the industrial capabilities of the United States. They weren't able to produce engines that big and that powerful. They just didn't have the industry capable of doing it. So they decided to go a different tact. They designed the NK-33 engine. Now they knew they couldn't go big, so instead they went efficient. They decided to design a rocket engine which had as good thrust-to-weight ratio as possible, and because it was lighter but still delivered a lot of thrust, they could use more of them. It wasn't as powerful, thus why they had to use more of them. And specifically, they used 33 of them. That's a lot of rocket engines to put on the bottom of your rocket, and all of those have to work and work every single time. So not only did they have to make a very efficient engine, they also had to make a relatively simple and reliable engine. Now when the US was designing their F1 engine, they did a lot of design work. They were very careful in their designs that when they launched the rocket, they were relatively sure that it was gonna work. It didn't work all of the time, but they had to be really cautious because the press was constantly watching them. And any failure was another example of how the US was losing the space race. The Soviet Union, on the other hand, decided to take a more hardware-centered approach. They'd still do lots of design work, of course, but they were more willing to launch a rocket and just see what happens. If it blew up, you could learn from that explosion, improve the rocket, and iterate, much like we do with agile software development. And so they blew up a lot of rockets. What's okay about this from the Soviet Union's perspective is that they didn't have the free press of the United States, so a lot of their explosions were hidden, and because they were doing them out in the middle of, I believe, Kazakhstan, there weren't really any people watching, and so all those explosions could be much better hidden. We didn't find out about a lot of them until much later. Now, the NK-33 engine was designed to replace the previous engines used on the N1 rocket, the equivalent of the Saturn V rocket, the rocket that was gonna take the USSR to the moon. The first four launches blew up, which is okay because they designed the NK-33 engine to be a more efficient, better engine, and it was gonna be used on the fifth launch in order to get them to the moon. The problem is the fifth launch was canceled because at this point the US had already gotten to the moon, the Soviet Union wasn't interested in going to the moon anymore, and so the Soviet government ordered that all the NK-33 engines, which were tested but never actually used on a rocket, should be destroyed. The engineers didn't do that though because these were really good engines, they knew they were good engines that they had something special here, so they just hid them away in a warehouse, and they didn't let the government know about this, and it wasn't until the 1990s that it was revealed that they had this stockpile of old engines from the 1960s, and so they got in contact with some American rocket engineers from a company and said, hey, we have this really cool rocket that we think you might be interested in, and when they started to tell the US engineers about the numbers, the US engineers didn't believe it because this was an engine that had a thrust to weight ratio significantly better than anything that they had managed to design in all the subsequent decades. Not only that, it used something called a closed cycle technology. Now for those of you who aren't rocket nerds, I can't really go into the details of what this is, but it's an engineering feat so impressive that the US had deemed it impossible because it involves powering those pumps that pump all the fuel, powering them by effectively the same explosion that is running your rocket itself, which means you have really, really hot rocket fuel going through a turbine. Now you can cool it using the liquid oxygen which you have, which is really cold. The problem is your turbine is made out of metal, and oxygen corrodes metal, and heat accelerates that. So you have crazy amounts of heat that can melt through about three inch thick of metal like it's a candle, and you're also filling it with a highly corrosive substance. That's why the US deemed this impossible. The US got their hand on one of these rocket engines, they test it out and found that yes, in fact, in the 1960s the Soviet Union did manage to design this rocket. Imagine if someone came to you today and said, I have a computer, I have a whole stock pile of them, really good servers, the best Soviet servers you've ever seen. Not only are they more powerful than some of the servers that you have, they're also more efficient and use less power. We're also using a bunch of quantum technology that your physicists thought was impossible. Would you like to buy some? You would probably think they're crazy, but this actually happened, and they managed to sell them for $1.1 million a piece to the US. They use them throughout the 90s and the descendants of this engine are still used today. To this day there have only been about two or three rocket engines ever flown or designed that are more efficient than this particular engine. And yet the Soviet Union never made it to the moon. They had all this amazing engineering. They could beat us every step of the way, partially because of the sort of agile process that they did and being willing to fail things. But whether it's due to lack of political will or a number of political problems because there were some political issues within the USSR, as well as a number of things, they never made it. And that's really interesting that despite all this really good engineering, they never managed to make it to the moon. Now the last story I want to tell you about is about one bad byte. We've seen how much damage one line of code can do, but what can happen when just one little byte goes wrong? Specifically one byte in the Galileo space probe. There was a space probe which was designed and built in the 1980s, launched in the 90s, and in I believe the late 90s reached the Jovian system and studied Jupiter and its moons and produced some really beautiful images and really expanded our knowledge of Jupiter and its moons. Now Galileo is great. It did a lot of really amazing science, but there were a number of problems along the way. If you look up illustrations of Galileo, the early ones show this big antenna dish that's folded out on the top. And then if you look at later illustrations, you'll see it's just a little thing that looks like an umbrella that's just half folded. That's because the antenna dish didn't manage to open all the way. That's another story for another day. What I'm gonna talk about is the magnetometer. That's the thing that measured the magnetic field of Jupiter. That magnetometer broke on Galileo because there was one byte, just one little tiny byte in the source code which went bad. Space and especially Jupiter is a really punishing environment. There is a lot of radiation going on there. And so probably a gamma ray or something hit one of those bytes and just cause it to go bad and fry that little bit of the memory. So let's put yourself in JPL shoes. We're the people who design this particular spacecraft. So you have the code. You have the code that goes with the magnetometer, the code that has the bad byte in it. This is a good thing. So you know where that byte is and you also have all the source code so you can figure out what that code did so you can work around it. Now that code was written in fourth. I don't know about you but I wasn't familiar with fourth not that long ago and it's a relatively obscure programming language though it's not uncommon to see it being used in the space program. I'll get back to fourth later. Now it was developed using a development environment for the Apple II. This is a relatively old system. And the reason why I know that is because by the time this bug happened, JPL had gotten rid of all their Apple IIs. So okay, we have to fix this code and we have the code luckily but we don't have the computer that the code was developed on. Nor do we have the actual development environment that was used for generating that code and compiling it for that particular computer. How would you feel upon hearing this? You need to fix a bug in this code. Don't worry, we have the code but you don't have a compiler. Also we don't have any development systems. Here are the bytes, good luck. Enter Ron Garrett. Well the engineers on the magnetometer deem had deemed that this would be too expensive and too complicated to be able to fix just because of how much time it would take for them to recreate all the development tools required to fix this. Ron Garrett thought a little bit differently. He thought that he'd be able to do it in a lot less time and save JPL money or potentially save the magnetometer as they were gonna give up on this thing. He had a couple advantages on the side. One was that the magnetometer used an 1802 processor. This is a relatively simple processor and it's very orthogonal. It's very simple in the way that it works and that everything kind of works like everything else. It's very easy to figure out based off how one command works, how all the other commands work. Unlike modern day x86 processors which are really powerful but they're really complicated and inconsistent in how they're implemented. So the processor is relatively simple so writing a compiler for it isn't gonna be the hardest task in the world. Another advantage of the ad is fourth and I wanna take you a moment to show the fourth programming language. This is a valid line of fourth. I don't know about you but to me this looks like someone just took code and just sorted it so all the numbers are over here and then all of the other things are over here. The reason why it looks weird is because fourth is one of the very, very few programming languages to use what's known as post fix notation and the reason why it does that is because of the runtime that actually runs it. It is a stack based programming language. Everything that you do is based off a stack sort of like an array that you can only pop and push on to. You can't access into the middle of. And so if we run this line of fourth code we first get into the number four. The number four isn't just the number four of value it's also a command. It's a command that puts a four on the top of the stack. We then get to the number five which puts a five on the top of the stack then a two which puts the two on the pop stack pushing all the things before it down. Next we get to the plus command. The plus command takes the top two items on the stack pops them off, it adds them together and puts them back on to the top of the stack. Then we have the multiplication command which takes the top two items off the stack pops them off, multiplies them together and then puts them back on the top of the stack. Finally the period command takes whatever is on the top of the stack and prints it out to standard out or whatever your equivalent output is. Now this way that I wrote it is a little bit confusing and that's intentional to kind of show what postfix notation looks like. More often you might see fourth code that looks like this which makes a little bit sense as long as you just kind of flip the way your brain works. So we're gonna take a five and a two, we're gonna add them, we're gonna take the result of that and we're gonna multiply it by a four and then we're gonna output that. Now what about functions? Here is a function in fourth, specifically a function which squares a number. The colon there is just fourth syntax to say hey we have a function definition coming up then we have the word square, the name of the function. Next we have dupe. Dupe will take whatever happens to be on the stack when you are running this particular function and duplicate whatever is on the top of the stack. It'll just copy it and push another one on the stack. Then we take a multiply command, multiply those two numbers together, put them back on the top of the stack and the colon says they're at the end of the function. Now what's really cool about this and what some of you may have noticed is here we have a definition of a function and it doesn't have any parameters, it doesn't have any return value. In fourth you don't have any of that. You don't have variables. There are direct memory accesses but you don't have variables because you don't need that. Everything just operates on the stack. Not only does this make fourth very small, it also makes it very easy to write a fourth interpreter because you don't have to deal with all the complicated things with objects and arrays and having variables in your program or return arguments or lexical scope. None of those things are actually needed because of the way that fourth works. And this makes fourth very easy to implement on very simple, small space constrained and also processor constrained microcomputers. So Ron Garrett had these two advantages for him that this was gonna be relatively simple to do. He also had another secret which was that he was a Lisp programmer. Now Lisp programmers are often people who lurk in shadows and have long beards and speak in hush tones in parentheses. I don't have time to go into how wonderful Lisp is but it is a really amazing language and Ruby has actually pulled some stuff out of it. But one of the really great things about Lisp and specifically Common Lisp which he used is it's a fantastic language for just hacking something together. Because the syntax of the language and the way it's run is relatively simple in the case of a Lisp, you can construct programs that construct other programs or modify programs relatively easily. Metaprogramming. Also in the case of Common Lisp, the parser that parses your code before it evaluates it is actually something that you have control over at runtime. So in Common Lisp, you can write some Lisp code that changes the way that the Lisp parser works so that midway through your code you can start writing everything in Ruby syntax and it will totally understand that. I don't know of almost any other language that has that ability and makes it so simple to do that. You should probably never do that but in the case of this, you don't have that much time and you need to hack something together. And so Ron Garrett worked really hard on this. He did a lot of Lisp work and he managed to come up with a solution and I believe about a little more than a month and he did get some help from the magnetometer team at JPL. They helped him validate the program was correct and validate that his bug fix was correct but in the end, he managed to come up with a patch that could save the Galileo probe in not that much time working on his own despite all of the issues. Here is the actual patch that he wrote. I am not a fourth programmer and I don't know 1802 bytecode. If any of you do know anything about 1802 bytecode, please come up to me afterwards because you're probably an interesting person to talk to. What I can tell you is that up there first we have a function definition, then we have the compiled version of that function, then we have another function definition which is another one that you had to patch and then a compiled version of that. I believe looking at the code what he had to do was fill out the space that was in that bad bit of memory with some code that would never get executed and then take another function and clean it up so it's a little more efficient and space compact so that they wouldn't take over the extra space. And all this code as well as the tools that he used to do it are available on GitHub for you to look at if you wanna look at 1990s list code. I actually contacted Ron Garrett not that long ago to ask him about this. Unfortunately, he didn't remember too many things because this was 25 years ago and he worked on a lot of really cool interesting projects at JPL. But I did ask him for lessons and he did say that there was one lesson that he could take away from this project and that might be worth sharing with you. And that's that Lisp is really awesome. I mean, it's really cool that you're able to do that. I would like to add the amendem that Forth is really awesome and that you are probably now the largest audience at any RubyConf to know what Forth is. So congratulations. Now, I've told you these four stories. This is where I normally would give you some takeaways but I don't really wanna give you too much about what I think about these stories. I've already told you most of what I think. I want you to come up with what can you learn about your own programming styles, your own development, what you do at work, what you can take out of this. But there is one thing that I wanna mention. That's that history is really awesome. We have documentation of a lot of these bugs and what happens because of NASA being a public organization. They have to share a lot of this stuff but there's a lot that we can learn from in history. If we take the time to learn from other people's bugs, to learn from things that were studied 10, 20, even 40 years ago, there are really cool things to learn about computer programming, things that we still haven't explored yet as programmers and we're doing ourselves a disservice if we don't explore history more. So if you'd like to learn more about this stuff, I have a couple recommendations. One is to watch the talk, The Remote Agent Experiment which was a Google tech talk that Ron Garrett gave. He goes into more detail about some of the other list programming that he did at JPL as well as a lot of organizational problems that happened. I'll just leave it at that. I'd also recommend reading the book Digital Apollo even if you're not that nerdy about space. This is a program that is about the Apollo guidance computer and the design for it. There's a lot of really interesting information there about UI design work and also the politics and philosophy of designing a UI when you're dealing with a situation where you have a bunch of people who really wanna control the system directly and a bunch of engineers who say no, we want a computer to do it. It's a really fascinating history and it's a really grabbing read. The Apollo 11 source code, which is what I actually pulled from is up on GitHub if you wanna look at it and you can actually go through and I believe it actually has proper code coloring on it too which is quite impressive. You can go through and read it and it's really fun to look at all the little Easter eggs that are in there and if you're really nerdy and you really wanna understand it, I would recommend the book, The Apollo Guidance Computer. Well, Digital Apollo is really accessible. The Apollo Guidance Computer is a relatively readable book but it's not very accessible. It's hundreds of pages explaining how the Apollo Guidance Computer worked, how all the bits were arranged inside of it, all of the commands. It was an invaluable resource when putting together this talk and at the very, very end of it, he tells some stories about various things that went wrong and various bug fixes they did in the Apollo Guidance Computer but he doesn't tell from the perspective of the people who worked on it. He tells it from the perspective of all the nitty gritty about the bits that actually were moved about to do these fixes. Finally, if you don't wanna read or don't wanna do a bunch of talks, I would recommend looking up the Moon Machines Documentary Series. They have a number of documentaries and they're available online. Whether or not it's legal, I don't know but they're available on YouTube if you wanna go ahead and learn more about the F1 engines and the Sarin V rocket. They have another one, the Apollo Guidance Computer which is really interesting if you're a computer programmer. Also, I'd recommend watching the one on space suits. You wouldn't think space suits are interesting but I won't spoil it but you'd be surprised who actually makes the space suits that we still use in the US to go into space. A couple of special thanks. Thank you to Ron Garrett for helping me and sharing the fact that he had put up this code on GitHub. Also thanks to my friend Rachel McGuffin. She's the one who did all the illustrations that you did in this presentation and I think that this talk wouldn't have been as good if it wasn't for Rachel McGuffin and I'd like you to just take a moment to please give a round of applause for Ron Garrett for this amazing bug fix that he did because I don't think anyone's ever given him a really thorough round of applause for all that work he did and Rachel McGuffin for all the work that she did putting together these illustrations. Okay, so the Merlin Heavy Lift, sorry, this is gonna get really nerdy for a moment. He asked how did the engines compare that I talked about to some of the ones on the spacecraft that Elon Musk's SpaceX has been developing. I don't know if the F1 engine is more powerful. The F1 engine actually isn't the most powerful rocket engine ever developed. It's the most powerful single nozzle liquid propellant rocket engine ever developed. The reason for that is you see those two big white tubes on the sides of the space shuttle whenever it launched. Those rocket engines are like stupidly powerful like orders of magnitude more powerful than the one that I showed you but those are solid rocket fuel engines they're quite different. Specifically to your question, the NK33 engine the only ones more efficient than this are some theoretical engines that were never built, some derivatives of it and other engines designed by the Soviet Union and then the Merlin engine which SpaceX developed. I don't know about some of the details about the specific engines that you developed. I think they're not necessarily more efficient in terms of thrust to weight ratio but they have a lot of other really interesting characteristics and because they haven't flown yet they don't really necessarily count as real rocket engines until they get on a rocket and fly. Any other questions? Already, well it's time to go. If you have any other questions or you want just come up and see me I love talking about this stuff and thank you again.