 I'm... That's me, Daniela Iperoncida. That's what I look like on television. I work for a company called D.V.O. It's a Swiss company, I'm very lucky to work for them. And we do cloud hosting for Django and Python developers who don't want to be system administrators. So, it's something I wish I had a long time ago. Felly, we also do everything on Django and Python. If you want to talk about Django and Python deployment afterwards, Please come and talk to me, I'll be very happy to talk to you about that at length. I'm also heavily involved in the Django community, I'm a core developer of the Django project. I'm a board member of the Django software foundation. Ac yn fwrth, lle'n gwrdd y gweithio cyntaf mewn hwn ar gynghach yma. Ond yn digwydd i'n gweithio ffraedigol o fynd i'n gweithio. A gynnaeth i fi ddim yn ei bod y rhani. Mae'n gwahogu i'r rhain, mae'n mynd i'n ddweud o fynd i ddangos o ddod y pwysig odevelopodd mwy o edrych. Ond o'r bufwyr yn fwy o mi, o'u cymdgrifent. O'r bhwysig i fi ddim yn gweithio i am gweithio eu cynnig i'i gweithio. Ond that's enough about mead, because I want to talk about tragedy and madness in stead. Because these are subjects that programmers need to talk about. You should know this picture. This is Peter Brugel, the elder, landscape with the fall of Icarus. You know the story of the myth of Icarus falling from the sky. ac mae'n dweud i'r ddweud yma yn y cwmniadau yma, dweud i'r ddweud yn y ddweud, ddyn nhw'n ddweud o'r cyfnodd yn ddweud ac mae'r gweithio'r gweithio'n ddweud. Felly mae'n gweithio'n ddweud y ffoel o Icarus a'r ddweud o'r ffair, agonwy a'r terer a'r ddweud o'r ddweudio'n ddweudio'n ddweudio'n ddweudio'n ddweud. Felly yna'n,. mae'n cwmniadau eu cyfrannu a'r ddweudio. Mae'r ddweudio'n ddweudio. Mae'n ddweudio. Felly mae'n deunydd i dda i'r ddweudio. Mae'n ddweudio, digonwyd yn ei ddadl. Mae'n ddweudio, dwi'n golygu systemau, mae'n ddweudio'n ddweudio amser o ddweudio. Mae'r drama er gweithio. Mae ddweudio'n ddweudio i ddweudio. Ie'r ddweudiau er gweithio aeryl a chydigon nhw'n fyddwch yn ei ddiddordebol. Bydd y cerddol yn fydd helpu i gyntaf i'r llos yn ymlaen. Mae'r tredydd yn hollwch yn fydd y cerddol i'r llos gyda'r hollwch o'r mynd i'r llos ychydig i'r llos. Yn ddiddordebol, i'r hyn o'r cyfrifwyr yn fyddwch chi wedi ddiddordebol, ond byddwch yn ei ddiddordebol yn ei ddiddordebol i'r hollwch i'r hollwch i'r hollwch. a'r gawis y ddweud i'w'r gweithio'r amddangos, neu i'w ddweud i'w pryd o'r ffordd a'r ddweud i'w ddweud i'w'r gweithio'u amddangos, neu i'w ddweud i'w'u ddweud i'w ddweud? Radddo'n i'w ddweud i'w trwy Icprus, ac mae'r ddofael a'r ddofael a'r ddofael. ond we still have the story to work with. So there was the time but we have the story of the time and in this talk I'm going to be talking about the story of the time as opposed to the time. I'm interested in what we can learn from the story of the time, from the record of the time. So what's madness? Well maybe you're familiar with this famous quote from Einstein that madness, the definition of insanity, is doing the same thing over and over again and expecting different results of Einstein. Actually it wasn't Einstein, it was the writer, Ruta Mae Brown who said that. I don't think it was her, it was Mark Twain apparently who said that and it might be an ancient Chinese proverb and actually nobody knows who said that. But if you're a programmer that probably sounds a little bit familiar this experience of doing something over and over again and expecting a different result because that's what we do, we sit in front of our keyboards run the same thing again and we're surprised and outraged when the same error happens again. And there's something slightly mad about this experience that programmers are familiar with. So what is going on what happens to programmers when they're afflicted by this madness, when they're sitting there in front of the computer, in front of their machinery doing the same thing again because the results weren't what they wanted or expected and they do it again and they look at it in this belief at the crash and then despite the evidence of their eyes all they can say is damn it this can't be happening. On the 31st of May 2009, FFF447 was flying from Rio de Janeiro to Paris and about three hours after take off it encountered icing conditions and ice crystals began to accumulate in one of the pitot tubes, these little forward facing tubes that measure air pressure and therefore air speed and this caused them to give inconsistent readings which in turn caused the autopilot to disengage and a warning sound went off in the cockpit and they were in some turbulent conditions so the pilot flying had to adjust the plane's attitude and as well as adjusting for roll he pulled back on the stick causing the plane causing the aircraft to climb and lose air speed and within about 60 seconds the icing event was over and the instruments were reporting correctly but the plane had climbed to its maximum altitude and it had lost air speed and lift and it was in an excessive nose up position and began to stall when it stalls the wings aren't supporting it any longer. The stall alarm sounded in the cockpit and the aircraft started to fall because it wasn't flying any longer and the pilots fought the controls down towards the Atlantic Ocean and they were trying to make sense of what was happening to them and over while Bonan kept pulling back on the stick trying to climb to recover altitude and not realising that what he was doing was keeping the plane in a stall. The captain Mark Dubois was on a rest break he was called back into the cockpit and he sat behind the two pilots and on the cockpit recording you can hear the stall warning going off about 80 times and finally just a few thousand feet above the water the first officer and the captain finally realised what Bonan had been doing and they correctly put the nose down so the plane could pick up speed and recover and it was too late and the plane crashed into the ocean and everyone on board 228 people died. So this story of Air France 447 has the same kind of hold on people's imagination as the story of Icarus. It has the status of a myth in aviation in which people think they're going to or hope they're going to find some kind of deeper truth or the revelation of a significant mystery maybe like the story of the Titanic that people also keep telling each other over again trying to find some meaning in it. How do we explain how an Airbus 330 one of the safest most reliable planes ever built fell out of the sky. There was no reason for it to crash the crew were not incapacitated the engines were working normally the plane responded to the controls there was nothing wrong with it apart from a brief sensor error and it functioned normally all the way down to the ocean and it can send a chill down your spine to think about it just for many reasons and one of the things that's chilling about it is that when we think about what happened in the cockpit it's like a glimpse into madness or irrationality and that's something that we find hard to look into. This is Laurie Anderson I don't know if anybody knows Laurie Anderson or this piece. She's a musician and performer and her piece from the air is the kind of story in flight emergency and it starts with the kind of calm authority that we expect from aviation and then it descends into demented confusion. If you don't know it you should listen to it not before you get onto an airplane necessarily. Now she makes the point that in an aircraft there are two times there's the time and there's the record of the time there's the time that we live through when something's happening and then there's the recreation of that time afterwards when we still tell the story of what happened. There's the time that belongs to the world now maybe of flesh and blood and fear and so on and there's the other time that belongs to a world of technology and numbers and data. There's the time when things happen and the time when we recreate the story. So our time now that we live right in now is contradictory, it's confused it's counted out in remaining years or in some cases maybe minutes and seconds but there's the time that's going to be pieced back together afterwards by the data recorders captured by the data recorders and put back together by the researchers and investigators who are trying to work out what happened. So this is the time and this is the record of the time and our time goes in one direction it unfolds inexorably and it comes to an end but the record of the time can be played backwards, can be played over and over again backwards and forwards we can slow it down and we can pause it. So in their time Air France 447 is gone but in the record of the time we can suspend the plane in its fall and in the record of the time the passengers are still there, unaware, asleep, the pilots are still fighting the controls, the passengers don't know that madness has taken over in the cockpit and that Pierre Cedric Bonin is doing the same thing over and over again and expecting a different result. So if you've been talking about madness and people talk about pilot error but these are really bad terms, they're simultaneously judgmental and unrevealing and aviation has a much better way of talking about what happens and it calls it loss of situational awareness. Bonin, Robert and Dubois lost situational awareness and it's what happens when a pilot loses the true picture of what's happening to them. When you lose situational awareness all the cues and clues you need might be staring you in the face but because of a contradiction with your expectations you will fail to understand what they are telling you. In aviation, needless to say, loss of situational awareness is absolutely deadly. It's a kind of cognitive breakdown and has a number of notable characteristics which we'll talk about later very briefly blind spots, faulty mental models misjudgments about cause and effect and poor decision making So the question I want to explore is does the same kind of madness, the same kind of cognitive breakdown afflict programmers and pilots when they make apparently irrational judgments and decisions? I want to look a little bit more closely at programming now to see whether we can find some deeper understanding of what programming is So some people think that programming is a science They're all wrong. Don't get confused by the idea of computer science it's not to do with programming, it's concerned with computation Programming is much more like engineering or flying a plane Programming engineering and flying a plane are crafts or skills or arts what in ancient Greece again would be called tekne literally meaning skill In fact if you read Plato one of the examples that he's always using of tekne of skill is piloting a ship Tekne of course is the root of our word technology and we can find various interesting parallels between programming and piloting so for example I don't know if any of you have done pair programming but it has its counterpart in the two pilot cockpit the pilot flying who's operating the controls and the pilot not flying or the pilot monitoring who is telling them what to do and handling communications and overall flight operations I'm interested in finding out some parallels that happen when things go wrong Now in nearly all crafts you'll find two phases or activities a primary and a secondary activity they correspond in programming to programming itself and debugging the creative phase on one hand and the troubleshooting phase on the other so in the primary activity it's synthetic we put things together in the secondary activity it's analytic and we break things apart so we've got assembly and disassembly in the creative phase we are moving forward in the secondary activity the debugging part we're moving backwards in the first we have imaginative and goal operated thinking and in the secondary activity it's logical and problem oriented in one side we ask questions like how can we get there from here and in the other we say well how do we get here or let's make something happen or how did this happen or why isn't it happening so that's the distinction between programming proper and debugging and this happens in all crafts but for programming it's quite unusual in how much time we spend on this half of the cycle this really dominates a lot of our work most of it will be in that side rather than this side and that's quite unusual and that's the part that I'm interested in not this part because we talk about this a lot as programmers we've got methodologies and models and frameworks and paradigms and a great deal has been said about the practice of programming about how teams work and how work can be planned and managed and executed and delivered and reviewed but about debugging and that's very little and that's odd because we spend so much time doing that I think there are a few reasons why this is the case so one is that programming is a creative craft like writing a book and in a creative craft it's acceptable for us to get things wrong many times until we get it right for example when you go to a surgeon you want to be sure that the surgeon has not only done this operation many times before but it's going to be conducted in exactly the same way with the same result same thing when somebody makes you a meal that's not the time when you want to hear that somebody is doing something groundbreaking and experimental on the other hand when you're writing a novel or composing a jazz masterpiece experimentation is fine failure is unarguable you don't get to argue about whether it worked or not because you're staring at a trace back and it means that we don't have time now to stop and think whether that really worked if you're not satisfied with your piece of writing or your painting you can sleep on it and think well maybe it does work or maybe the critics haven't caught up yet maybe if you're John Coltrane you can say no no it works it's good to catch up with me in programming you can't do that it has failed in programming trying again is instantaneous or nearly instantaneous and cost free and that means we're tempted to do the same thing again immediately whereas if you're an engineer building a bridge or an aeroplane that temptation is much less because it's so expensive to try again in programming feedback is nearly instantaneous and cost free so it means we're tempted to try things just to see if they will work you tend not to do that with aircraft, bridges and dams and finally and this is a really important one debugging is a mostly private affair there's the programmer and the code and the error and they're living in a miserable menageatoire it's a misery that we deal with it's not like being sick it's not we want to do it on our own and so debugging stays in our heads programmers love talking about programming but you will never get a conversation out of a programmer who's in the middle of debugging afterwards they'll say well there was this fantastic bug and I did this and then that happened and they'll happily tell you the war story but the only thing they want to do when they're in the middle of debugging is look at their screen and so we fight with our code alone so all these things come together and what they do is they tip the programmer more quickly into debugging and keep the programmer there they're a vicious cycle which forms a tighter and tighter spiral with no inclination to step back or reconsider it's a closed cycle in which we don't want to talk to other people about it and get away from the code programmers debugging finding ourselves doing the same thing over and over again as if we expected a different result fighting our controls and my argument is that the nature of debugging leads us towards cognitive breakdown that debugging itself provokes loss of situational awareness in programmers I'm arguing that the activity of debugging is responsible for producing the same kind of mental conditions cognitive conditions that tore flight 447 out of the sky you're a web you're a web application developer and you're trying to restart an application that crashed and you've checked the logs, you've flushed the caches, you've restarted the application you've restarted the web server and now you're logged in directly in the terminal inserting print functions into the code desperate for any clue damn it can't be happening and next thing you know you are in the grip of madness doing things that you would never dream of doing like changing production code on the fly or trying to reconfigure the web application gateway and finally you discover what is happening you've been doing all those desperate and futile things to recover from the crash on the wrong server you are typing the commands into the wrong terminal window you were desperate for clues and all the clues were right there in your face and you couldn't see them and you didn't see them because you lost situational awareness and you were fighting the controls or you're debugging a complex algorithm that was working until yesterday when you made some minor and insignificant changes and you know this algorithm inside out because you've been living inside it for a month and now somehow its logic is collapsing all around you and you take it apart piece by tiny piece and you're checking and rechecking parts of it and you know you can't possibly have anything to do with a problem changing it so much that your own mental model of it starts to break down and damn it this can't be happening and finally you realised that in one of the changes what happened is that the algorithm is now receiving a different data set you dived deep into a rabbit hole chasing after a completely irrelevant solution to your problem it was just receiving different data and you lost situational awareness because you were too busy fighting the controls and then you fly your plane into the sea telling yourself that this can't be happening those are amongst the last words on the cockpit voice recorder from that flight so loss of situational awareness is a common enemy of the pilot and the programmer what can we do about it when it happens to programmers that makes us look and feel foolish when it happens to pilots they die so unsurprisingly aviation has developed very good strategies for dealing with it to strategies to mitigate it and they target the kinds of cognitive breakdown that characterise it so remember these I mentioned them earlier so your eyes got a blind spot at the right distance if you close one eye you won't be able to see the other dot because it will be in your blind spot where the optic nerves leave the eye they're deadly blind spots you can't even tell when you have a blind spot the only thing you can do is get another perspective which is why we have two eyes or maybe invite another person in to look at it or change your own perspective we need mental models of the world as a kind of short hand for navigating the world and the more complex the world or the part of the world the more complex the mental model when they break down we need to rebuild them and we need to rebuild them from something objective that's not inside us Germans are very useful language it has multiple words for thing so one is ding as in thing and another is which means literally something like stand against and that's what an object is it's something that we can stand against it's something outside of us and we need something that we can stand against something that's outside ourselves in order to rebuild a mental model we don't have access to causality in the world we can simply observe the world and make observations and judgments and we make judgments about cause and effect we don't see cause and effect themselves in complex this is difficult enough at the best of times in complex stressful situations it's much harder so the flight computers alerted Bonan to the problem but when he pulled back on the stick and the attitude of the plane increased even further the stall warnings stopped because the flight computers were receiving such anomalous information that they couldn't even process it and when he let the nose recover which is what he should have done the stall warnings went off again because now the plane could detect what was happening so he was trapped in this horrific feedback loop in which his actions and cause and effect were 180 degrees out of phase with each other so we need to find ways of rescuing our judgments about cause and effect pulling them out of these complex and tight stressful loops and finally our decision making processes need to be exposed and made explicit had Bonan said what he was doing his reasoning would have been clearly and immediately obviously faulty as panic set in his actions became more and more irrational and the decisions he took the way he was thinking went unchallenged because it stayed inside his head and the rest of the crew made poor decisions too that were revealed in actions or implicit in remarks but they were never brought out into the light until the final moments so aviation has spent over a century learning how to deal with these problems lessons are woven into every pilot's training in aviation cognitive breakdown is dealt with through the employment of very simple practical strategies and procedures and it means that pilots don't have supernatural competency, supernormal competency they are ordinary people doing ordinary simple things and those simple things are what make them safe they use checklists when you're flying it doesn't matter how many times you've done a procedure even if you've done it several times that day you do it from a checklist so here's a checklist for a 747 normal procedures manual this will be on a card in the cockpit this is for the primary activity of flying, the programming part of flying the forward going part and it covers everything and they also have checklists for emergency procedures for debugging the secondary activity of flying there are checklists for absolutely everything, there are big manuals each pilot has one down by the side in the cockpit in the few moments after the autopilot disengaged actually Robert should have had the checklist on his lap working with Bonan and going through the checklist with Bonan instead Bonan fought his controls relying on his own memory, his skill and his experience so all those characteristics of cognitive breakdown a checklist serves as an alternative perspective to avoid blind spots it helps rebuild a mental model it gives us something to stand against a kind of objectivity it helps guard against poor decision making we use checklists often system administration sometimes in programming I've never seen a checklist used in debugging communication in aviation it's always explicit and verified you have the controls, I have the controls descending to flight level 350, descending to flight level 350 again it brings in the other perspective it rebuilds a mental model it helps reconnect cause and effect and exposes our thinking processes this is what happened in flight 447 and you can read this and weep because Bonan had been pulling back on that stick trying to climb and the other pilots did not know because he hasn't told them the left hand side of the cockpit had been pushing forward trying to lower the nose but the system was in dual input mode where it sums the commands and Robert hadn't made his decisions or actions explicit either nobody in the cockpit knew what was happening until it was too late we seal this all the time in debugging and it happens to me I can struggle with something for hours and hours and I describe the problem and the answer pops into my head because I've made it explicit and you don't need a co-pilot or IRC you can just tell it to the wall often the act of description will turn out to be 90% of the work towards the solution stress and panic drive us tighter into our loops of bad decision making one thing that aircraft like to do and do really well is fly an airliner wants nothing more than fly level and in a straight line had Bonan let go of the controls when the autopilot is engaged or almost any point in the next few minutes they would have been okay he could have done nothing and they would have all lived so it's quite good advice so simply doing nothing not only allows an aircraft to recover allows a pilot to recover now a pilot can't go on doing nothing indefinitely especially when for example an engines on fire but they can do it for longer than you think even in an unexpected emergency but programmers we can leave the code alone almost indefinitely if you go on holiday and come back through later that code will be waiting patiently for you to come back and tackle it again from where you left off if you walk away and come back and that's fine because there's no pressing need for you to do anything because often the best thing that we can do is get away from the code because when there's no pressing need to do anything then we should do nothing and I've solved more problems falling asleep or riding my bicycle or doing the dishes than I have by fighting with the controls and I've certainly caused less damage so we could learn to do these things we can adopt checklists how hard would it be for us to develop systematic checklists for debugging and programming we can improve the way we communicate so that we do nothing and even think nothing without communicating without exposing it to another person as we do for example in pair programming although we really do pair debugging and most urgently I think we need to learn is to stop fighting the controls so these are all ways of getting away from a failed failing perspective always stepping outside yourself and finding something else in the universe to get a grip on to help you their ways to reset your thinking and prevent bad judgments and decisions and they can work in programming they work in aviation we're talking about flight 447 not because it's something that happens it's because it's something that never happens a once in a lifetime anomaly when an airliner I can assure you is the safest way to travel in the meantime other disciplines like nursing and surgery and the nuclear industry are also learning and adopting these strategies and programmers love to think and talk about programming and about their practice and methodology and theory but they don't tend to theorise about debugging as though it might be an embarrassing private misery so as it said in an aircraft there's the time and there's the record of the time and the record of the time is scrutinised and replayed endlessly in order to force it to yield up its secrets so that we can learn from them and make things better and we don't do that in programming as soon as we've you know the last thing you want to do when you've had a horrific debugging session is look back at it you just want to get on with your life and work but we need to learn to dwell on our failures and confront them the way aviation does because we're so often caught up in the time and we won't look at the record of the time by the way these techniques also work for other things you know other crises and emergencies like a malfunctioning malfunctioning washing machine or relationship the truth is these things work in aviation they're built into it at every single level not because somebody heard about them at a conference talk aviation is safe because they're a defining part of its culture and processes the truth is as individuals we will only get so far they work for aviation because they're a systematic approach adopted throughout the industry until our debugging mistakes routinely kill us but with our customers I think probably our habits as programmers won't change very much our industry won't change and this is an industry this is just deep in the culture of the industry aviation paid for its lessons with people's lives people died to make it safe in aviation every time you break a rule however simple however banal every time you decide you don't need to follow a checklist every time you do something the way you happen to feel like doing it at that moment you are literally dishonoring the dead in programming it's not going to change thank you very much we've got a few minutes for questions let's have some questions maybe you want someone to make some comments but let's have questions first so we'll start at the front please to me it seems like your conclusion is necessarily pessimistic it seems like your conclusion is necessarily pessimistic so for example you changed my opinion that perhaps I should use checklists and that seems kind of like a good idea so maybe by talking more about it actually can achieve some progress in the industry what do you think we might you know I think that yes I am pessimistic and cynical but let's keep talking hi thank you for your talk it's very interesting do you think there's a role for professional bodies of programming similar to how doctors pilots and other professional professionals have you know their own body which regulates their behaviour their actions and enforces penalties within that their own industry conceivably yes and I think there is a kind of movement for this starting for example in academic and research software I think there have been one or two talks about this already at this conference and the keynote Catherine's keynote yesterday about ethics in programming also touches on that but look at us are we the kind of people who are going to or our industry does it look like a very regulatable industry I think one thing that's going to be really interesting look at the companies that are building self-driving cars Apple and Google when you've seen how Apple's programming works on like iCloud's synchronisation think okay you're going to be making self-driving cars that's interesting or Google maybe the regulation will come in not from the programming side but from the engineering side and the real world effects but you know I can crash my programs 10 times a day and nobody even notices so yeah thank you Daniel I am aware that checklists are actually used in programming interestingly in aviation software so you find there things like has the variable been initialised exactly once and this is because of regulations so I would like to re-ask the question are you aware or is anyone aware of checklists being used in Python it's interesting you say that and we sometimes use checklists in programming but never in the debugging part in that secondary activity and yesterday I was doing a workshop on my Divio Cloud workshop and I was thinking why the hell is this URL conf not working and I was literally typing it into the whisky.py file I was typing adding URL confs into the wrong file and finally my checklist has made sure that you're in the right file and that you spent several hours trying to debug something and as soon as you wrote it down you discovered what the problem was a few years ago I don't know if you're aware and Akamai engineer wrote the 15 minute rule in case no one else has ever heard of it it basically says try debugging for 15 minutes if you can't write it down and ask someone else I think it's great advice to live by at least then you only waste 15 minutes it's some Akamai engineer Akamai the CDN company the 15 minute rule you'll find a blog post by him about it it's a really great read in case you didn't hear that the 15 minute rule debug for 15 minutes and then tell someone else make it someone else's problem so thank you very much for your talk I think when I saw you putting up analysis and synthesis that maybe that fits very well to highly iterative processes and I think all these programming things that we're using like fail fast fail often that should lead us to doing more iterations more quickly and maybe your proposals here could help us working quicker on iterations and kind of like acknowledging debugging steps as kind of like in a very serious step of the activity that we're doing and not like kind of like you failed you have to debug but more like okay you have to do debugging now and to comment some aeronautical engineer working for Lufthansa so congrats on very good story that you told and very accurate and you should try making this into a keynote in my opinion thank you well thank you every time I get a trace back or a whatever error I'm surprised every single time you know really and so this is the kind of mentality we're dealing with how long have I been a programmer and I am still surprised that oh that really has to change I don't know why you know maybe my head's just a bit I should bang my head hard enough so the idea somehow gets in but there's a real block there about expectation of failure we never expect to deal with it the best thing the closest we've come I think is automated testing but that still doesn't help us with debugging first of all thank you very much for your talk I think we are programmers I mean as a programming role we don't have the most stellar let's say record of you know debugging things by methodology because I was actually reading a blog by a guy who started as a sexual liability engineer in some small like online casino or something that became one of the largest online casinos in the world and he became a head of sexual liability engineering group and I think we have a lot to learn from sexual liability engineers because they have a lot of methodologies how to debug right because they don't debug code they debug infrastructure right they debug services and well in this blog so I never really worked with the sexual liability engineers but in this blog he said that he spent a year going through all the tickets that were created like since the birth of the company right and failed something went wrong in sexual liability right and he classified everything that went wrong right and he created a huge pretty much like combination of like checklists with a flow chart of things that can go wrong and then like any juniors of sexual liability engineer can basically just go through it so what I'm saying yeah we don't have the best record but there are actually people right who have way stronger more robust methodologists and yeah I think we can definitely learn from our site reliability engineer colleagues because they can't just do nothing as easily as we can as programmers I think I'm also a convert to the debugging checklist idea excellent although I did want to ask about the overhead of constructing checklists I think we're in an industry that's unique and that no one that I know of has died from one of my tracebacks but is there an overhead to always using a checklist given how easily and cheaply we can fail? I don't know as I say my debugging checklist so far has got one item on it it took me 20 seconds to write and since I've started using it I've always forgotten to use it so it's just not in I was going to say in our natures it's certainly not in our training what the overhead might be probably less than the time we've spent futile debugging things do you have another question so maybe congratulations because your talk is a real success your talk is just a success