 Don't start without yourself yeah, I have a somewhat unusual background I started with a PhD in nuclear physics, but was never able to turn that into employment and so Through a variety of random walks. I ended up in the new medical school at McMaster Doing research in medical education pretty well unqualified to do so But over time I've learned a few of the tricks of the trade and over the last few years I managed to call myself a psychologist So nuclear physics that was that's quite a step Yeah, it was an easy step towards things like measurement and statistics because you had to know a little bit of mathematics to do physics but Learning how to be a decent psychologist and do decent research in psychology did take an awful lot longer and took a few very good Mentors like my dear colleague Lee Brooks yeah so part of the the research that That you do and that I Suppose in medical education that a lot of what it's based on is is categorization is diagnoses How do how do people generally diagnose? I mean it's really kind of a categorization problem, isn't it? So yeah, what what is what is categorization more generally and how do people how do people do it? Well before we get there, let me back up a second and give you an idea as to how we got into this mess I was originally hired to look at clinical problem solving and the idea was the diagnosis was not a categorization task It was really a matter of some kind of problem-solving skill And our initial quest was to go after some kind of general skills that people acquired as they became experts What emerged even back then in the late 70s was that in fact the general skill really wasn't very general At one level was too general and that everybody did it the same So that it didn't discriminate experts from novices at another level is too specific because success on one problem had very little to do With success on the next problem, right that led to thinking about well Maybe this isn't the way to characterize it at all and yes, you're absolutely right diagnosis is a form of categorization And I think for me personally the breakthrough was to work with people at McMaster who were really into what it's called concept formation or categorization and Some of the models about everyday categorization categorization really had enormous power in explaining medical diagnosis So to go back to your initial question Historically psychologists have been interested in categorization because it's the basis of communication. I Can't talk about a tree or a dog or a cat or a butterfly unless I have some senses And you have the same sense as to what the term butterfly stands for and so Categorization I think has a long history in psychology. I Think it's fair to say that initially the thinking was that there must be some implicit rules that governed each category But a moment's reflection on say a beanbag chair tells you that the implicit rules of chairs are unlikely to be necessary and sufficient Right, so what makes a chair a chair is normally someone would think Four legs, maybe wooden chair, but that that sort of rule might not be all Encompassing it might not define what everyone means by a chair I suppose no rule will encompass what everyone means by every kind of chair because you have to consider all sorts of possible legs Including none at all. Yeah, or cats or dogs or cats or dogs and so on I mean cats includes everything from house cats to lions and dogs include everything from Chihuahua's to Mastiff's And to come up with any general rule simply doesn't happen The two more prevalent views one is that essentially we average our experiences into a prototype And that prototypes are distinguished as having more of the features of the category and fewer of the features of other categories So we can play experimental games and find that a carrot is the prototypical vegetable and a robin is the prototypical bird and Penguins because they don't look like because they don't fly and they don't have feathers are very atypical So the older view is the prototype view that essentially our experience gets averaged into internal prototypes a More recent view and one which seems to play out really really well in medical diagnosis Is what's called the exemplar view that essentially says as you walk around the world you gather examples and every natural category as you learn you acquire examples and the the active Conversation or the active diagnosis ultimately amounts to matching the incoming information with some prior example and in memory That has many of the characteristics uniquely of the of the new stimulus So applying that to dogs and cats for example if I'm trying to recognize a beast that's running toward me It's probably not going to be on the basis of a prototypical dog or a cat but instead on the basis of The previous examples of dogs and cats that I've encountered before sure Glenn reger actually has a very nice example that he says suppose you grow up in the Yukon and all you see is Huskies How many Chihuahuas do you have to see before you recognize a trawl Chihuahua as a dog in the answer to one? And in fact, I I have a talk about this whole issue of medical diagnosis and one of the One of the key features and the one that grabs the audience Inevitably is the video of my year-and-a-half old daughter Where we show her playing the fish for priced toy with dogs and cats and BS skinner would be happy as can be because What she's learned is that if she puts that thing in that slot then mommy gives her a popsicle and tells her how wonderful she is But then we show her this an adult magazine a house and garden magazine Which contains a bunch of photographs and two of them have dogs in them and she points to the two dogs And what's interesting about that is that she can't say dog Right. She's a year-and-a-half old. She's pre-literate. Yeah, she can't say dog It is implausible that she would have a rule for dog Yep, and yet she can classify dogs already from what amounts to a fairly limited experience And probably helps that she has a family dog and they go to the dog park every day So she's got a few examples and that's all she needs to get by. Yep So it only takes just a few examples to be able to get the job done. It seems so. Yeah, obviously I mean the mystery would be what exactly is that similar similarity matching How does that come about and I think then you can invoke More fundamental models of the nature of memory which is Associationist or connectionist which basically says that we are somehow in a very rapid and unconscious way matching on individual attributes And and seeking connections that way That's far too basic science and psychology for me to play with but that's I think a pretty decent model Yeah, and so what about your doctors so when we're when we go from everyday classification or Categories like dogs and cats and tables and chairs And you only need just a few exemplars to be able to get the job done Did the docks work the same way? So if you're learning about skin lesions or mental disorders or More complex types of categories. Is it just take a few exemplars there as well? well clearly medical diagnosis is a bit more complex in part because the Yes, for skin lesions we can imagine a picture that's kind of like looking at a picture of a dog But for something like multiple sclerosis, it's much more abstract than that secondly self-evidently the stakes are higher Right, we don't want to make mistakes and so what you see coming about in the clinical encounter It's what has been called since the 1970s the hypothetical deductive method Which is within a few minutes or maybe even seconds. We don't know the timescale a physician will Advance a number of hypotheses as to what the diagnosis will likely be and then we'll go on a systematic search to gather information By and large to confirm one or another diagnosis Clearly what differentiates the expert from the novice is not how many hypotheses or how early But what are the hypotheses and expertise resides entirely in generating better hypotheses? Even going back to our original studies in the 70s We found that that early hypothesis generation was enormously powerful Basically, if you thought of the right diagnosis in the first five minutes your chances of getting there was 95 percent If you didn't your chances were 20 percent Wow Yeah, and that then leads to the fundamental question that's kept me interested all these years Where do those hypotheses come from right and exemplar models are a very powerful way of thinking about that sure that essentially prior experience is available to you and So the idea then is that you? Throughout your career as a diagnostician you're accumulating experiences of the sort yeah like like complainers or or Particular skin lesions and you develop a bunch of those experiences enough to be able to categorize new ones Efficiently without any conscious effort. Yes, certainly something like dermatology We've actually documented that a dermatologist looking at a skin lesion if he gets it right it takes them eight seconds Wow, there's not a lot of room for analytical processing in that process One of the a couple of games I play with clinician audiences quite often one is to ask them How long after you graduated before you felt you were competent? Now to the average person on the street You think that would be ruled out of order from the outset they're confident when we graduated them aren't they right? The answer is no The answer is routinely five years post graduation you put that with five years of training And then you say the ten years and ten thousand hours that everybody's that's now become common street talk yeah The second thing that's really intrigued me to illustrate the power of this method. This is the diagnosis of sick Now the emergency physicians are preoccupied with the diagnosis of sick This preeminent skill in the emergency department is to be able to tell the sick ones from the not so sick ones right and yet There's no textbook that has a chapter on how to diagnose sick There are no signs and symptoms of sick you might be too blue or too red you might be too hot or too cold your pulse Maybe too high or too low your blood pressure may be too high or too low and on and on and on There are a number of conditions that run to pages that are all considered sick conditions. Yeah But a very recent study actually just published this year showed that You turn to the emergency doc and say is this one sick or not? And he makes a judgment in seconds and that is about an 85% accuracy in terms of predicting whether ultimately when they get to the Wards they end up with a very serious diagnosis The doc with his experience can recognize sick from a mile away So you mentioned that process of categorization and diagnosis and so on is is fast and accurate We can do it very quickly Is that all there is I mean there seems like as doctors it seems that they should be able to I mean That's what they do in medical school isn't it is is learn the rules and and They're they're very careful and deliberative in coming up with these diagnoses Sure I think in every clinical encounter there's really two fairly distinct phases the initial phase of hypothesis generation which Is so effortless it just seems to be too easy And then a second confirmation phase where they go back to the rules the 29 causes of anemia the signs and symptoms of pernicious anemia And so forth just to confirm that that's what they're dealing with now Because it seems so effortless that first phase is often called just pattern recognition and it's kind of denigrated It's people sort of think that you know, you're not really doing your work You're not working hard enough and in fact that whole idea that that first System and it's called a dual processing model than that's called system one that that rapid generation of hypotheses There's a kind of a prevailing wisdom that that's bad for that that leads to errors and that what we should do is teach people analytical processes to correct those errors and that the second phase is actually the fundamental phase in terms of reducing errors This has been popularized a lot by a guy named Kahneman who's got a Nobel Prize psychologist He's written a book that I think many people know it was on the New York Times best seller list for at least a year called thinking fast and slow His claim is just that that essentially fast thinking system one thinking leads to errors And that we should be cautioning people doctors and everyone else to be more Slow rational thoughtful and that that would reduce the errors that happen in human judgments. I happen to believe that's wrong And I've accumulated a number a bunch of evidence to suggest that indeed that's wrong We encourage one group of physicians working through cases to go fast We encourage a second group to go slow and take their time We find that the group that goes slower indeed takes longer on each case But study after shoddy study has shown that the accuracy is exactly the same in both groups So the notion simply by slowing down and being thoughtful and analytical You'll solve the problem and the errors will go away seems to be simplistic at best Sorry, I think we should wait just like that for that high pitch. There we go So if the goal then is For accurate judgments, I mean we see this across a few fields where The ideal is to slow down take your time make sure that you're doing it correctly. Yeah It seems fairly counterintuitive that that would actually that wouldn't be as good as making a rapid judgment as that the fast sort of judgment that diagnosticians are making are as accurate as Diagnosticians or decision-makers who take their time. Is that really? I mean is that yeah, it's almost like there's a trick in an egg thing If I ask you what's 12 times 12 the answer is a hundred and forty four It's rapid and it's accurate if I ask you what 17 times 17 The answer is I don't know but I guess I can work it out The answer is actually 289 by the way only statisticians seem to know But that's an example where the slow process is actually more error prone than the fast process It seems I agree And in fact the I think it's safe to say that the majority view is that the slow process is going to be Ultimately higher benefit in the long run, but that's against the evidence This has really been captured in medicine where a book called the reflective practitioner by a guy named shown Was published probably ten years ago and argued basically that if every physician learned to be reflective in the course of their process And at the end of the day looking back on things that the world would be a better place It plays out particularly well in health professions because like any professional they are basically autonomous They have clients they make very serious decisions about those clients And they have no peers in general looking over their shoulder to see whether they're right or wrong So there is undoubtedly a need for physician for practitioners or any kind of professional to keep up to recognize their strengths and weaknesses The notion therefore that reflection being able to reflect on your process is somehow going to achieve that Has real cash right sure Unfortunately, it doesn't have a shred of evidence Right, so the opposite are people good at assessing their own ability. I mean it is There's I've read a little bit in in the field a colleague of ours Kevin Eva has done some work on on the role of self-assessment And it seems at least given his preliminary work that That we're not very good at it. That is we don't seem to be We don't seem to be very good at assessing how well we can do Various things is that is that right? You've published it's not there's nothing preliminary about it Kevin should be here to speak to it But I think I can pretty well paraphrase what Kevin would say. Yeah, in fact, I'll be more blunt than Kevin ever would Self-assessment sucks. Yeah. Yeah, I think the evidence is utterly overwhelming that people can't self-assess the way out of the paper bag Sure, in fact Kevin has a nice demonstration of this if I think of all the people who are watching this video I'm gonna ask them all a simple question Are you in the bottom half of driving skills? Hmm now half of you should say yes But having seen this but perform having done this in live time probably one out of a hundred will say yes Right half of you by definition are in the bottom half of driving skills, right? The evidence is absolutely crushing that people Cannot assess where they're at. They basically start with the premise that they're at about 70% and then go up and down from there Hmm You're better than I am, but I'm about 70% so you must be 80% yeah, it was captured by a Psychologist whose name I sadly forget, but he said it so beautifully. He said How can I know what I don't know? When I don't know what I don't know right and it's that simple Do you have no way of judging what the universe of that of that domain is and so your only guess is to say I guess I got 70% of it even experts Experts of course are the exception because they do know the domain Now at some level know the experts fail too, but in terms of judging your overall knowledge base Yeah, the people who know it all are the ones who can accurately determine that they know it all right But that doesn't help an expert judging his own performance on us on a specific case particular for example We have some evidence. Yes, actually we have a fair amount of evidence in medicine that Physicians perform better than chance and the more expert you are the better you perform But still their hit rate in terms of I'm confident in this diagnosis versus I turn out to be accurate in this diagnosis It's probably about 60% hmm. They're far from perfect. Wow. They're better from better than chance, but far from perfect. Yeah Yeah, interesting So how do they improve as an expert? You mentioned that we Going slow doesn't necessarily help being deliberative And there's this compounded issue of of self-assessment that we often don't know how good We're actually doing it any given case. I mean, what does that mean for improving practice? How do we get better? How do we? The the goal of this course that we're taking is called the science of everyday thinking So given your experience In in the field of expertise and in medicine. How do we improve everyday thinking? You improve by knowing more So just the accumulation of experiences Yeah, it seems so tantalizing it would be really nice if I could be very prescriptive and say well if you do this that and the Other thing then you'll be much better At some level, I guess there's got to be a bit of a germ of truth in that I mean this course wouldn't exist if we didn't think that there are some that being explicit about everyday thinking and The fact that the traps and everyday thinking Wouldn't help people think better and and to some level that's true, but that's generally Locking into what's been called general problem-solving strategies, which are not very powerful Sure, that's going to help a bit I suppose reflection is going to help a little bit, but it seems that all of these strategies To sort of generalize horrendously are good for about a 10% improvement. Okay It's not zero, but it's not night and day or black and white either Mm-hmm and very clearly the single best predictor of how good you are is how much you know about the domain not What problem-solving skills you bring to bear on it? We began there that was wrong Yep, and so then would you suggest in in getting more experience in gaining and kind of accumulating knowledge Is it just a matter of studying the domain more of getting more experience? So if I'm a if I'm a novice Diagnostician in order to become an expert in order to become an expert in the true sense is it just a matter of Working hard and studying the rules and getting as much exposure to a wide variety of Examples within that domain. Yes, we could but we can do better than that This really segues into a whole other branch of educational psychology The question is are there strategies we can do to optimize the acquisition of knowledge? Yeah, and again, we're talking about two kinds of knowledge formal knowledge and experiential knowledge Yeah, and we're now beginning to discover and I can't take any personal credit for this one This isn't my domain, but people like mayor Bjork Rotiger Have been working hard on taking models of the nature of of the mind short-term working memory long-term memory associative and Turning that into very prescriptive and powerful strategies to enhance the efficiency of learning right things like An obvious thing like mixing up the practice the examples from across multiple chapters so that you have to try and figure out Which is which it turns out to be an extremely powerful strategy for learning Yeah The idea of transfer which is being able to take knowledge that you've learned in one context and apply it to another One it doesn't happen at all as easily as we think it does but to Psychologists of a device strategies to make that happen better So I think this is moving much more into the instructional educational psychology into things There are things we can do to very much enhance the efficiency with which you acquire the knowledge You need to get the job done as a as a diagnostician or as a human All right. Well, thanks Jeff. That's a good one to finish Go away My name is Jeff. I think about reasoning