 Good afternoon. A few people are still arriving here. I'm pleased to get to welcome you today to what will be the first in a sustained effort to examine and address something that's very, very important to me, my colleagues, many of you, which is the image of justice delivered and justice denied. It's really pleasing to me to get to roll out an inaugural event for a new series that the School of Social Ecology is putting on called Science to Fight Injustice. My name is Valerie Genus. I'm the dean of the School of Social Ecology, which is the home to two of the units that are sponsoring today's program, the Department of Criminology Law and Society, and the Department of Psychology and Social Behavior. It's also the home to the Center for Psychology and Law, which you'll hear more about today, under the direction of the distinguished professor, Elizabeth Loftus. That Center and Professor Loftus are the host of today's events, but it's co-sponsored by some partners, and I resist using the word partners in crime in an opening like this, but we have many partners that make this a reality, and I want to mention just a few of them. First of all is the School of Law at the University of California, Irvine, partner in this series in many other endeavors, as well as in this series, the California Attorneys for Criminal Justice Foundation, which is our primary sponsor for today's program, and today's program simply would not be possible without their sponsorship. This foundation has provided some brochures and other information about their services out front, and I would encourage you to please pick those up and get more familiar with the foundation. I'd like to pause and just give this foundation a round of applause for seeing the value in what we're going to talk about today. When I was growing up, my mother used to routinely impress upon me that justice is both an abstract idea and a daily proposition. Within that, I think most people would agree that justice should be fairly and predictably delivered, and sometimes it is, and sometimes it isn't. When justice is not fairly delivered, it harms all of us. It certainly harms the people impacted by that failure to deliver on justice, and it harms entire communities. In fact, I would go as far as to say when justice is not delivered, we all suffer directly and collateral. Just last Friday, as I was thinking about this series, the LA Times, which I'm sure many of you read and saw, front page above the fold reported yet another story of justice that was not delivered, although we don't know the full story yet. It reported on a man, Frank O'Connell, who was freed after being locked up for decades for a murder that he had held from the beginning that he did not commit. Here's how the LA Times described it, and this is a quote. Frank O'Connell sat in the same Pasadena courtroom where more than a quarter of a century ago he was sentenced to life in prison for a murder he insists he did not commit. In front of him, a new judge on Friday delivered the words he had long awaited. He could go home. Behind him, his relatives sobbed with relief. His lips trembling and tears in his eyes. O'Connell turned to look at his son, who was just four when a judge convicted him of gunning down a maintenance man at a Pasadena apartment complex. Nearby, O'Connell's mother blew kisses to him. Then he was embraced by one of the attorneys on a legal team that had worked for 15 years to secure his freedom. He said simply, I'm going home. As he walked out of the courtroom, still cuffed. The problem in this case, as reported by the LA Times, was a problem of withheld evidence and misidentification. The LA Times reported on both sides, the prosecutorial side and the public defense side. This and so many other cases for me and for many other people raises so many questions about how we deliver justice, how we fail to deliver justice, and the consequences for when we get it right and when we get it wrong. It's within this spirit that we introduce this inaugural event for the science to fight injustice. The idea with this program is to use science, the very product produced at the University of California, to help us make good on the promise of delivering justice to get it right. And within this series, we have a commitment to relying on research, the knowledge of practitioners, the criminal justice system, and the community coming together with the common goal of delivering justice, of getting it right. Today, I want to extend a special thanks to some groups in the audience who have shared with us a commitment to this particular goal. We have folks from the Orange County Public Defender's Office, the Los Angeles Public Defender's Office, the Orange County Alternative Defender's Office, and the UCI Police Department. Of course, over time as this series unfolds, we'll have more and more partners in the audience to help us deliver on the promise of justice. What we want to do here at the University is we want to provide a forum for presenting scientific information. We want to develop programs. We want to fund scholarships and activities that help examine justice and the criminal justice system more generally from police to courts to corrections. This to me is as bold a promise as it is important and timely. We're also undertaking this important, important initiative under the leadership of one of our most distinguished scholars at the University of California, Irvine, Beth Loftus. Beth Loftus, and I'll introduce her formally here in a minute, it was her idea to use what we produce, science, to do better on the delivery of justice. She had a vision for how we might bring people together and how we might partner to reach common goals, common to the system as it explains what it's doing, common to our students who have expressed interest, common to our researchers who are motivated by using knowledge to better communities and society. She had an idea for how we might do this. So this project that I'm going to describe and have been describing, and you'll hear more about tonight, is really the vision and the hard work beginning with a proposal by Professor Loftus to deliver on the promise. Now, introducing Beth is always a little daunting. One could go on and on and on, and you can see the audience fade because it's so much. So instead of watching you fade, I will tell you just a few things about our visionary leader tonight. She's colleague and friend, that's important to me. She's a colleague and friend to many of you in the audience. She is the School of Social Ecology's most distinguished professor, member of the National Academy of Sciences and many other clubs that acknowledge the best of the best. She's the author of over 500 published papers, and I think most importantly for our purposes here tonight, she's the person who envisioned this series, who had an idea about how to put science to work to fight injustice. It's with that I really revel in introducing Professor Beth Loftus. Well, Val, thank you so much. I wasn't expecting all that. I'm here really to introduce our inaugural speaker for this series. I knew that Dean Jenis would be talking and thanking some people, and I wasn't sure who she would thank, so I will not be redundant and thank only the CACJ. Thank you so much for your financial support and of course everyone else, but also importantly to Mickey Shaw who helped to raise the funds for this inaugural series and what we hope will be a continuing series, to Patricia, to Tony and Ellen, and to other staff who worked so hard to actually make this happen. To set the stage here, I do want to just mention that in this country there are some 75,000 prosecutions that rely on eyewitness testimony every year in the United States. This is actually a figure that I got from one of the papers written by our distinguished speaker tonight. Many of these identifications are correct, of course, but the kind of evidence is very convincing and so you'll hear something like, I'll never forget that face as long as I live, but at least 100 years of research shows that eyewitness testimony is susceptible to error, people make mistakes, and this does have huge implications for the legal system. Yesterday I went on the website of the Innocence Project and as of April 24th there are now 298, 89 cases of individuals who were convicted of crimes, crimes that we now know they didn't do because DNA testing has proven that they are actually innocent. Mistaken identification faulty memory is responsible for these errors in about three quarters of those cases. And this has had ramifications for people over 3,000 years, combined years in prison. These innocent people have spent many of them on death row. And this brings me to our speaker, Gary Wells, our inaugural speaker for this series, Science to Fight Injustice. I want to tell you a little bit about Gary. Gary, I think it's right you were born in Kansas, right? Okay, he was born in Kansas, I was trying to find that out. He went to college at Kansas State, and it might interest some people that he was an internationally ranked pool player, and that's how he paid for his college education. He then went to graduate school at Ohio State University. I found out some interesting things by reading a profile of Gary in The New Yorker and in this lovely profile that was written all about Gary and Gary's work, we should have sent it around. One of the things that I... In fact, I could probably say it much better. The profiler describes him as a blonde jeans and tweed wearing midwesterner. He was a 23-year-old graduate student, and many students are here tonight, and of course you like to know how people got into the area that they're working in now. According to the profile, a letter arrived to the psych department at Ohio State asking for some help. A lawyer convinced that his client was innocent, and that's what really got Gary thinking about legal cases and the role of science in those legal cases. He went on to earn his PhD at Ohio State in 1977 and is now a distinguished professor at Iowa State. One of his... I've known Gary since the late 70s, and one of the things about his interest is that he realized that the police are not going to be throwing out all eyewitness testimony, so he wanted to examine the aspects of the eyewitness situation that are under the control of the system, under the control of the criminal justice system, and he is quite famous for an early paper that he wrote in which he divided the various factors that can affect eyewitness testimony into two categories. System variables, variables that are under the influence of the system and under the control and estimator variables, and so system variables would include things like how do you structure the lineup, what kind of instructions do you give to witnesses who are going to a lineup or looking at a set of photos? Do you conduct those procedures with the investigator knowing who the suspect is or not knowing? He focused on these system variables and one of his rationales was he wanted to figure out some ways of preventing the mistakes before they even happen. He's also known for his work on the relative judgment theory and it's a process that witnesses use in making identifications that sometimes leads them to astray and to make mistakes and he has invented a number of ingenious ways of reducing relative judgments. It might interest you to know he is past president of the American Psychology Law Society. He's won Distinguished Contribution Awards from APLS and also the American Psychological Association he has an honorary doctorate from John Jay College and he is one of the founding members of a group that put together by the U.S. Department of Justice to develop a set of national guidelines for eyewitness evidence. That was a group that consisted of prosecutors and defense attorneys and researchers, police officers put together by then Attorney General Janet Reno and they produced a guide for law enforcement and I will close this introduction by just telling you a little bit about what I read in terms of a profile of Gary in a wonderful book by the attorney and law professor James Doyle called True Witness which I heartily recommend to you. Like most people who came of age in America in the 60s he describes Gary. Gary wanted to see justice done. He was not a radical, he had a sturdy Midwestern skepticism about movements and causes. He was a scientist who put science far above politics but he was also one committed to the idea that psychology could play a role in improving the world. Wells even had a vision of how it could be done and he's going to tell you a little bit about that vision today. So let's welcome Gary Wells. Sorry that's a little bit of an inside thing there. Let me just skip through some things that we're going to be in front of this. So thank you Beth. One of the things that was not in the New Yorker article but it's no secret, I tell people all the time that I was also greatly influenced by Beth Loftus while I was still in graduate school and before I did any of this work she paved the way. She was really I think the first person to show that you can do this real world, high impact, relevant stuff with regard to using science to address issues that do relate to justice and publish that work in the most rigorous scientific journals and I really think that that is a contribution that Beth made that has influenced scores and scores of people. So I want to talk about using science to improve the accuracy of identification and by the way I'm very happy to be on this campus because it's such a powerhouse in this domain. What better place with the people who are here than this campus to be addressing how one uses science to advance justice. Number of acknowledgments. The Wendy and Mark Stavish endowed chair is something I hold now. I always want to thank them because it's such a great situation for me, helps support me in a variety of ways. National Science Foundation has been funding my research consistently for especially the last couple of decades and then lots of former, current former Ph.D. students who have been critical in this work. Well, we all know what a lineup is. It's really permeated our culture. Everyone is familiar with the idea of the police lineup. It has permeated, you know, you see it in movies. You see it on television programs. It pops up in more cartoons. I have well over a hundred of these kinds of cartoons. It seems like it's very easy to do lineup cartoons because there are so many. Here we have a baseball trying to pick out what assaulted it, made a little more difficult by the fact that the bat is wearing a disguise. Some of these have certain elements of truth to them. You know, it is the case, very reliable finding that it's harder to recognize someone of another race than it is of your own race. That's the cross-race effect. Apparently, according to this cartoonist, there's also a cross-species effect. And just when you start to think, this is fairly recent, when you start to think that you've seen these introductions to the Simpsons, keep your eye on this. Wow, I think I could have left that last part off of that. This is my favorite lineup. This is an episode of Seinfeld in which, keep your eye on Kramer, he was asked to be a filler in a lineup. What's this? So he's just a filler. But of course there's a serious side to this. Now of course there's long been skepticism about eyewitnesses from people both inside and outside of the legal system. Voltaire said, he who has heard the same thing told by 12,000 eyewitnesses has only 12,000 probabilities which are equal to one strong probability which is far from certain. I think I like that quote. I'm not sure I completely understand that quote but I think he was expressing a certain skepticism. The last Supreme Court justice though to ever say anything really much about eyewitnessification was Thurger Marshall who said the annals of criminal justice are rife with instances of mistaken identification. And probably the only way he could have known at the time maybe to have read some of Ron Huff's work because it's really been within more of the last decade or 15 years that we've been able to come up with maybe somewhat more definitive proof of that. Well this is about science and justice. I'm going to go straight into some of the science part here and talk about the idea that, look, what does it mean? How do we study something like eyewitness identification or eyewitness memory scientifically? Well this is pretty straightforward really and so I just want to run through this so you kind of know where it's coming from in case you don't already know but I mean any student who's well trained in any aspect of experimental psychology can create this kind of paradigm. What we do of course is we create events for unsuspecting people and because we created the events we know exactly what happened. We know who the perpetrator was it's one of our people. We know anything that was said any actions that were taken and we do this then over and over again for large numbers of witnesses so that we know that what we're looking at is something that applies to people in general not just a few people that we tested we can then do things like show them a lineup and when we show them the lineup again the beauty or the power of this is because as happens in science when you take control of these variables when you create the very phenomena that you want to study we know who the perpetrator is again it's one of our people or whether the perpetrator is even in that lineup right so then we can have people attempt to make identification decisions we can score them for whether they're right or wrong we can also measure other things like how long it takes them to make the decision or the certainty with which they make their identification so we can study what's the relation between certainty and accuracy and so on and then within that paradigm that basic paradigm we can go in and begin systematically manipulating things like the nature of the witness event the characteristics of the witnesses young, old, black, white, male, female and so on instructions given to witnesses prior to viewing a lineup to see what kinds of differences that might make the type of lineup that they view we've created alternative approaches alternative kinds of lineups and tested those out to show what we think is a better approach and the behaviors of the line administrator which turn out to be a big issue one that's been largely hidden and yet it's a very important one so that's the basic paradigm and pretty much anything that we want to know we can then study within this kind of paradigm and now we have after these years all these years a quite large literature and it's so large that really all I can do is touch on it today but I'm going to touch on things that I think are among the most important of the observations and for other reviews of the broader reviews of this literature there's the article that we published in Psychological Science in the Public Interest a few years ago which I think is a pretty good review Beth and I have what I think is a very good chapter this is the second version of this chapter in second edition we did an earlier version that is very broad covering a broad spectrum of the eye witness domain not just line-ups, not just identification and then I think that especially those who are in the courtroom trying to deal with these cases of eye witness identification I think the Wells and Quinlivan Law and Human Behavior article we published a couple years ago is very important because it shows why from a scientific perspective you're going to tend to not be able to win these cases and you're going to tend to not be able to win a suppression hearing because the way that the law works on it is flawed it's not taking into consideration the nature of the science anyway, now eye witness identification research by applied experimental psychologists which I guess is what I am I was actually trained as a social psychologist but I would call myself I guess more an applied experimental psychologist has now had a huge impact on the legal system I'm going to talk some about that later but this has not been true for very long the legal system was largely dismissive of this science until really late in the 90s early in the 2000s and it was only after forensic DNA testing came along in other words we were doing all this work we were doing the science we were publishing this work and making a case and blowing a whistle and saying there's a problem to be addressed here but the legal system was largely for the most part dismissive of it but then once forensic DNA testing came along in the 90s as many of you know it began to change things and we began looking at these DNA exoneration cases we published an article then in 1998 that for example examined the first 40 DNA exoneration cases because we ought to be interested in knowing so these are cases where people were convicted of crimes they did not commit forensic DNA testing came along and proved that they were in fact innocent we ought to be interested in knowing as a culture and as scholars of the legal system what went wrong and it turns out most of the cases are cases like that of Kirk Bloodsworth Kirk Bloodsworth had never been in trouble with the law in his life he served his country well in the U.S. Navy he was just going on with his life but he was convicted of murder and rape in 1985 he was sentenced to Maryland's death chamber and spent time on death row but on a technicality it got reduced to a life sentence and then the other number there, the slash nine that's the number of years he served in prison until forensic DNA testing came along and proved that he was innocent now what we ought to be interested in is what we've created here this was in the original article in the right hand column the evidence producing the conviction and in Bloodsworth's case five witnesses mistakenly identified him now if you look down this column you'll see that with only a few exceptions there's always something else like in Bloodsworth's case self incriminating statements now think about that how can you make self incriminating statements when you're innocent but what happens is it's the eyewitness ID that's driving this and then within the context of that even ambiguous statements by Kirk Bloodsworth get treated as self incriminating and so things like proximity of residence weak alibi these things are brought in but they don't mean anything in and of themselves so what's driving this basically is the identification evidence now this should not this does not come as a surprise to Ron Huff who well before any of this was identifying as causes of wrongful conviction prior to DNA the strong role that mistaken eyewitnesses played but the DNA stuff somehow has a lot more definitiveness and punch and you don't have to look at all the kinds of nuances that Ron Huff discovered about you know so innocent right and you get people sometimes wondering well you know so the DNA became much more definitive and it moved things along pretty well there now as I think Beth pointed out 289 fully complete exoneration cases 219 of which are cases of mistaken eyewitness identification average time in prison before being proven innocent 13 years range 3 months to 35 years many were on death row these are pictures of just 99 of those 200 and some mistaken eyewitness identification exonerations I have had the good fortune of meeting a large share of these individuals not all of them but and certainly not for example Timothy Cull who was in fact who died in prison before he could be exonerated but there's Kirk Bloodsworth right there that I was telling you about before but here's what's important I think to keep in mind this can only be a small slice of the cases a small slice follow this with me because you can talk to any innocence project and what you find is that by the way which are fairly new phenomena themselves when I started off in this area when Beth started off in this area there were no innocence projects there was no innocence conferences there were very few who believed there were innocent people in prison it was just amazing but this can only be a small slice of the cases of DNA exonerations so let me tell you why so do a little bit of math in your head and I'm going to end up giving you some conservative statements about this even because when you go back to these cases so these are people all who were convicted prior to the advent of forensic DNA testing and then they appealed to various sources to try to get their cases heard and what you find is that when you go back to those cases as any innocence project will tell you well the biological evidence wasn't collected sorry oh well the biological evidence was not collected properly has to be done properly sorry or the biological evidence was destroyed sorry because after all there were no laws in any state that required preservation of the biological evidence after conviction some a few places kept it right or the biological evidence deteriorated heat light our enemies of biological evidence it has to be stored properly but here or the biological evidence was lost if you've ever been to certain evidence rooms around the country you know that you know it's just a mess right but here's the big category there was no biological evidence you see when I went through those DNA exonerations I don't know if you noticed what the charges were I mean in some cases murder robbery blah blah blah but there's always sexual assault involved there are a few exceptions but basically the DNA exonerations are sexual assault cases why not because sexual assault victims are bad eyewitnesses they're probably the best eyewitnesses we have in some ways I mean they get a much better look longer look and so on closer look than in a 7-11 robbery or whatever but because that's where the DNA is right so it turns out that only a fraction of cases a small fraction can be solved with DNA tests because most serious crimes do not leave behind definitive biological evidence so for all the reasons I gave you on the previous page most sexual assault cases there's no biological evidence to be tested because it wasn't collected it wasn't preserved it deteriorated destroyed whatever right so this is a small fraction and it's a small fraction of cases very small fraction because in fact it's rare to have any DNA rich biological trace evidence for murders for muggings for burglaries how excited is this guy going to have to be robbing the 7-11 to leave behind that kind of evidence not going to happen so we're still very much dependent on eyewitnesses and so it's important to look forward and say how can we prevent these mistakes from being happening so only a fraction of cases can be solved this way this can only be a very small slice of the cases because in fact we estimate that fewer than even going forward 5% of eyewitness cases have DNA trace evidence that could possibly than trump mistake of an eyewitness identification and it takes something pretty much as definitive as DNA to trump an eyewitness persuasive it takes something almost that persuasive to trump it now there are lots of reasons that we can talk about why do eyewitnesses make mistakes why are eyewitnesses so potentially unreliable and a lot of it has to do with witnessing conditions and so on and so forth but they're not that great a lot of it has to do with the fact that it's a very unusual situation for people to be in they have no practice at it but one of the things that I described relatively early on and Beth Loftus alluded to this is this idea of what I call the relative judgment process that is when you get down to looking at what happens when witnesses view a lineup okay in general and it took me quite a while lots of experiments to sort of realize what I was looking at you know sometimes the simple and you'll see this is very simple sometimes the simplest things are the hardest things to discover right because they're so simple you're looking for something complex and the simple thing is right in front of your eyes it took me a while but basically the idea is this that when I witness a view of a lineup I tend to select a person who looks most like the perpetrator most like their memory of the perpetrator relative to the other members of the lineup and that's why I call it a relative judgment process right now usually when I tell people about this it's sort of like yeah so what what's wrong with that right and so let me tell you what's wrong with it let me point out that this comparison process and homing in on who looks most like him and identifying that person can be problematic so here even though I had shown this in much more complex ways before this is a somewhat later experiment that I think illustrates the problem with relative the process of relative judgments and the problem with relative judgments so what we did was we staged a crime 200 times for separate witnesses and then for half of them randomly determined they individually viewed this photographic lineup by the way most lineups are done with photographs it's relatively rare to have live lineups today and 54% were able to pick out the perpetrator now nothing magic about 54% right that number goes up or down as a function of all kinds of of variables how good of a view they got how long between the time of the crime and lineup and so on what's important as it is in general in science are patterns we observe patterns and what I'm going to show you is a pattern a very repeatable pattern so 54% pick them out now notice down here and I just underlined it all witnesses were warned that the actual perpetrator might not be in the lineup we're going to come back to that point later it's very important to do it real perpetrator might not be here and in fact looking down the lower right end corner 21% made no choice 21% said I don't think he's there or I can't be sure enough to identify him now what we wanted to know in this experiment was now what happens we take the perpetrator for the other 100 of these witnesses also individually tested eliminate him from the lineup replace him with no one where does the 54% go because it has to go someplace these numbers have to add up to 100% it has to go someplace you can't pick number 3 there's nobody in that slot so one possibility of course that 54% on all these witnesses were also warned that the actual perpetrator might not be there same instruction one possibility is that 54% slice down here joins the 21 so now you have 75% making no ID right but you know that's not what happens and you know that because otherwise what am I telling you about this and also because I told you we talked about relative judgments right so it's not 54% that slides down there instead in this experiment 11% slide down there and the dominant tendency is to go to the next best guy so now his jeopardy goes up dramatically that's relative judgments right and now there are a number of important really important implications of this relative judgment process and despite how long this has been around we still discover new implications of it all the time but the problem with the relative judgment process then is that some member of the lineup is always going to look more like the perpetrator than the remaining members of the lineup even when the actual perpetrator is not in the lineup now well the real perpetrator is not in the lineup isn't that sort of a trick and it's not and in fact and I'm going to come back to this point at the very end of my talk there is there's no there's no law, there's no rules about whether you have to have evidence against anybody to put them in a lineup witnesses are viewing perpetrator absent lineups all the time it's just a detective you know maybe Joe did it okay put Joe in a lineup well if Joe didn't do it he's looking at a perpetrator absent lineup now there's jeopardy among those people in that lineup so we don't want witnesses really using relative judgments in quite this manner but given that they do we have to take account of it so this is from 60 minutes actually Beth was on this 60 minutes too and this is Leslie Stahl and this is something that aired it's just a little portion of it and I had spent time with Leslie Stahl going over everything that I talked to you about up to this point talking about the great difficulty is detecting you know so difficult to detect the absence of the perpetrator and that's a real problem I talked about relative judgments she's a very smart person she understood it completely right and then we got to this point so let's start this and of course you're particularly cautious right now you know now after we've talked probably not to pick anyone no no actually I actually know who it is because if I come upon that I think it's this guy am I wrong? am I wrong? okay so there you go it's not it's so and you know that you know that you know one of the one of the great lessons I think of scientific psychology is my own belief is that when something comes very naturally to people it's just a part of how we process our world or whatever do you think we do I think we're relative judgment processors you can't just tell somebody to do otherwise sometimes you have to do something else which I'll get to in a minute but I want to show you now here was the line here were the people in the lineup that she saw and I want to point something out here here's the actual bomber on the rough here's the person she picked right which is the most popular pick now I'm going to put them side by side here right all I did when I put together this lineup which I just use as a demo but I is six males you know in their 20s white males with no facial hair and short dark hair that's it I didn't try to find a clone in fact there are no clones in there this is the closest guy this is the one she picked now why is that is it because they have exactly the same nose not even close oh maybe it's the same eyes no oh the same mouth no right and in general it's interesting that in the DNA exoneration cases with only a few exceptions when you get a hit on who the real purpose he's not a dead ringer for the guy they picked he's not a dead ringer for the guy that the witness set on the stand and said I'm absolutely positive about right these are not for the most part cases of coincidental resemblance right there cases of relative judgment in other words what happens is well this person looked more like them than the others in the lineup just relative in that sense right well there are lots of important implications of the relative judgment process one of them is that this is why it's so critical to give what we call the pre-line up admonition that is warning the witness look the real person might not be present you know the correct answer might be none of these now that doesn't take care of the relative judgment process completely it helps you know in the study that I showed you before with that removal without replacement we gave that instruction and they still made relative judgments right but if you don't give that instruction it looks even worse than what I showed you another implication is this is really the reason why the fillers you know fillers in a lineup are these people who are just in there to fill it out they fit the general description you put them in there you're trying to make it fair right the fillers in the lineup need to fit the witness's general description of the culprit because if the if the only person who fits that description is the person that you are testing might have done it and the other people don't well then guess who's going to look more like a perpetrator relative to the others for example in this Oklahoma lineup where the witness described the perpetrator as a white male slight build early 30's dark hair just over the years clean shaven about 6'2 that was the witness's description and as you can see here you've got that's not that's not hair just over the years that's not clean shaven or hair over the years that's not hair just over the years or clean shaven so it's between these two guys right and look how they left the height lines in the background at 6'2 for this guy right who was their suspect I don't know if he did it but he's the person certainly he got identified no DNA in this case we'll never know for sure but now another implication of this is again the idea that well if witnesses are prone to this relative judgment process what can we do about it and I said before I don't think you can just tell people I mean we've tried that don't do that it's just too compelling Leslie Stahl could not hold herself back from doing it even after I was talking about it for quite a while so one of the things we did was we created the idea of a different kind of lineup called the sequential lineup and the idea of the sequential lineup is to try to reduce reliance on relative judgment that was the theory behind the sequential lineup and so in introducing the sequential lineup first of all we had to like decide okay and this was myself and one of my PhD students at the time well what do we you know we need some terminology here we had to first of all name this which is now called the simultaneous lineup the traditional police lineup is referred to as a simultaneous lineup now whether it's done with the live lineup or photos they're all presented simultaneously and you recall that I'm saying that a big problem here is relative judgment who looks most like the perpetrator comparing one to the other and bringing it down to somebody like that which makes a witness sensitive to whether he necessarily the real perpetrator is there or not so the sequential lineup is a very at one level simple variation what happens is you tell the witness you know Mrs. Jones I have a number of photos to show you I'm going to show them to you one at a time and you make a decision is this the man you saw pull the trigger yes no we're not sure so ideally the witness does not know how many there are but notice then okay and I'm going to say no no not him no right now notice and so now we're on number three the witness could still make sort of a relative judgment well this guy looks more like him than one or two did but I don't know what number four is going to look like or five or seven I think not many have so the witnesses the idea is the witness have to dig deeper make more of an absolute decision comparing each person to their memory not to each other that was the theory behind it what do we know about it well you know it's been around long enough that there have been lots of studies and in the most recent meta-analysis which combines all of these studies there have been 72 experimental tests in labs all across North America Europe and Australia and New Zealand using 13 thousand 143 participant witnesses and basically you collapse all this into one large data bank and see what happens and basically what happens is you know everything else is equal and you're randomly assigning either the sequential or simultaneous procedure the results based on that meta-analysis show mistaken identifications from culprit apps and lineups reduced by 22% with the sequential identifications the culprit were also reduced by about 8% so a lesser reduction but we're going to come back to that in a little bit here because there are variations in how the sequential is done many of these studies did not use back loading and that's where the witness doesn't know how many are to be viewed many of these studies would stop at the moment that the witness said yes those are not the ways in which is done in the real world and it is done in the real world now which I'll come to diagnostic ratios these are the ratios of hits to false alarms right you want that to be as high as possible of course diagnostic ratios are higher for sequential than simultaneous so when you get an ID from a sequential procedure you can more trust it I'm not going to show you that these are some Bayesian curves and you don't really need that well you know in an attempt to share in sharing this work with law enforcement and jurisdictions all across the country of course we've encountered some pushback in certain circles and a lot of it comes from this idea that well real witnesses are not the same as your laboratory witnesses as your simulated witnesses and so they're criticisms of the lab what are those criticisms well they really kind of come down to about three things one is these are not actual witnesses serious crimes real witnesses would be too cautious to make these errors whereas we tend to think that you know memory is memory alright but alright this cautious notion but if that's true then how do you account for all the DNA exoneration cases where these not serious actual witnesses they weren't too cautious to have made those errors but we're going to come back to this these witnesses were not experiencing this stress and fear that ingrained memory I don't know where the legal system got this idea the fact is that the evidence stacks up otherwise fear stress harms memory now I think that that that many people's intuitions is that it helps memory because but only in one kind of sense if something was really stressful fearful you know big type event that you will never forget that it happened right it's not like you know a trivial event that you might forget that you're not going to forget that it happened but encoding the details that's another matter and in general and I'm not going to review this evidence but the evidence suggests that detail encoding is worse under stress and fear and then another criticism that well most of these studies you're using college students I mean what are college students I mean what about real people right and you know it turns out we have used real people as well and you find something very interesting namely there's no better witnesses than college students we've never found a population that outperforms college students college students outperform people who are older than them outperform people who are younger than them outperform people who are their age who are not in college I mean they are the best witnesses so if anything you know we're using a very adept population to make any kinds of estimates that we're making nevertheless about four now four and a half years ago and this this was a four year study took us a long time to do this this was a big undertaking much bigger than we thought when we went into it we decided we have to take a variable we took the simultaneous sequential variable and test it with actual witnesses to serious crimes and we released this study just this last year and so we called the simultaneous for sequential field experiment and there were four participating police departments so we're using real witnesses who did not know they were part of a study these range from murder cases all the way down to like simple assault cases we're just taking witnesses as they come the police participating police departments were Austin Texas police department the Charlotte Carolina police department San Diego police department and the Tucson Arizona police department funding I want to mention this because they were great to us the open society foundation George Soros thank you George Soros the jet foundation one of the things that took so long we started with the jet foundation they were going to fund the whole thing turns out that the wealthy couple that had the jet foundation had their money in all in with Bernie Madoff and so we got one payment from them and then that folded but we did get a payment so and the Laura and John Arnold foundation out of Texas that was particularly interested in the Austin site partners my scientific partners were Nancy Stabli and Jen Dysart and we had other organizations as partners and they proved to be extremely valuable in this endeavor the American Judicature Society the police foundation in Washington D.C. and the Innocence Project in New York. The key characteristics of this study are that these police departments only use photo lineups so we were using photo lineups these are actual witnesses in their ongoing criminal investigations and this was key the the lineup the photo lineups themselves were actually administered using a laptop computer and for the most part self-administered by the witness this is critical we developed this software for that purpose because it turns out we dabbled some with detectives as experimenters right I love detectives they're great they're not great experimenters they deviate from protocol all the time you can't tolerate that in a scientific experiment so the laptop computer has these great properties what happens is the case detective loads the photos like he or she would selects the photos here's my suspect into a program it goes into this that we wrote that then when the detective turns this over to the witness the witness hits a button the laptop administers the photo lineup it gives all the instructions that assure that the instructions followed the protocol all the responses that the witness makes on mouse and so on and this is audio taped as well by the computer are automatically entered into the record I'm missing any of those records critical, random assignment at the very last second to simultaneous or sequential that's important so even though the case detective is building the lineup they don't know if it's going to be a simultaneous or sequential lineup and that's important because maybe at some level they might build better lineups for one or the other depending on what they think is happening random assignment at the last second so the order gets scrambled of the photos in there so those are very important characteristics that on top of that it was all double blind in other words what happens is the case detective can't even be in the room the only other person in the room with that witness on the opposite side of the laptop is a person who even if they got around there and looked at that lineup in fact which ones are fillers and that brings me to the last point here in the real world I mean in the lab we know exactly who the perpetrator is so we can score accurate and inaccurate we know in the real world you don't that's why you're doing the lineup but yes but we got we made sure that in these jurisdictions the actual computer program itself made them do it this way you have one person who's a suspected person right they may or may not be the perpetrator we just call them suspect right and the other five are known innocent fillers right that means if the witness picks one of them you know they made a mistake and if the sequential is doing the job that we do in the lab studies it should reduce filler packs right it should reduce I mean they become the proxy for mistaken identification of a suspect right so that's a very important characteristic down there and so in the computer program you know it either comes up with and there are all the instructions are there and the instructions are delivered both in writing with a voiceover from a female voice so if the witness picks for example number four this is in fact one of the lineups that I have actually permission to show you and this was out of Charlotte but if the witness picks this person that's a mistaken idea how do we know? well because that was one of the fillers right we know she didn't do it she was in a jail cell at the time so but if the witness picks this person because that's a filler then we would call that a suspect pick we don't know if it's right or wrong for sure but we know that it's more likely to be right than picking that filler right so the sequential procedure the simultaneous procedure and so on everything's recorded well let me get to the main results you can find this report in more detail like for example on my home page would be an easy way to get it main results well so this is the percentage of witnesses identifying a suspect a filler suspect which may or may not be right a filler which we know is wrong or making no identification with 497 of these lineups and we find is that identification of the suspects basically did not differ between the simultaneous and the sequential now we aren't doing it exactly as it's done in the lab we're doing as it's done in the real world we're doing it at you know because as I'll mention later lots of jurisdictions have now changed the sequential as a function of this research of this that we've created all of New Jersey does sequential all of North Carolina does sequential and and various other individual police departments all across the country most of Wisconsin and Minnesota and so on Boston does them but but we're not doing it exactly as it's done in the lab we're doing it as it's done in the real world in the lab what happens as researchers tend to as they go along as soon as the witness for example well in the lab a lot of the studies aren't back loaded so they know how many they're going to be in the lab they don't explicitly include a not sure option which can keep the witness going through the sequence in the real world you have to include that option so as we went out there as I went out there began applying this in various jurisdictions first in New Jersey we had to become realistic about how it's really done and I think the way it's done in the real world in those jurisdictions is better than what the lab literature had shown which I think accounts for in part why identifications of the suspects are basically equal here but filler IDs that's a and there's the no identification rate that difference is not significant that difference is statistically reliable so it did in fact as we had thought it would as we had certainly predicted on the basis of lab work reduce the identification of fillers right so it seems to be doing its job well that's one you know this whole idea of relative judgments and everything that comes out of that that's one very important I think kind of contribution of science to trying to improve reliability by witness evidence reduce the likelihood of mistaken avocations but there's something else that I introduced many years ago that I think in one respect maybe the most important reform to be made and it just comes that really what's great about this one I think is that I just borrowed it from science I didn't invent this I didn't invent blind testing I didn't invent double blind testing right blind testing you know for our purposes just we don't have to worry too much about the distinct between single blind and double blind basically we're talking about in this context blind testing meaning that the person administering the test does not know the condition of the person being tested like in tests of new drugs right the medical person who examines the subject does not know whether that subject received the the experimental drug or received a placebo right technically that's double blind is when the patient doesn't know right we assume single blind with lineups we assume that that the detective doesn't say Mrs. Jones I'm going to show you a lineup to see if you can pick out number three right but the standard practice except in recently in some jurisdictions now has been for the case detective to be the one administer the lineup right double blind right we're not talking about by the way intentional influence here the purpose of blind testing is to prevent unintentional influence because people are unaware when they test somebody of the ways in which they might influence the witness so what we're looking at is the following the standard default kind of procedure is you have a detective that thinks maybe Joe did it has maybe reason to believe Joe did it puts together let's say a photographic lineup and puts Joe in position three right and then of some kind of procedure and then begins showing it that sits down with the witness and shows it to the witness and what we're concerned about are verbal and nonverbal influences consider verbal influence so the suspects in position three put yourself in the position of the detective you know that one, two, four, five, six those are just fillers your suspects number three and you show this to the witness and the witness says number two what's natural to do under that kind of circumstance as a tester well be sure you look at everyone take your time right and that makes sense it's sort of like yeah that's what people would normally do but notice that if the suspect is in position three and now the witness says number three what's the reaction tell me about number three so think about it who's doing this identification when you say now be sure you look at everyone and you're saying get the hell off number two it's not two even though that's not the intent I mean it's not a malevolent intent here it's just that's how people test we require double blind testing in science not because we don't trust scientists but because scientists are human and human beings influence the people they test right it's just human nature or and this is what really got me the the double blind lineup the eyewitness says nothing this is a real case from South Carolina where we know according to the testimony both of the detective who took the stand and the witness that the witness looked at this photo lineup and said nothing for some period of time we don't know how long the detective thinks it was a really long time right and the witness says nothing and at some point then the detective says I noticed you paused at number three and I think the witness did pause on number three and I think the witness also paused on numbers one two four five six but the pause on three seemed so much longer to this detective right the last person you want interpreting subjectively the length of a pause is somebody who has a theory about where you ought to be pausing right so even if you had a rule when I first came out with this stuff the FBI quickly changed their procedures because it got a lot of kind of publicity at the time and they quickly changed their procedures and said don't say anything to the witness and it just hand ties them it's really a bad rule because witnesses say things that are ambiguous right I mean like three but five what you got to follow up you got to talk to this witness now if you don't know is it three is it five is it six I don't know could be one if you're that double blind administrator right I don't have a problem with your saying are you saying it's three right are you saying it's five nobody's have that discussion but if you are the case detective right and it says three but five oh okay five good right notice how it just goes a different direction right so anyway even without words there's all kinds of pauses and leanings of displays of interest and disinterest and so on so who should conduct a lineup of independent administrators someone who is not aware of which member the lineup of the photo spread suspect like we did it in our field experiment that I was telling you before and here's another reason for that you know I wouldn't a certainty very important how certain is the witness how confident is the witness we use the terms confidence of certainty interchangeably how certain is the witness this is really important because the the the certainty that the witness expresses how sure are you that that's the guy positive right or maybe right is the primary factor determining whether or not everybody in the system the jurors the prosecutors the judges the detective believe that the eyewitness made an accurate application jury simulations we show that that's what's driving would you believe this witness you can believe that witness that's the primary not the only thing was a primary thing driving those judgments all of the witnesses in the DNA exoneration cases absolutely positive totally positive on the stand right so that's a that's a big deal now as I showed you before I talked about the paradigm we're always looking at what's the correlation between certainty and accuracy so we've been calculating that for years and years and years and unlike what some might think and what some early studies actually show there is a correlation between confidence and accuracy it's not huge but it's far less let me put it this way it's far less than the correlation between height and gender right so if you take a witness's certainty and try to predict their accuracy from it and that's what we would call a point by serial so certainties continuous accuracy's either or and you say you do a lot worse job with that than you would do trying to predict whether somebody is male or female from knowing their height okay if I say here's a person who's here's an adult five feet six inches tall what do you guess male or female guess female and you could be right more often you could be wrong here's an adult five feet ten inches tall what do you guess male or female guess male you could be right more often you could be wrong but sometimes you're going to be wrong that's a much stronger correlation than the correlation between certainty and accuracy so it's imperfect but it's not useless the problem is that we've identified is what I call false certainty which is where you're positive and wrong and in particular the creation of false certainty in other words it turns out false certainty a lot of false certainty doesn't just happen it's actually a system variable it's actually created by the system the key is to understand that witnesses can be influenced even after they've made a choice from the lineup so witness picks number three you thought it was good you identified the actual suspect perfectly legal in many jurisdictions absolutely routine with the exception of jurisdictions that now use double blind procedures it's kind of guaranteed that the witness is going to get feedback because you're going to react somehow even if it's a smile right just the difference between a witness picking a filler and a witness picking your suspect just for a whole the detectives entire demeanor changes witnesses are sensitive to that good you identified the actual suspect yes you got him or and this is the case I get a lot of ideas from dealing with actual cases and this is an actual case in which the witness what happened was the defense attorney contact me said hey I know that my client's innocent but this witness is absolutely positive we had a pretrial hearing okay nice to call me now right but so how does that happen we don't fully understand it but here's what you should do when you get the witness on the stand next I don't think there's any cost to this ask the witness what if anything because there were three detectives who administered the photo lineup to her what if anything the detectives say or do when you pointed to number three in that photo lineup so you know what did the detectives say or do when you pointed to number three and she said they clapped so so when he got back to me on that I'm like oh man we've got to study this is that a problem I mean what does this do so that's what led to the creation of what we now call the post indication feedback paradigm which we first published the first bit of work in that 1998 and it's pretty straight forward what happens is we create an event people witness the event then we get a lineup identification in the data that I'm going to show you but I'll show you some other data here in a little bit the data I'm going to show you all these witnesses are wrong okay because the real perps not in that photo lineup at all so anybody who makes an ID they're wrong when they make an ID basically like the equivalent of flipping a coin to getting confirming feedback good you identify the suspect we're not doing cartwheels we're not applauding right but just a comment from the line permission good you identify the suspect right or nothing in a control condition say nothing then ask them questions take what we would call measures what in the legal system you might think of as like testimony relevant questions about the witness event about their lineup identification that occur after the identification and then after they either get feedback or don't get feedback questions like how certain were you at the time of your identification they identify the real government now remember at the time of the identification they hadn't gotten any feedback right so it shouldn't there shouldn't be any difference between those who got feedback and not in terms of how they answered that question right how good of a so at the time of the identification how good of a view did you have of the gunman how close we were paying attention to the gunman and so on right there shouldn't be any differences as a function of feedback at least at some level right but of course that's not what we find and what I'm going to show you here this is kind of an unusual way of displaying data but I think very forensically relevant this is the percentage of witnesses who score at the extreme so this is the percentage of witnesses who say I was positive or nearly positive all along I had a great view I could make out details of the face quite well and so on right now so in the control condition what happens is we're getting in this initial study about 12 or 13 12% I guess that is probably of the witnesses saying that they were positive or nearly positive in their identification we don't that's false certainty because they're all wrong remember we don't know where that came from exactly right only about 2% are saying they could make out say they had a good view which is good because we gave them a lousy view right and but what about the confirming condition these are really the same witnesses and the only difference is after they make their mistaken ID just a comment you know good you identified the suspect right now what happens is over 45% say I was positive or nearly positive all along right over 25% say they had a great view so this is a reconstruction of their cognitive processes at the time so this is the full 46% or whatever here that's all false confidence because they're all wrong but they're all saying they were positive right this portion is manufactured false confidence manufactured it didn't come natural to them to have that false confidence it came from the comment of the line up administrator right why does this happen well I'm not going to go into great detail here but our original conception which hasn't changed much it's changed some since then is that people do not form online memory traces for these things how much you know what their attention was you know what their view was how certain they were and so on they're not really forming any memorial traces of that at the time right so what happens is when they get asked that question later they construct the answer based on inferences well I was right so I was probably certain or I was right so I probably had a good view so it's a backward kind of thing there are lots of reasons why we think that kind of interpretation is correct one is that we've shown that if you have witnesses think privately they're not coming out and publicly saying anything just privately about their certainty their attention and their view after they've made their ID but before they get any feedback right it largely eliminates the feedback effect right why because now they have a trace now they have a trace of what they thought before they got feedback right so that's consistent with our interpretation that's just kind of also consistent with that kind of interpretation is showing that witnesses cannot really accurately report on whether feedback influenced them so if you ask them is it possible that how you answered the question about certainty was influenced by what you were told about whether you picked the suspect or not most witnesses say no but enough of them say yes that we can look at them as a group right now on the question of attention and view virtually all witnesses say no couldn't have affected me on that but on certainty you will get reasonable percentages saying yeah that could have affected me and so we break those out in this way we gave though some of them confirming feedback and some of them disconfirming feedback someone were told that wasn't the suspect someone were told that was the suspect right depending on how the coin flip came out and among those who said feedback influenced them and you look in this case this is certainty what you find is that indeed it did right and among those who said which is the vast majority who say feedback did not influence me it also influenced them right so in other words they don't really have an ability to know whether it influenced them or not right so that's consistent with this idea that they're not forming online traces at the time actually our more recent interpretation is that which is a safer interpretation is we don't know if they're forming online traces or not for sure what we seem to be observing though is at the very least whatever trace might exist does not seem to be accessible so maybe there's a trace maybe they form online traces but it's just you can't they can't cognitively access it why does the why does the post-definition feedback affect matter this Laura Smallars one of my PhD students right now was the lead on this these data that I'm going to show you are so fresh that they still have that new data smell to them whatever that might be this just shows you how sometimes you've got nothing better to do late at night than make a PowerPoint slide that entertains you for some reason I can't remember why it entertained me so what we did was you'll recognize sort of the standard post-definition feedback paradigm where we create an event and then we get lineup identification but in this case half the witnesses are right and half are wrong and we just create that accurate and accurate experience by manipulating whether the target is in fact present so that we can take people who are right or the target is absent and then we take people who made IDs there, they're wrong so we're manipulating that so we're going to look at both target present accurate and mistaken eyewitnesses who then get feedback or not they either get confirming feedback or they get no feedback and then what happens is we take them afterward after they've made they've made the identification and have gotten feedback and they get cross-examined they give testimony and then we do high-def video of their testimony right and so we have 112 of those 66 of which were accurate 66 of which were mistaken right and then we have various people view these and judge and try to decide determine was this witness accurate or inaccurate did this witness make an accurate identification or mistaken identification Beth you'll recognize this from the old paradigm that I did back in the late 70's and early 80's and so what do you find well so this is the percentages of these observers of these videotapes who believed the witness to be accurate right in the no feedback condition what happens is they believe the accurate witness is 64% of the time they believe the inaccurate witnesses 37% of the time this is very important that is the index of their ability to classify the witnesses having been accurate or inaccurate so they're not at chance by far but it's still pretty lousy performance ideally this should be 100% and this should be zero they should be believing all the accurate witnesses and disbelieving all the inaccurate witnesses and of course that's too much to ask but they are performing but what happens now with the confirming feedback so this is accurate witnesses and inaccurate witnesses the same randomly assigned we got the confirming feedback now what happens they can't tell the difference they all look like accurate witnesses to them right so here we have no difference and that's why this feedback effect is important it just masks people's ability to tell who's accurate and who's inaccurate you hear this from defense attorneys all the time why is it that every witness every case I have no matter how shaky that I witness the conditions are they're always positive because they've gotten feedback and you're not going to be able to see these inaccurate witnesses because they're already prepped and so this is like a prep effect and that's what happens right now Beth mentioned the system variable estimator variable if you take all these things that can influence I witness the implication this is not a full list and you draw this line through here above that line these would be estimator variables below that line these would be system variables and we have to figure this out for a reason these are estimators and these are system there are things like post event information Beth's pioneering work in that area that can be estimator variables and can be system variables but what we've been trying to do is take what we know about system variables what I've been trying to do and sharing that with the legal system sharing that with law enforcement and others across the country policy makers, legislators in some cases and trying to basically improve the accuracy of our estimations by dealing with these system variables the kinds of things that we've been talking about and so what do we have basically it's a list our short list of about five reforms there are more five reforms get us in the door once they start making the reforms you also should do this but you can't go beyond five in terms of anything that you're marketing to anybody properly selected fillers a lot of work on what is a proper filler pre-linup admonition double blind lineups that we talked about the sequential lineup and importantly the part here about confidence inflation that comes from feedback the double blind administrator takes a statement from the witness this is a proper procedure now at the time of the identification that is a matter of record because once that witness even with a double blind administrator leaves that room did you pick a filler did you pick the suspect we need to know from them what their certainty is before they learn anything more and so that is a central reform lots of jurisdictions have made these reforms New Jersey now all of North Carolina Ohio, Wisconsin, Florida is making the transition as we speak should be complete by the end of this year Connecticut as well cities and counties Boston, Minneapolis, Denver Santa Clara County, California one of the first to adopt all these reforms Dallas Virginia Beach Baltimore an estimated 1500 now law enforcement agencies in various states have adopted these reforms so it's a success story but I want to come back and so this is my final this will be my final set of observations love just like good yeah our matter is worse than we thought you know when the field study came out last year hit like 400 newspapers including the New York Times and everybody did the same story and you know which shouldn't surprise you because they all borrow from each other right but they they kind of missed something that I want to highlight to you when I showed you this slide before what they always talked about they were concerned with identification of fillers identification of these innocent people were reduced with that procedure but it's kind of a force for the trees what I want to point out is that look let's take a look at this like if we take just ignore the no identifications for now where they didn't make any kind of identification and say well of the times in which they did make an identification right what percentage were of the suspect well with the simultaneous 58.5 right so you see where that's coming from and we can do it well over here for the sequential as well and so let's look at this which was in the paper that we released but media types didn't pay much attention to it and I want you to look over here look 4 out of every 10 times witnesses identified somebody from a simultaneous lineup they're wrong 4 out of 10 even with the sequential which does better 3 out of every 10 times they identify somebody they're wrong and these are self-selected witnesses witnesses say yeah I think I can pick the guy out right these are witnesses who most of them don't even attempt an ID among those who then do real cases, real witnesses I think that's horrible performance if we got that in the lab we would what would we do, we'd like you gotta give them a better, you gotta do something bring that performance up cause we have some control in the lab it would just be such poor performance you gotta do better than that that's terrible performance but that's real witness performance and this is very important this is the first blind study of real eyewitnesses with real lineups in which you make sure that you record everything right cause in the real world what happens is these filler IDs tend to get buried right so we did all of the best procedures in this field study we had double blind, it was sequential cautionary instructions, warning appropriate amount of you present good lineup composition with our fillers pristine records and despite that even with our best procedure of the sequential 3 out of 10 who make an ID are wrong what does that tell us well and the real rate of mistaken IDs has to be higher than that cause remember we're assuming that all the suspect picks are accurate no in fact we already can point to so many of the specific cases and show that they're not so it raises an important question for me at least have we simply found a limit to the system variables in terms of how much they can clean up the eyewitness identification problem I mean if we're still getting 3 out of 10 are making mistake when they make an ID with our best system variables there are lots of possibilities I'm going to just skip through these about why performance is so low so I'm going to skip through this in the interest of time but here's the message I think that comes from that even with reforms in place unless or until better performance can be shown in other words look this is what our science tells us so unless a better performance can be shown post identification scrutiny needs to be more strongly developed and practiced we need to be more skeptical and we need to find better methods of scrutinizing this one possibility shift the admission burden to the prosecution right now what happens is you can never get these things suppressed no matter how egregious no matter how bad the eyewitness conditions you can never get it suppressed because the defense has to prove that it's unreliable which is a very high burden especially in light of Manson vs. Braithwaite and the kinds of procedures that you have to use so one possibility is why not make the prosecution prove that it's reliable to get it in instead of the defense having to prove that it's unreliable to keep it out right that's one possibility wider use of expert testimony maybe call for here as well or better instructions to juries that are directive in nature and informed by science jury instructions generally haven't worked very well but they're so weak I mean instead of saying you can consider blah blah blah blah blah what about a instruction that says you will consider this and this and this and point to key things instead of this if you want that's the direction that jury instruction work is headed and I will stop there thank you very much