 Hi, I'm Gina Matthews. I'm a computer science professor at Clarkson, and we have Nathan Adams, who's a systems engineer forensic bioinformatics services, and Jerome Greco, who is a defense attorney at the Legal Aid Society. We're all super thankful that you got up for 10 a.m. to come here, what we think is a really important topic for all citizens of the modern world and and people interested in technology. We're going to be talking about adversarial testing of software using the criminal justice system. You're just complaining because you're guilty. So software is increasingly used to make huge decisions about all of our lives. I think in this room we know that from hiring to housing to how we find partners and friends, how we navigate streets, how we get our news and the weightier the decision, the more crucial that we can understand it and question it. What input is being given to that decision? Is the decision correct for whatever metric you would like to measure it by? Is there other information that really needs to be considered that's not being considered? And what kind of bias is involved in that decision? Are there protected attributes that are being considered like race and gender? Or even if those attributes are not considered directly, what about proxies for those characteristics that are just as effective as the characteristics themselves? The criminal justice system is just one example of this, but it's a pretty important one. And software and algorithmic decision making is increasingly used throughout the criminal justice system. And usually it's black boxes for which trade secret protection is aggressively claimed. And often the rights, the intellectual property rights of companies are being deemed more important than the rights of individual defendants to understand or question the decisions that are made about them or the public's right to a public trial, understand the public trial process. And even besides that, there are many evidences of problems that bubble up. So it's not just that it's a black box. We have evidence that there's trouble. And how are we going to find bugs and fix the problems if the answer is always, you can't question that. You're just complaining because you're guilty. For example, can you imagine being sent to prison rather than given probation because proprietary software says you're likely to commit another crime? But you can't ask how the software makes that decision. That's the Loomis versus Wisconsin case. What about the primary evidence against you in a murder trial being the results of DNA software? But one program says you did it and another says you didn't. That's the Hillary case. What about being accused of murder solely because of DNA transferred by paramedics to the scene? But they don't figure that out for months. That's the Anderson case. Those are real examples. For those of us who build technology, software, we know that software and complex systems need an iterative process of debugging and improvement. That's just a fact. Anyone who uses technology, let alone builds it, knows that there are glitches and bugs and unintended consequences. You know how easy it is for there to be substantial bugs that you just haven't found yet, that you're shocked when you find them. There's a huge advantage to independent third-party testing. We just know that. It's well documented. You need teams that are incentivized to find problems rather than teams that have a vested interest in showing that the system is working just fine. Thank you very much. We're dealing with a system that actively de-incentivizes that. If only those with interest in the success of software see the details, we have a huge problem and a big recipe for injustice. That's what we're going to be talking about this morning. I'll hand it over to Jerome. Black boxes and proprietary software and trade secrets are increasingly becoming a problem in the criminal justice system. Unfortunately, so much so that we can't discuss all of them today or in as much detail as I'd like, but I'm going to give you an overview with some examples so you understand what the problem is and how it's actually affecting cases. Just quickly, this is a graph from OSAC which shows all the different forensic disciplines that are being used in the criminal justice system. Some are a lot more accurate and reliable than others. We've broken down the technology being used by law enforcement to four distinct categories although they're not as distinct as they may appear. Some technology fits in multiple categories. In fact, the evidence gathering, evidence assessment categories often bleed into each other but I will be giving at least one example from each category. Before we get into that, I've broken down a lot of the technology based on three different secrecy levels. There's secret which is we don't want you to know this exists but if you find out it exists, we don't want you to know that we have it at all and then there's secret as applied which is we have it but we don't want to tell you when we're using it or we don't want to tell you how we're using it and then there's the trust us category which is okay, we have it, we used it in this case but don't look at the man behind the curtain, stop asking questions, just trust us. It works exactly like we say it does. I mean why wouldn't it? Starting with predicted policing, so predicted policing is basically using data and algorithms to make decisions that were traditionally left up to human law enforcement officers which in theory that sounds great, right? You can remove the bias from the system except in reality that's not actually how it works because if you train the algorithm based upon data that was from years and decades worth of racist policing, you're going to end up with a racist output and so if you have, you know, and the problem with that is that you have officers who now can say well the computer told me, right? Oh it's not my fault, the computer made the decision and you know the computer has no bias and it's like well you kind of trained it to have a bias. So for example if you over police a neighborhood, you're going to make more arrests in that neighborhood whether or not there's more crime there. If you feed that into the algorithm, the algorithm's going to think oh there's more arrests there so there's more crime there so we'll send more officers there which then will increase more officers making more arrests to meet their quotas and to justify their jobs and their existence and it becomes a self-feeding circle which is not always actually the best method in fact it often is not. With this comes a lack of transparency, most of these companies are requiring nondisclosure agreements claiming that they have proprietary trade secrets so you can't see how it works under the hood and also saying that the data that they're using to train these programs or to make the programs work are sensitive so you can't review them. All this is preventing public scrutiny of the programs themselves and how they're being used. For evidence gathering today we're going to focus on cell site simulators and mobile device forensics. So cell site simulator for those you don't know is a device that mimics being a cell phone tower and forces all the cell phones in range of it to connect to it and then it can lock onto a particular phone and use that to get a very precise location for example a particular apartment in a multi-story building which is the usv lambis case. It also some of them also have the capability of intercepting content meaning they can intercept text messages and voice phone calls. The reason why I don't use the term stingray device which is what probably a lot of you have heard it be called is that stingray is a very specific model there are other models like the hail storm and they all have their own capabilities and differences and so cell site simulator covers all those models instead of just referring to one specific one. Most people had no idea actually pretty much anyone outside of law enforcement and military had no idea that these were being used because they all required non-disclosure agreements so local state and local state and law enforcement were signing non-disclosure agreements with the company usually Harris corporation and also with the federal government and so they were being used in criminal cases without without defense attorneys knowing without defendants knowing and of course without the general public knowing that is obviously changed but they are still making their efforts to keep it secret in fact the NYPD used one of these devices over a thousand times between 2008 and 2015 without ever once getting a warrant because thank the NYCLU for their great work and being able to prove that on top of that to this day we still don't know which mile the NYPD is using because they still refuse to give up that information and they are doing everything they can to keep that quiet including spending lots of money litigating against it but we're going to talk about a case in which I had worked on which people v. Gordon a case that of Brooklyn so and people v. Gordon this was that Matthew Coretta was the attorney of record on the case and we also had a lot of help from becker wexler who is at that time a legal felt legal aid society and essentially they found our client location that really was not connected to him and traditional cell phone tracking was not accurate enough to get them to where he was and so we said well the only possible way they could have done this is a cell site simulator so in our motion we said we're moving suppressed you use the cell site simulator without a warrant and if you didn't use a cell site simulator explain to us what you did because we can't think of that or technologically possible way prosecutor responds and says concedes and goes yeah okay we did we used one right for us this is the big deal this is the first time we're aware of an in New York state on an open case that we had been able to identify when a cell site simulator had been used so we're ecstatic we think we're gonna win we've got a great thing going uh judge issues a decision a few months later grants our motion to suppress the alleged ID and we're on top of the world we think we've broken new ground the decision gets published the New York Times article comes out and then all of a sudden the NYPD says no no no you're all wrong we didn't use one well a prosecutor in the case just filed in court you know with affirmed in court that you did use one obviously not beneficial to their case so it seems weird that they were lying about something that was only gonna hurt them uh and then on top of it you had months to correct that record and you did nothing it was only when it became very public and there was an article about it that all of a sudden you said no we we have to deny this and the only thing we could possibly think of is that they were bound by their non-disclosure agreement they felt it was necessary to continue their denials especially when it went public which is particularly problematic because now it's already been established and they're still in denial they're still trying to keep it secret even afterwards keeping him with this year's DEF CON theme looking from 1983 looking a year in the future 1984 this basically is a description of all the all of us what we have in our pockets as most of us know our cell phones are the the most successful largest mass surveillance tool ever created and so we're going to talk a little about mobile digital forensics Riley v California was the case with the U.S. Supreme Court said a warrant is required to look through somebody's phone or pull data from somebody's phone this is often done with a device called the celebrate you fed touch this is a virgin to they see on the see on the screen but there are other companies like magnum paraben and others that also provide similar hardware and software and the purpose of these is to extract data from your phone so they could be reviewed and and tagged by law enforcement now this isn't really as terrible right because it's available to outside law enforcement you we can test it we can see if we get the same results we can see what mistakes it has there is a financial barrier but beyond that you know my office has one and I can see if I get the same thing as law enforcement if I pick up something different or or the mistakes it may make right but that's not true with celebrate advanced services and gray key so we all probably remember the 2015 the San Bernardino shooting case and law enforcement the FBI and Department of Justice saying to apple you need to help us get into this iphone right you need to put back doors in your encryption and apple saying hell no we're not going to do that thank you apple for once and yeah it's one time to deserve a round of applause but with that they we get the FBI and Department of Justice later withdraws the request and says well we got into the phone we don't need your help anyway and everybody goes well you just told us it was impossible and you couldn't crack it so what did you do shortly after celebrate advanced services pops up and what that is is it allows law enforcement to send the phone to celebrate they conduct some secret process and then they send it back to law enforcement agency unlocked without the encryption and no longer a problem recently gray key which is a product by gray shift has appeared and it also does a similar process or has a similar result I should say but instead of sending it off to a lab they actually send up an actual product called gray key to law enforcement agencies and they can do it in house the problem with these though is they won't sell it to me and I can't look at any of this as you can see there's an even an email telling me no and I can't know exactly how it works and I can't verify that's not deleting information that's not changing metadata but law enforcement still trying to put this into evidence without anybody and including law enforcement they're not really sure how it works they're not in and celebrates lab and they're not taking apart the gray key device and they're just trusting it because it benefits them and to be clear I don't think celebrate or gray shift are doing anything intentionally malicious but of course we all know just because you program something to work one way doesn't mean it actually is going to work that way right there are bugs there are flaws there are plenty of problems in fact if that wasn't true most of this audience wouldn't have jobs or at least would have to find a different hobby and we probably have a very different conference at DEFCON so in terms of evidence assessment I'm talking about facial recognition is that is one of the big things that we're seeing now in the media especially with recently the ACLU challenging Amazon's now foray into this and connecting to actual politicians instead of the the actual people that it was meant to connect to the problems we're having that this is multiple one is we're often not being told what company is the actual company being used to determine the facial recognition the match and then even if we are we don't know how the algorithm works or how it's programmed and we're being told that this blurry surveillance still has a 70% confidence match to either a mugshot or driver's license photo or sometimes social media profile right and the whole thing comes out to okay let's assume let's assume that's right let's say it is a 70% and you know I can't even verify that because I don't know how it works or you won't let me see how it works but is that enough for that to be used as evidence in a in a trial is that enough for you to arrest somebody and okay let's say 70% is not enough is 80% is that what really really want is evidence in court cases and if you say 70% is is more than enough okay then what about 60% where do we start drawing that line who gets to make that determination right and most of the law enforcement that has very limited rules if any on how they're using this and how they're being trained including examples of them actually manipulating photos to make it more likely to get a match which seems just like evidence tampering to me in particular when we have had limited ability to do testing on on this facial recognition which and facial identification through examples of perpetual lineup which is Georgetown law and also the gender shades project we have seen significant flaws and based on race gender and age for example the gender shades project had showed that women dark skinned women were more likely to be misgendered by the program than say a light-skinned man right and leading to more uh false false identification and so part of the reason that that's believed is the way that a lot of these programs are trained their algorithms the the data they're using is uh are light-skinned men so they're tend to be more accurate for that than they would be for a dark-skinned woman and as a public defender a large percentage of my clients are people of color and it makes them already vulnerable uh in the criminal justice system even more vulnerable and more likely to be falsely identified so talk about individualists uh individualized assessment just there's a couple different examples there um today i'm talking about sentencing algorithms in particular the state versus lumus case this is a case that came out of wisconsin um it's the us supreme court chose not to take it up so it is not law across the country uh but is indicative of the fights that are happening everywhere right now across the country and if and if those local defense attorneys are not challenging that they should be and if they're any of them in the room i'd be happy to talk to you about that later um in this case they use the risk assessment tool called compass made by north point uh in order to get a report of a recommendation of a sentence for the defendant lumus uh one of the things that is is acknowledged even by the company is that it takes gender into account when it makes its decision uh one of the ways it does that is that uh there's the idea that men are more likely to be recivitous meaning they're more likely to reoffend or be re arrested therefore uh there should be less likely to be given probation so if you take a both a man and a woman uh who are exactly the same in all other aspects same crime same criminal record everything else uh this program is less likely to suggest probation for the man than it is for the woman and it's also more likely to suggest suggest a higher sentence uh for the man rather than woman that seems extremely problematic especially when we're calling this individualized assessments when you're doing based upon the history of a group that uh you were born into uh that that sounds terrible to me the other problem is oftentimes we don't know what factors are being included at all we don't know exactly how what factors are using um and this is not just for sentencing this we're having this problem for uh bail for parole uh these decisions are being made and it's not just compass it's it's plenty of programs out there and more coming up every day uh to try to take over the market and even when we know the factors we don't know how they're weighing it so for example I don't know how much compass took into account gender how significant was that when it makes its decision uh that seems like a pretty important thing but of course they're hiding behind proprietary trade secrets by saying well if we release this information somebody will steal it from us a competitor and all of a sudden we'll we'll have no jobs and oh look I'm sympathetic to some extent but that doesn't trump somebody's right you know we're talking about people's liberty here this isn't like uh of a minor thing right people are going to prison right and that's really important that they be able to challenge it that their defense attorneys be able to give them a full defense and it's not that we don't want to it's just that we're actually being hamstrung from doing so and that's uh obviously very uh problematic um and so these black boxes and these claim trade secrets should not be able to be used in the criminal justice system to override somebody's right to face their accuser at the challenge what's happening to them oh thank you uh with that I'll I'll leave it to Nathan hi so I'm Nathan Adams I work for a forensic DNA consulting company in Ohio and my background is in computing so I have a little different flavor than a lot of the folks who work in forensic DNA who are typically biologists so we had the opportunity in a criminal case to examine a previously secret software program that evaluates forensic DNA information um that was developed by the new york city office of the chief medical examiner so when I say OCME that's the the lab that developed this program and FST forensic statistical tool is the name they used for it a little background on it is that FST is approved for use on DNA mixtures containing DNA from two or three individuals so as a as a general rule the more DNA from different individuals you have and in a mixture the harder it is to evaluate whether any single person could have contributed the program does attempt to account for missing data so if you have an incomplete uh sample if it's low level the signal doesn't uh isn't very clear it also allows for spurious noise uh drop in of of uh DNA information and the output of it is intended to be a very concise likelihood ratio which is a a statistical weight in the united states at least all DNA conclusions uh that suggest some defendant could be included as a contributor to a sample that is their DNA uh is possibly present on the item of evidence in question they need to provide a statistical weight because if every other person in the world could have contributed their DNA if that's as specific as we get for that test that doesn't give us very much information at all half the the box the jury box could similarly be contributors on the other hand we get statistics that are suggesting that only one person in the world could have DNA that matches this item and oh look here it's the defendant so FST is supposed to streamline this process for complex mixture interpretation um problems they never sold this to other labs although they tried it what we learned ultimately is that it is a fairly straightforward visual studio project running C sharp with a sequel back end and the timeline will during the middle of the timeline we'll take a break and go into the the problems that we looked into and identified but FST's initial use was approved by the new york state commission on forensic science in 2010 it evaluates data at 15 locations on the human genome so we're looking at at 15 separate genetic locations that it will evaluate a single reference to that is a locus plural as loci those are the locations that we're looking at for those mixtures up to three people they initially attempted to go up to four people they published an article that expressly stated their intent to do so they never did that in 2011 it took them a little time to bring it online but online it was online as of April of 2011 started to be used on on criminal casework so keep in mind that that all forensic science is fairly expensive to to conduct these tests DNA is no exception to that so a lot of these investigations are reserved for particularly violent crimes so in New York City that would include possession of a firearm there's a lot of sexual assault homicide investigations that use DNA or or they evaluate DNA at least and sometimes property crimes but these are particularly significant investigations that they're making so if someone is incriminated by the soft where it is in a pretty serious situation where they could be facing a lot of time in prison in 2011 I don't know if anybody wants to predict but what happens next but that same month they modified their production version of the system and cause it had they had to take it offline so this goes back to what what Jean and Jerome were saying you know everybody makes mistakes OCME made a big one by modifying their live version of the software taking it offline I think we have documentation now through freedom of information requests that suggest this happened either the first or second week that it was being used on casework again and likely in sexual assault and homicide investigations so these are pretty serious investigations we were only told that it was taken offline last fall that it went seven years without us knowing that they had messed up and had to take it offline they fixed the problem that they caused taking the system offline but they also made some additional modifications to the program it was later claimed that these modifications made after the system was validated after it was approved for use in casework after it was brought online for casework they claim that these changes that they made did not affect the underlying methodology of the program but they didn't tell us they thought that until 2017 because nobody knew that it had happened in the first place in July of 2011 they finally brought it back online so it took them three three months or so to actually get it up and running again after they broke it in 2016 my company was hired to to work on a case where the source code to the software had actually been ordered over by a federal court so Chris Flood and Sylvie Levine the two public defenders in that case contacted us and asked us to take a look at the software so we were involved in an investigation into what FST was actually doing because this was not only the first time anybody had access to the the source code but anybody the first time that anybody had access to an executable version of the system so nobody had been able to put through different sets of data to test it at any point in its its lifetime until we got it this is a short fairly small set of output that FST produces when it's run in a single case on a single evidentiary item it produces a single PDF as output this is a portion of that PDF that we generated during our our investigation if you look at the columns they have alphanumeric designations that indicate which genetic location we're looking at on the human genome so these are are 1 through 15 that FST is looking at the first row is a reference profile so this is somebody's DNA profile typically it would be the defendant's DNA profile that they got from a cheek swab or a blood draw and we're making an evaluation to see if that person could be contributor to the sample and the three lines below the reference profile or the evidentiary profiles OCME tests each evidence item three times two or three times to develop DNA profiles from those items so now FST is intending to compare the reference profile to the evidence profile to see whether or not their support that that person could have contributed their DNA to that item the statistical weight is reported for four different subpopulations in New York Asian black Caucasian and Hispanic or the designations because DNA has a tendency to be more similar between or within a population then between populations there is going to be a different statistic reported for each one of these and in an effort to be conservative the laboratory will report the lowest of those four statistics the higher that number is the more support there is for this person included to be included as a contributor and typically that is an incriminating issue if the defendant's DNA is present on an item typically that's a bad situation for the defense so this is the laboratories attempt to be conservative to report the lowest statistic so this is the significance of the statistic this is a likelihood ratio it says that the evidence that is these three rows of DNA profiles generated from from evaluating that sample is 70 times more probable if the sample originated from the reference profile that is the defendant and two unknown unrelated individuals so this is a three-person mixture the this is the prosecution's hypothesis they posit that the defendant and two unknown unrelated individuals contributed their DNA to this item as opposed to the defense's theory that it's just three random people whose whose identities we don't know so in the comparison of these two hypotheses the statistical weight is that it's 70 times more support for the prosecution hypothesis if the prosecution hypothesis is true rather than if the the defense hypothesis the issue with this is that we have documentation of the validation studies conducted by OCME in 2010 and for this same sample the same evaluation that I just showed you it wasn't 70.6 that should have been reported it was 157 so we were scratching our heads when we came across this this was just a sample that we put through the executable of the the executable version of the the source code that we were provided in this case and so we found that something was wrong at first of course I thought that I hadn't configured my version of it correctly but after double and triple checking everything we realized that these were in fact two different values to be reported there had been no noise from OCME that this should be the case that this is the case that this will be the case so it was upon us to identify what happened so at these 15 genetic locations we identified an issue where if we ran the um so sorry these are the different likelihood ratios that were reported between the 2010 validation study and the calculations I'm doing in 2016 we realized that in the the bottom most image you'll see that there are three columns that's three genetic locations at which I did not give the system any information about the DNA present on the item and it came up with the same 70.6 value as if I gave it information at all 15 locations so this is uh this was the the smell test that led us to uncover some code that was actually tossing data so they're without acknowledging it to the analyst running the system to the defendant or even saying to the world that they do this OCME in 2011 had started tossing data uh based on some rule within their system so this calls into question whether the the validation study is is relevant um because it's studying a system that had been modified and pieces of information were not being considered any longer in the casework version so as a bit of a refresher likelihood ratio above one is incriminating the higher it is above one the stronger that that evidence is supposed to be a likelihood ratio below one is generally exculpatory is generally uh supporting um the the defense's theory so when we did a breakdown of this we identified that one of these locations actually has a likelihood ratio these likelihood ratios are calculated at each look genetic location and then multiplied together uh using the product rule but we identified that one of them was actually exculpatory so OCME uh had told FST to throw out certain types of data and it turns out that sometimes that data is exculpatory so they are removing information that supports the defense's theory without telling anybody including the casework analyst running the system two of these other locations for this particular sampler are inculpatory so they would support the inclusion of this person as a contributor but what we do know from the validation study is this particular individual whose reference profile we're we're comparing is not a contributor to the system so it is a false positive that it's above one at all and we're kind of at a loss as to why it would be so high as as a statistic of 157 in the validation study and then after they make modifications to the system which now in 2017-2018 they're purporting to make it an improvement to the system or or that it has no substantial impact we're finding out that they're throwing out data some of which should be considered exclusionary should be considering supportive of the the defense's theory of the case the first public acknowledgement of this was in 2017 it was acknowledged by a U.S. attorney an assistant U.S. attorney so the first person to publicly disclose that this was in fact happening was a prosecutor that is not a biologist that is not a laboratory director that is not a scientist of any sort but but a prosecutor the protective order that had been covering our investigation was vacated after a substantial effort by pro-publica and the the Yale media freedom and information access clinic who who wrote to the judge and asked for the protective order to be vacated in the interest of of the public good public interest the OCME for some reason did not oppose this and then pro-publica posted the code online so if you if you want to go to this this github repo has everything that you need data-wise to get this running on your system you will need some some um microsoft products though at least to make it expeditious so as a brief recap and i'm sorry i need to to wrap this up quick but um 12 samples were tested as regression tests when this modification had been made and only two of those 12 had samples where data was tossed so we're modifying they're modifying fst in a way that is throwing out data and then to demonstrate that that doesn't affect the operation of the system they only evaluated 12 samples but only two of them were affected by the the modification made so we have an incredibly small sample size for them to be basing their their conclusions on and this is out of a total of 439 possible samples that they could have evaluated in this regression test so that's a problem and then we recently learned that they have had 16 additional quality control tests they call it uh which could indicate that additional modifications to fst have been made that we're not aware of yet it's only 70 lines including white space and comments uh of the modified modification that was made in 2011 uh so that's just demonstrating how much uh a little bit of code can affect it and we're just going to run you through a few quotes from uh other from another case that involved similar probabilistic genotyping software and the reasons why a defendants should not have uh access to the source code the responses include from a developer of one of these probabilistic genotyping systems these complex software systems is that you don't use source code to validate software and it doesn't get better a professor of medicine says that the only reason you would need to have source code is if you want to modify the program uh DNA technical leader this is an forensic DNA laboratory says we don't need the source code because the source code isn't normally used when we validate software so we don't need it because we don't need it another laboratory director said that uh you know what DNA analysts only get one class in statistics and typically none in computer science so you know what are we going to do with it well I'm here so you know so I think this is a great one to to cap it off and then Gino take it over so he poses a question to the court these are sworn declarations to a court by the way they're not just you know I was chatting with them and they told me these things um says if we're to discuss errors in DNA testing would you want to to capture an error rate for the entire workflow for the entire DNA testing process yeah exactly like is that really a question so Gina's going to uh explain what else we got going on so um we in addition to this talk we are uh we have uh Brown Institute magic grant to do comparison testing of probabilistic genotyping software systems we're we're focusing on FST within without this uh check frequency for removal function and also comparison to other systems there are other open source systems available um we're trying to tell the story to a variety of audiences including um hopefully coverage in the press for a general audience to the technology audience like you guys and also to a legal audience we recently had articles in the champion which is the magazine for the national association of criminal defense lawyers and basically we're arguing what I think all of us in this room know perfectly well that independent third-party testing is essential and unfortunately independent third-party testing of these systems are really hard it's hard to get access to the executables to the hardware um and even if you do it's often under protective order or it's very expensive you know it might be $30,000 to get the program and $5,000 to go to to training or you might not even be able it might even be sold to to independent testers um it's difficult to get old copies of the software or match up results that were reported for particular defendants to the version that was applicable at that time let alone getting this is just access to executables let alone source code or bug databases or testing plans or design documentation or other things that would be incredibly relevant to understanding what the heck is going on and even if you acquire these things there's often terms of service that limit the publishing of results how crazy is that this problem of trade secret protection aggressively being claimed over the rights of the public and of defendants we feel is often done really to shield from legitimate questions of quality and fairness legitimate questions of quality and fairness much more so than to protect from competitors and it is fundamentally thwarting the essential iterative improvement and accountability to stakeholders beyond buyers if the people purchasing the software are basically just asking the question are we sending people to jail okay good then we have a big problem um and there's also a difficult to connect audiences if you were to find a bug in fst how would you be hooked up to a particular defense attorney that might that bug might be relevant um and we would really love you to help what are some ways you can help um i hope just listening to this talk you say to yourself i could have debunked those quotes right and i have some serious credentials why don't they ask me so if you say that to yourself we could hook you up with some defense attorneys that could help you say those things um also we would really love to see um advocacy for um requirements in the procurement phase of software once it's in use you know under certain terms of service or whatever it's harder but why not say if we're going to use public money for criminal justice software require would be great or at least give a lot of credit in the procurement phase for source code software artifacts like bug reports internal testing plans software requirements no clauses presenting preventing third-party review how hard would that be come on access to executables for third-party testing under reasonable conditions and here's a big one scriptable interfaces to facilitate automated testing you can get your hands on these things and if you want to run it through you know a a thousand sample tests you know what are you going to do that by hand um bug bounties for finding things would be great uh funds for nonprofit third-party entities to do independent testing all these things would be on our wish list we'd love to see you be third-party reviewers go get your hands on fst or lab retriever lr mix or like ltd or ur formix other open source pg systems predictive policing software like civics gave take a look find some bugs or bad code please do something about it yourself but also let us know construct software yourself based on published alternatives and then compare the results you're getting to the black boxes so many things our community could do to change this conversation please help us do that and the big picture is that black box decision-making is happening all around us and our community could do a lot to bust open those black boxes or to compare them to one another or to fight for accountability and transparency um the association for computing machineries tech policy groups came out with a set of principles for algorithmic accountability and transparency that could be a place to start if you're involved in building software systems you could point your team and your boss and your company to this is some professional ethics guidelines that say we should be building an awareness access and redress accountability explanation data provenance auditability validation and testing we can all do a lot to provide the evidence that's needed to improve systems for all stakeholders so that we're not running our society on buggy or possibly even malicious algorithms that are hidden from view we would like to thank the many people without whom our work would not be possible a special shout out to we have four of our students who are working with us here in the crowd uh maria marzia steven and abby you guys could wave um and i will simply end with um please get in touch with us if if you if you think you can help with this effort in full disclosure the best way to get a hold of us is probably the three direct emails but uh we tried to set up some more joint ways uh we set up a twitter software justice it's just recently set up so um be gentle with it we also set up a discord channel if you find us software justice on twitter there's a recent link uh with an invitation to our discord channel we're working on setting up a sub a subreddit for software justice but it's not up yet but please get a hold of us and let's all work on this because i think our community could make a big difference and a big difference is really needed