 This talk is called Don't Get Distracted. It used to be called Finding Responsibility, but this name worked better. You should be able to tell why. After my brief introduction, I'm going to hit the ground running with some potentially disturbing content. It includes references to but not descriptions of killing and suicide. Please take the next few minutes to decide on another talk if you'd like to. There won't be any judgment, and if you're sitting in the middle, people will be happy to give it out of your way. Yes, these slides are blank. No, they are not broken. I've been a developer for about 10 years. I have a bachelor's degree in software engineering. I've worked in software jobs in multiple industries, including at a consultancy where I worked with many clients varying business needs. I built tools from a social support network for people with type 1 diabetes to a shipping rate comparison service. I've even worked for banks building task management software. Now I work for Heroku on the support team where I get to help all sorts of developers like you run your code and solve your interesting problems on our service. For the past four years, I've organized Keep Ruby Weird, a community-oriented Ruby conference in Austin, Texas. I'm going to tell you about how I took a job building software to kill people, but don't get distracted by that time. I didn't know it at the time. Even before I walked across the stage of graduation, I accepted an offer for an internship. It paid half as much as the most I'd ever gotten at my highest-paying job to that point. Not to mention that I'd just spent four years working low-paid student jobs or living just on student loans. I'd be joining a contracting company for the Department of Defense. The Department of Defense, or DOD, is the part of the government made up by the military in the United States. The DOD outsources all sorts of things, from P-8A multi-mission maritime aircraft to Blue Shade 451 polywool cloth. They also outsource a lot of software projects. At the time, I thought nothing of the fact that I'd be helping the military. Besides, they're the good guys, right? My dad was in the military, so was my grandfather. I have great respect for those who serve in our militaries. It was good money, a great opportunity, and a good friend of mine had gotten me the gig. Life was good. I showed up for my first day of work in North Virginia, or NOVA, as the industry likes to call it. I met the team of other interns, and I learned about what we would be building. A tool to build... A tool to find Wi-Fi signals using your phone. It seemed pretty cool compared to what I'd built up to that point. The most complicated thing I'd ever done was an inventory management service. The app didn't concern itself over much with persistence. Who really needs to stop and restart a program anyhow? The data was right there in memory, and if you forgot how many Aerosmith CDs you had, who cares? Honestly, the idea of finding Wi-Fi routers based on the signal strength seemed pretty intimidating. The idea impressed me. But don't get distracted by all this. The software was intended to kill people. I joined the team after they'd already gotten started on the project. The gist of the tool was that they would look at how Wi-Fi signal strength changed as your phone moved around. If the signal strength got stronger, you were probably moving closer to the source. If it got weaker, you were probably moving away. To find this information, we'd collect two pieces of information for each Wi-Fi signal in range, your phone's location, and the signal strength of that Wi-Fi source. To predict the actual location of the Wi-Fi signal, we used a convolution of two algorithms. The first was R squared. It measured the distance between the signal strength we'd observed and the expected signal strength at that distance. And it did that for every point in the search grid. Locations that had the lowest R squared error rate were the most likely to be the source of the Wi-Fi signal. We'd combine that calculation with a Gaussian estimation. It creates a probability curve using the standard distribution or the bell curve that you've probably seen all over the place. It started with an inverted curve, a probability hole of low likelihoods that the Wi-Fi signal originated from those distances. That represented the idea that the phone is probably not standing right next to what you're trying to find. It added a normal bell curve up that was a high probability that the source originated from a distance further out. The algorithm adjusted the width and height of each of those curves by consulting past measurements. It created a heat map of probabilities for the signal source. We'd normalize those two probabilities for each location in a search grid, and we would combine them. The combination of those two algorithms was much more correct than either individually. We stored this probability matrix for each location that the phone collected from. Using these, we were able to give you the distance if you moved in a straight line for how far away you are from that Wi-Fi source. If you turned a corner, we could also tell you the direction so that you could find it in two-dimensional space. Climb some stairs, and we'll give you altitude as well. The technology was the most interesting thing I had built to that point. It may still be the most interesting thing I had built, but don't let that distract you. It was designed to kill people. I mentioned earlier that I had a software engineering degree at this point. My teammates were much earlier in their educational careers. Most of them were a year or two into their four-year programs and were math or computer science majors. My expertise was in the design and process of building software, while theirs was in the theory of mathematics or how computers are used. I'd help to translate the working algorithms that they built in MATLAB into the Java code that would need to run on Android phones. And let's be honest, we spent plenty of time deciding whether we preferred Eclipse or NetBeans. Can I say how happy I am that as a Ruby developer, I've not had to figure out where to put a jar file in almost five years. I also spent a lot of time bike shedding code organization and pairing on performance improvements to the code. It worked, but it took almost seven minutes to find the Wi-Fi sources. This was partially because of the code that we translated from MATLAB, which is optimized for working in matrices of numbers. We wrote in nested loops. Quadruply nested loops with some pretty expensive calculations in the middle. One example is that we were calculating the distance between two points. We used great circle distance, which is where you measure the shortest distance between two points on a sphere, such as Earth. The function performing that calculation was being hit hundreds of thousands of times for each collection point, often with the same two locations. It was very slow. We solved that by implementing a hash with the keys of the two locations and the values or the distances between them. This at least meant that we didn't have to redo those calculations. That and other optimizations we made sped up the performance from seven minutes to a few seconds. But don't get distracted. That performance made it faster to kill people. The accuracy of the locations wasn't fantastic, either. I don't remember exactly what it was, but the number 45 feet sticks in my head. That's about the length of a shipping container. That's significant when the Wi-Fi signal strength of 802.11n is only about 100 feet. That meant that we could be almost half the range of that Wi-Fi router away from where it was. It's a big error rate. I talked about the Gaussian estimation, the two curves that were part of the second algorithm. We hard-coded the numbers that defined the width and depth of those curves. They were only starting points, but they were starting points that we used every time we made this calculation. Does anyone know what a genetic algorithm is? It's a type of machine learning program that produces a set of values that optimize for a desired result. Each of the Gaussian estimation values is a gene. The set of values in a genome... The set of values is a genome in genetic algorithm parlance. The genes were our three constants. A fitness function is what the genetic algorithm uses to measure the performance of a genome. From my data sets of readings that I was using, I knew the actual location of the Wi-Fi source. So I was able to plug that genome into the geolocation algorithm and measure the distance between the location that it found and the location that I knew was accurate. That was my fitness score, the smaller, the better. Genetic algorithms take a set of genomes called a generation, and they keep a certain percentage of top performers. These performers survive to the next generation as clones. Sometimes the algorithm mutates those clones by adding a Gaussian random value to each gene. That meant that each of them had a slight chance of performing better or worse. New random genomes were created for the remainder of the population. We saved the top performers across all populations, which meant that when I finished the genetic algorithm, I could look at the best performer and plug that into our geolocation algorithm permanently. I let this geolocation algorithm run over the weekend. It was able to increase the accuracy from 40-odd feet to about 10.3, 25% the error of the original. That's less than the GPS accuracy on the phones that we were collecting data from, which means that it was probably not accurate overall. It's called overfitting, and it's solved by using separate sets of test and training data. I loved this stuff. Genetic algorithms are squared, Gaussian, random values. It's the sort of thing that you hear about and you learn in school, but they tell you you'll never use again after you graduate. But we were using it in a real project. It was great. But don't let that distract you. This made it easier to kill people. The location algorithm now worked accurately and quickly. The next feature was to add tracking a moving Wi-Fi access point. I briefly wondered why a Wi-Fi access point would be moving, but that wasn't as interesting as figuring out how to find it with our code. We made use of call-man filters to observe three state variables, the position, velocity, and acceleration of the Wi-Fi signal source. Given these values and the time since the last measurement, a call-man filter is able to improve the current prediction surprisingly well, and it throws away values that have low accuracy automatically. Each time we ran the real-time algorithm, we'd also run the call-man filter. With only that information, it was able to produce an estimate that was more accurate than the calculated value and that showed movement. At the same time, we added the ability to track more than one Wi-Fi signal. We'd filter our collection of readings by the unique identifier of each signal known as the MAC address. The filtered data sets went through the full algorithm to produce predictions. We used the APIs of the phones to pull the location data for each Wi-Fi signal in range, basically anything you can see in your Wi-Fi connection list. It was all very exciting. These seemed like academic problems, but we were getting to use them in a real-world project. Being a programmer was going to be great, but don't let that distract you. This meant that we were able to kill multiple people with our software. We'd been working with the project owner throughout this process. It was fairly laws-eye-fair about most things. He'd check in maybe once a day and then go back to the part of the building dedicated to undercover, to classified work. Whenever we hit one of these milestones, he'd be happy about it, but a question always came up. He wanted it to sniff out signals put up by phones in addition to those put up by Wi-Fi hotspots. It's a much harder problem from a technical perspective. The functionality necessary to do this is called promiscuous mode. It's a setting on the wireless network controller. Neither Android nor iPhone support this option, so we'd have to either root or jailbreak the phones. We looked for packages that would help us sniff out these packets as they were sent back to the router. We found something on Sourceforge, but it wasn't well-documented and we didn't understand it very well. We told the project owner we'd get back to this later. None of us thought it was that important. We had the technology to find these Wi-Fi sources, and that was the goal. Each time we'd demonstrate a new and exciting tech, though, the same question came up. We got Wi-Fi signals located. Great. Does it find phones? It's taking seconds instead of minutes. Great. Does it find phones? We looked into it finding phones and it doesn't seem likely, but we'll get back to it later. Okay, we can come back to it later. We got moving targets working. Great. Does it find phones? I had been distracted. All the cool problems we were solving. Finding nodes, speeding things up, making more accurate predictions. It was all so cool, so much fun. I hadn't thought about why we were putting all this work into finding a better place to sit and get good Wi-Fi. That doesn't even make sense if you think about it for more than a few seconds. Does it find phones? This was never about finding better Wi-Fi. It was always about finding phones. Phones carried by people. Remember I said I worked for a Department of Defense contractor? The DOD is the military. I was building a tool for the military to find people based on where their phones were and kill them. I tried to rationalize this, then. The military is in place to protect truth, justice, and the American way. But this is the same time that we found out that the government had been spying on us in the United States with drones and that they'd lent out that technology to local, state, and federal government officials, law enforcement officials over 700 times to run their own missions. The military is the tool of the government and it seemed like we couldn't trust the government as much as we thought we could. I didn't want to be a part of something that was going to be used to kill people, especially since I'd never know who it was used against, let alone have a say in that decision. I rationalize it now, too. We were interns. We didn't have a clearance. The projects that this company did for the government were top secret. I wasn't allowed to know what they were. My code probably got shelved and forgotten about. This was an extreme example of code that was used in a way that the developer did not intend. The project owner conveniently left out its purpose when he was explaining the goals and I conveniently didn't look too hard into that. It was great pay for me at the time and it was a great project. Maybe I just didn't want to know what it would be used for. I let myself get distracted. I was distracted by the technology, but it would be just as easy to be distracted by a cool framework that the company was using, the great design of an app that would look super good on your portfolio, or some really nice office amenities. Maybe others are doing the same thing and clearly they've already thought about it, so it must be fine. There are other examples of when code was used in ways that it wasn't intended and of code that just does bad things. A year and a day ago, a developer named Bill Sorer wrote a blog post. It opened with the line, if you write code for a living, there's a chance that at some point someone will ask you to code something a little deceitful, if not outright unethical. Bill had been asked to create a quiz that would almost always give a result that benefited the client. Bill worked in Canada. And in Canada, there are laws in place that limit how pharmaceutical companies can advertise prescription drugs to patients. Anyone can learn about the general symptoms addressed by a drug, but only those with the prescription can get specific information. Because of this law, the quiz was posing as a general information site rather than an advertisement for a specific drug. If the user didn't answer that either they were allergic to the drug or that they were already taking it, every result said ask your doctor about this drug. That's what the requirement said to do, and that's what Bill coded up. The project manager did a quick check and told Bill that it didn't seem to be working. It always gave the same answer. Those were the requirements, Bill said. Oh, okay. A little while later, Bill got an email from a colleague. It contained a link to a news article. A young woman had taken the drug that Bill had written this quiz for. She had killed herself. It turns out that some of the side effects of this drug are severe depression and suicidal thoughts. Nothing Bill did was illegal. Like me, Bill was a young developer making great money and doing what he was told. The purpose of the site was to push a specific project. That's why it was being Bill. Bill had chalked this all up to marketing. He never intended for this to happen. Maybe Bill got distracted, too. In his conclusion, Bill writes, as developers, we're often one of the last lines of defense against potentially dangerous and unethical practices. We're approaching a time where the software we build will drive a vehicle that transports your family to soccer practice. There are already AI programs that help doctors diagnose disease. It's not hard to imagine them recommending prescription drugs soon, too. The more software continues to take over every aspect of our lives, the more important it is for us to take a stand and ensure that our ethics are ever-present in our code. Since that day, I always try to think twice about the effects of my code before writing it. I hope that you will, too. I think it's poignant that all of the examples that Bill listed as something that might happen in the future happened today. Bill's story isn't that far off from mine, but there are still other examples. Earlier this year, a story came out about Uber. It had built into its ride-sharing app code that was called Grayball. It's a feature of their violation of terms-of-service tool that can populate the screen with fake cars when its app is opened by users in violation of its terms. In a statement, Uber said, this program denies ride requests to users who are violating our terms-of-service, whether that's people who seek to physically harm our drivers, competitors looking to disrupt our operations, or opponents who collude with officials on secret stings meant to entrap drivers. In practice, as the New York Times reports, it was used in Portland to avoid code enforcement officers looking to build a case against Uber for operating without a license. When triggered by Uber's logic, it populates the app with cars that don't exist, with fake drivers who quickly cancel after accepting a ride. I'm not a lawyer, but this seems like an obstruction of justice, itself a crime outside of the illegal activities in Portland. Grayball is used even today, though mostly outside the United States. I'm a huge fan of ride sharing services. It's not uncommon to see in the news these days articles about these drivers doing heinous things. Grayball may have enabled some of those things to happen. Again, this is an unintended consequence of a tool that was built. Maybe the internal pitch for this was to Grayball users who were violating the terms, people who were under 18, people who didn't pay to clean up their explosive accident last night. Rather than block them, potentially causing them to create a new account, they could just be put into an alternate dimension where they could never get a ride for some reason. That's fine, right? If these developers had thought about the worst possible use for this code, this circumvention of justice might have come up earlier on and it could have been addressed. Maybe they were distracted by the face value of the request from looking deeper at its purpose and uses. There are all such things as well that aren't as black and white. Apps that listen to the microphone to tailor ads to you based on what you say near your phone. Websites designed to exploit psychology to take up as much of your free time as possible, which have been linked to exploding rates of depression. Any number of apps that opt you into an email newsletter. These aren't as obviously bad as the previous examples, but at least in my opinion, they're still kind of shady. This value system is different for others. Richard Stallman, for example, believes that eBooks are unethical. Others may think that's a little eccentric, but that viewpoint agrees with his overall system of beliefs. There are actually words for what society decides are good or bad versus what you or I or Richard Stallman individually believe. Ethics and morals. While modern psychology more or less uses these terms interchangeably, an understanding between you and I will be useful later on. Ethics are imposed by an outside group, a society, a profession, a community like ours, or even where you live. Religions provide ethical systems, so do groups of friends. Societies in whatever form determine right or wrong, good and bad, and they impose those definitions on its members. Ethics in societies at the local, state, and national levels are often coded into law. Morals on the other hand are a more personal version of the same thing. Society as a whole imposes its mores on smaller communities, and all of that trickles down to the individual level. That's not to say that your morals can't conflict with society's ethics. Maybe you believe that freedom of speech is a core tenet of human rights, but you live somewhere where defacing political or religious objects is wrong. Let's not get distracted by morals and ethics just yet, though. We'll come back to those later. The unifying factor of all of these stories is that developers built code that did these unethical or immoral things. As a profession, we have a superpower. We can make computers do things. We build tools, and ultimately some responsibility lies with us to think through how they're going to be used. Not just what their intention is, but what misuses might come out of them. None of us want to be building things that will be used for evil. The Association of Computing Machinery is a society dedicated to advancing computing as a science and profession. ACM includes this in their code of ethics and professional conduct. Well-intended actions, including those that accomplish assigned duties, may lead to harm unexpectedly. In such an event, the responsible person or persons are obligated to undo or mitigate the negative consequences as much as possible. One way to avoid unintentional harm is to carefully consider potential impacts on all of those affected by decisions made during design and implementation. So how can we carefully consider potential impacts? Honestly, I don't have the answers. I don't think that there is a universal answer because I have to believe that if there was, we wouldn't have this problem. We wouldn't be building this code in the first place. I do have a couple ideas, though. One, I got from my friend, Schneeens. Add to the planning process a step where we come up with the worst possible use for our software. Opting in folks to a mailing list by default. The worst case is probably that we send them a bunch of emails and they unsubscribe. Maybe they stop being our customer. As Schneeens said, am I willing to sell my hypothetical startup Sol for a bigger mailing list, especially when that might be all that keeps my company afloat? Yeah, why not? And I can see that. I understand it. I still think it's a little bit shady. It's not a best practice, but it's not physically hurting anyone. If I had sat down and thought through what my code would be used for when I was building the Wi-Fi location app, I would have come to a much different conclusion. Actually, I think that thinking through the worst possible uses of a code could be a fun exercise. You might come up with some pretty wacky examples. If we send Batman an email and he has notifications for new emails on his phone, then he might be looking at his iPhone when the Riddler drives by in the Riddler car. And he might get off his witty one-liner at the crime scene. Riddle me this, riddle me that. Who's afraid of the big black bat? It's not so plausible, but it shows that you can come up with some pretty wild-out-there examples that aren't obvious at first glance. Another thing that I think I should have done and that we can all do more of is to just not take the request at face value. The project owner at the Defense Contractor didn't spell out what the code would be used for, but at least in retrospect, it wasn't a big jump in logic. We're going to build an app to find Wi-Fi signals is all true, but it's not the whole truth. Asking myself or them why enough times might have led me to an earlier understanding. Why? To find the Wi-Fi signal source. Why? To go to them. Why? Why? Why? Comedian Kameil Manjani, best known for the TV show Silicon Valley and his recent film The Big Sick, took to Twitter recently on this subject. I know there's a lot of scary stuff in the world right now, but this is something that I've been thinking about that I can't get out of my head. As a cast member on a show about tech, our job entails visiting tech companies, conferences, et cetera. We meet people eager to show off the tech. Often we'll see stuff that's scary. I don't mean weapons. I mean altering videos. Stuff with obvious ethical issues. And we'll bring up our concerns to them. We're realizing that zero consideration seems to be given to the ethical implications of tech. They don't even have a pat rehearsed answer. They're shocked at being asked, which means that nobody is asking these questions. We're not making it for that reason, but if people choose to use it that way, it isn't our fault. Safeguards will develop. But tech is moving so fast. There's no way humanity or laws can keep up. We don't even know how to deal with open death threats online. Only can we do this. Never should we do this. We've seen that same blasé attitude in how Twitter and Facebook deal with abuse and fake news. Tech has the capability to destroy us. We see the negative effect of social media. No ethical considerations are going into the development of these technologies. You can't put this stuff back in the box. Once it's out there, it's out there. And there are no guardians. It's terrifying. It's a major problem when we're given so much power in tech, but we can't do anything to ensure that it's used safely. Thinking about what we're doing and being careful not to build things that can be used maliciously is really important. And it's the least that we can be doing. This is Chelsea Manning. In an interview with the New Yorker radio hour, she's discussing the ethics of what developers do. No, she's not. I guess technologists should realize that we have an ethical obligation to make decisions that go beyond just meeting deadlines and creating a project. Let's actually take some chunks of time and say, what are the consequences of the system? How can this be used? How can it be misused? Let's try to figure out how we can mitigate a software system from being misused or decide whether we want to implement it at all. There are systems that, if misused, can be very dangerous. Don't get distracted by deadlines and feature requests. Think about the consequences of what you're building. Build in safeguards to prevent misuse or don't build it at all because it's too dangerous. I'm asking you to do something about this. Well, I guess I'm asking you not to do something because of this. It's only fair that we talk about when and where to take that stand. Let's say I had a time machine and I can go back in time to 2011 and do it all over again. The tool has just been explained to me for the first time. What do I say? I think the first thing is to establish a mutual understanding of the task. It's entirely possible at this point that I don't understand what the actual thing is and I'm overreacting. I ask, why are we finding these signals? The project owner says, we want to find people's cell phones. Who's finding them and why? I don't know, probably some soldiers in the Middle East. Why, I repeat. I can't tell you that. I can't tell you that. It's something that I got a lot from the project owner. It's code for, I have a clearance and I know things about this project. I know what you're asking and I know the answer, but I'm not allowed to tell you. At this point, I think we have our mutual understanding. The task is to help soldiers find people's phones, probably attached to those people. The reason is left unsaid, but I think we both know that too. This organization is a defense contractor. They build things for the military. That is their core competency. They're not not going to do this. On the other hand, I care a lot about not killing people. The company's goal is to build things for the military. The military is necessary and sometimes they need to use force. Personally, however, I don't want to be a part of that. If my goal is not to build those tools, then there isn't a good fit for me at this company. That probably means that the worst case here is that I'm going to leave today without a job. Either I'll say no and they'll fire me, or I'll say, and I'm not comfortable with this, but good luck and I'll quit. Those are worst case scenarios. They're not necessarily what's going to happen. And so before I do this, I need to consider some questions. Can I afford to leave here without this job financially? Can I rely on my personal network to get me a new job? Have I built up enough trust with my employer where I can go to this and be heard out? The answer for me at the time was no to all of those questions. Sometimes something is important enough that you still need to do it, but that's a very personal decision. There's a lot to go into these decisions and they have consequences. I would like to think that I would still say no. Let's look at another situation where someone did the ethical thing. A developer we'll call Alice received a strange request. We want to identify weak passwords in our system and notify users to change them. We'd like you to run a password cracking tool on our very large database of users. Alice thought this was kind of a strange request, but she said that if the appropriate paperwork were filed, she would do it. Alice received the paperwork and she ran the crack. The next request was, we'd like the list of users' email addresses as well as their passwords. Alice knew that her coworkers had a valid desire to help customers improve their passwords. She also knew that a lot of users reused credentials across websites. If a report was created that combined those two pieces of information, that it could be misused to log into another... to those users' accounts on other websites. Alice pointed this out to her manager and together with the customer success team, they designed an email that didn't include the password. Customers received notifications about their weak passwords and were able to change them. Nobody got fired and Alice built up trust within her team. Different scenarios required different ways of thinking about what you should do. Sometimes the right thing to do is to say nothing and just do the work. It isn't a simple thing and it has consequences, but don't get distracted by having to think about it. Sometimes your code can kill people.