 If you made the lovable Android from Star Trek the next generation into a BioShock mini-boss, would he be a big data? So the new Amazon Firephone has been announced, and from day one there's been a lot of criticism about its primary function, which doesn't have anything to do with making phone calls or playing Angry Birds. It's a market data collection device. Of course, that's not really news. Google doesn't keep the Android operating system open source solely out of the goodness of their hearts. They also know that every smartphone sold is also a boost to Google's market analytics. Big Data is one of those buzz terms that brings to mind Carl Sagan's observation that any sufficiently advanced technology is indistinguishable from magic. Basically, if you collect enough terabytes of user browsing data, you can use specialized algorithms to find trends in that data, and predict future behavior in ways that are shockingly unintuitive, almost magical. Like Big Data shows that people are more likely to buy things around the time of a new moon, and that if you like the curly fries page on Facebook, you probably have an above average IQ. I mean, that's not far removed from inspecting sheep entrails to see if it's going to be a good harvest this year. OK, well, it's actually pretty far removed because Big Data is totally empirical. Even if those relationships don't make sense to us intuitively, they're totally based on evidence. Millions of terabytes of evidence. Which leads to an interesting question. If algorithms can find patterns that we'd never think of and optimize things in ways that we could never imagine, how would we feel about letting those algorithms run our country? Think of a program that was instructed to propose effective legislation, or to make the US economy run like clockwork, or to sentence people to a fair amount of time in jail. Any values that we decided were important, we could program software to collect data, isolate trends in that data related to those values, and then make recommendations to optimize them. It's a little disconcerting to think about something that we don't fully understand telling us what to do. Religious people might believe in and accept an ineffable authority in the universe, but how about an ineffable block of code? Now, I'm sure that anyone who's read or watched any science fiction is currently imagining Captain Picard or Jeff Bridges cursing artificial intelligences who take their directives too far. But that's just fiction. There's some actual facts that are very interesting. For example, the problem of pervasive police surveillance was brought up in the United States Supreme Court in 1967 and still hasn't really been resolved. How long should police with probable cause be allowed to monitor your activities before they're just creepily stalking you and waiting for you to screw up? A few hours? A month? A year? Lots of people had lots of different opinions, but a few smart guys at Columbia University decided to use big data to find an answer that wasn't just a stab in the dark. Analyzing thousands upon thousands of criminal cases involving surveillance, the algorithm developed by these programmers found that after about a week of watching a suspect, there wasn't any appreciable increase in the probability of discovering whether or not they were guilty of a specific crime. If the Supreme Court decides on that figure as a reasonable timeline for police surveillance, it'll be one of the first times that big data has assisted us in interpreting the law. And despite the recent fiascos of privacy invasion by the NSA, there's something to be said for the use of big data in the prediction and prevention of crime. Once again, quit thinking minority report. I'm going to give you some facts. The software PredPole is a predictive analytic tool that looks for trends in crime, location, and type, recommending places and times for police officers to patrol and hopefully prevent those types of crimes from occurring in the first place. Since implementing the use of PredPole, both the Los Angeles and Atlanta police departments have reported significant decreases in burglary and car theft up to 19% in some districts. That's awesome. Of course, this is software with very clearly defined goals and scope. What will we think of an algorithm that was designed to maximize something more subjective, like egalitarianism or justice? I recently did a thunk episode about the dangers of giving too much power to a single ideology. And those arguments still apply. We might use the seemingly magical power of big data to get what we think that we want out of our government. But what if that leads to a world that we don't really like? On the other hand, maybe the very nature of big data makes it perfect for that sort of project. A big part of the problem with singular ideologies is that they're limited by human intuition and imagination, while big data regularly gives us results that are outside of our expectations. Could we find empirical ways of defining the values that our country is based on? Would you do something that was a little bit weird if a program told you that it would improve your life? Please leave comments and let me know what you think. Thank you very much for watching. Don't forget to blog. Speaking of huge numbers of people, hi to all the new subscribers. My name's Josh. If you'd like, there's a playlist of all the thunk videos from the very first ones, which were pretty awful, to the more recent ones, which are, I hope, less awful. Thank you all so much for watching and sharing the show with your friends or family or enemies or strangers or pets. It's the sharing that matters. And for anyone who hasn't yet, don't forget to blog or subscribe, blog share, and I'll see you next week.