 I got it. We load a think tank with heat rounds. Boom. Intelligence explosion. Way back in episode 65, I summarized a famous argument by philosopher Nick Bostrom about the possibility of a runaway artificial superintelligence. According to Bostrom, it's only a matter of time before we develop a computer program that has human-like, general problem-solving ability. And if we throw a few more processors at it, it's a short ways from there to a program with superhuman abilities. A little bit of software that can puzzle its way through any arbitrary problem faster than any human. Turn that program to the task of improving itself, and soon we're stirring down the barrel of something orders of magnitude smarter than we are. Which may lead to a sort of disastrous, be careful what you wish for situation if we haven't designed it exactly right. Making a simple request of an unfathomably brilliant, silicon-based intelligence might have all sorts of unpredictable consequences. More or less the way a poorly phrased wish might leave you wishing you'd never found a magic lamp. I want a million bucks! Bad move. The rogue AI scenario is a fantastic, yet plausible premise for a sci-fi apocalypse. A simple program set loose by a naive coder that rapidly claws its way to godhood, devouring the planet in a fanatical attempt to satisfy some mundane request. The fantasy is so gripping that it has moved many to dedicate significant thought, time, work, and money to solve the so-called AI alignment problem. Intelligence funded by Alarmist chew through millions of dollars every year in the name of developing a theoretical basis for friendly AI. Celebrities like Elon Musk and Neil deGrasse Tyson have articulated concerns about the potential for an artificial superintelligence doomsday scenario, and a number of folks have scared themselves so badly with the Roco's Basilisk Evil AI thought experiment that they've had nervous breakdowns. It's clear that many find Bostrom's reasoning about the feasibility and potential dangers of superintelligence compelling, but in order to touch on every topic he wants to include in a pop philosophy book that has less than 350 pages, he kind of has to start in the middle of things, skipping over some questions and assumptions that don't lend themselves to a rapid-fire argument about the timeline and importance of aligning AI with human values. With all the enthusiasm and anxiety that has built around Bostrom's imagined futures, it might be worth acknowledging the foundation that he's reasoning from, the premises we have to swallow to get to the good stuff. Those assumptions are certainly plausible, but the way people have gotten wrapped around the axle about an inevitable AI apocalypse, it might be worth picking some nits and thinking of ways they might not hold. For example, to launch his case for the imminence of strong AI, Bostrom invokes an idea called technological accelerationism to suggest that superintelligence will be necessary at some point to maintain the historic trend of increasingly rapid technological change. The speed of innovation and production during the Industrial Revolution wouldn't be possible without automation. The speed of today's world wouldn't be possible without computers. And it's hard to imagine how that hastening process could continue without something that can think faster than humans. We've talked a little about various reasons to be suspicious of the tech-drives-history narrative, archeological bias for artifacts over cultural, natural, or social factors, false starts of technology that was ultimately very influential but didn't take the first time around, the much larger historical influence of boring constants like access to arable land or squishier cultural factors. Bostrom's argument doesn't really hinge on this point. He explicitly says the exponential curve-fitting deal is the weakest reason to expect and worry about superintelligence, but it's worth noting how it's a point well-suited for convincing a sympathetic audience. Folks who are predisposed to thinking of history in terms of technological progress will find themselves nodding along to a familiar tune, but people new to or unconvinced of the notion may find this whole chain of reasoning bizarre. How are you measuring growth? How are you linking it causally to technology? Why must that trend continue? How do we know AI must be the facilitator of that process? Futurists and sci-fi fans might not even think to ask these sorts of questions, but they're not trivial to answer. Another potential point of contention in the runaway superintelligence scenario is weirdly how optimistic it is. In his book, Bostrom cites a poll of expert AI researchers guessing how many years it would take to develop a human equivalent general artificial intelligence, the immediate precursor to superintelligence. He points out just how little time these experts think we have to figure out how to control such a thing, and chimes in with his own opinion that they're overshooting the mark that is likely to happen sometime in the next century or so, at which point superintelligence is right around the corner, but he doesn't really go into one of the assumptions baked into the poll. How many years would it take without significant negative disruption? That might be one hell of an ask. In March of this year, researchers at the conference of the Swiss Federal Institute for Nuclear Biological and Chemical Protection had a disturbing thought. They developed a machine learning-based tool to help them discover new drugs. After being trained on a library of known medications, the algorithm iterates through a list of untested molecules and gives each a score based on two criteria, how well they interact with parts of the human body that are favorite targets for disease, and how unlikely they are to have toxic side effects. This is a great method for narrowing the list of candidates for new drug trials, but the researchers also realize that if they simply flip the sign on the drug toxicity, the same program could be used to find highly bioactive, highly toxic substances. After training the model on VxNerveAgent and running it for just six hours, they obtained a ranked list of 40,000 candidates for new chemical weapons, a list which included several known agents of chemical warfare, as well as numerous pesticides and environmental toxins. In order to get within spitting distance of a human-level artificial intelligence, it seems like we'd need to pass through several intermediate states where more powerful AI programs like the chemical weapon generator are developed and refined. Activities like policing, facial recognition, user engagement maximization, and surveillance already use machine learning algorithms to expand and enhance human capabilities in those areas, sometimes with disastrous consequences. As these sorts of programs increase in power over time, and as we bring more and more of them into existence, it's plausible that a large number of new opportunities for disaster will present themselves. In that scenario, to get to a point where we need to worry about an intelligence explosion, we'll have to have successfully navigated a number of good old fashioned problems, politics, warfare, greed, pollution, terrorism. Each turned up to 11 by whatever power, dumb, single-purpose AIs have to offer. Bostrom doesn't find this possibility sufficient cause for concern, saying software that uses AI and machine learning techniques, though it has some ability to find solutions that the programmers had not anticipated, functions for all practical purposes like a tool and poses no existential risk. You know, just a tool, like bioweapons, nukes, or PFCs. Again, this might be perplexing to folks who aren't already bought into Bostrom's premise. Why aren't we concentrated about a human using machine learning tools to deliberately or accidentally destroy humanity? Why is our perspective so rigorously constrained to the possibility of an omniscient machine-god? It seems reasonable that Bostrom might have only wanted to write about super-intelligence and skipped over anything that was even slightly off topic. But again, if you're the sort of person who would buy a book called super-intelligence with a cyber-owl on the cover, you might overlook the omission of more pedestrian apocalypses that don't involve super-human AIs. Speaking of which, the last assumption I want to look at is right there in the title, intelligence. Intelligence is notoriously difficult to define. We know that a rock, a dog, and a philosopher have different capabilities when it comes to interacting with the world, but drawing a tidy box around a subset of those capabilities and saying this is what makes a philosopher smarter than a dog is squirrely and prone to weird circular justifications. It might seem like a nitpicking language problem for pedants, and fair, guilty is charged. But when you're talking about an intelligence explosion and what that entails, you're magnifying something to incomprehensible proportions, and how you define that something is going to determine what conclusions you reach. Rocks and dogs don't struggle to get out of bed because they're crushed by existential despair about the ultimate meaninglessness of existence in an uncaring universe. They don't develop technologies that might potentially kill them. They don't engage in self-destructive behaviors and suffer because they feel powerless to stop them. There are obviously things a philosopher might be able to do with their huge brain and education that dogs and rocks can't, but if we define intelligence as something like a combination of mental processes directed toward effective adaptation to the environment, a dog might well be smarter than a philosopher in certain contexts. That's obviously not the sense Foster means, nor the sense his readers are probably here for, but we aren't really treated to a deep analysis of what intelligence is. He's pretty much content with whatever lay definition his audience decides to extrapolate from. Despite his warnings not to anthropomorphize artificial superintelligence, we might wonder how many of those readers might be conceiving of intelligence in a particularly narrowly defined way, and whether they're imagining an AI god in their own image. Now, I don't want any of this to come off like a critique of folks who find the runaway AI problem interesting to think about. I certainly do. The thing that I'm trying to convey is that the problem itself and Bostrom's framing are shaped in a way that makes them particularly convincing to folks with a certain way of looking at the world, perhaps to an extent unwarranted by the strength of his assumptions. Maybe for those who have found themselves despairing that the AI apocalypse is nigh because his reasoning and conclusions are so compelling, it might be worth taking another look at our starting point and wondering if it's in the right place. Isn't that what AI alignments all about? Are there reasonable grounds to question Bostrom's framing of his intelligence explosion? Are there maybe more pressing catastrophes we ought to be worried about? Please leave a comment below and let me know what you think. Thank you very much for watching. Don't forget to blah blah subscribe, blah share, and don't stop thunking.