 Good morning slash fireside stage. How's everyone doing this morning? Up bright and early Can I get a little round of applause if you are out late? very good Me too So I have here very Special guest yon Tallin who is the founding engineer of Skype and Kaza and one of the leading thinkers on AI and existential risk I for my part am not nearly as impressive, but I'm here to help yon Explain some pretty complex topics to you So maybe let's just quickly start yon. Could you give us a brief introduction to who you are? What you did what you're doing now? Yeah, I I made Skype is something that I sometimes say among other people and After Skype for the last almost decade or so I've been kind of looking at What could be the big problems that humanity is not paying sufficient attention to? found the topic called existential risks, which is risks from Various adversarial developments that might either kill the entire human species or permanently limit our future potential as one of those really Relatively underappreciate the topics that I'm focused on so stuff We should be thinking about as a species as people working in tech as we you know Kareem forward into the future how we make sure that the future exists in a way that is compatible with people us yes, so Martin Reese Lord Martin Reese who is my co-founder at the Cambridge Center He says this thing that I think he's actually quoting Burn on Russell in this but I'm not fully sure that there's a difference between killing 90% of Humanity and 100% of humanity and the difference is not 10% right exactly Well, I'm glad you're working on it. Could you so you've sort of talked to us about what existential risk is but Maybe tell me a little bit about how AI fits into this schema and and why When we're so seemingly far away from Terminator 2 we should be thinking about AI in the same sentence as existential risk right so One way of looking at it is that we are witnessing a tail end of a hundred thousand year period and that hundred thousand year period Was so called human era on this planet and meaning that the most powerful Future shaping force is and was The human brain like if you look around everything you see here went through some human brain as As it he or she was planning What to do with with with a future so once we are yielding that power to non-human minds And we're kind of doing that increasingly so But once we have systems that are as powerful as humans in controlling the environment We are risking to lose control over the environment and because we are very very Delicate when it comes to Environmental changes that's why we send robots to space That's what that's why we send robots to highly radioactive areas because they don't care about the environment And we don't know we still don't have a good idea how to make them care right so That segues quite nicely into my next question so environments We're living in a in a weird world where we have some interesting people governing some of the more powerful nations on the planet We face runaway extreme climate change ostensibly caused by humans bioengineering Nanotech and a bunch of other stuff that is potentially quite scary and bad Why are we focusing on the far future and on AI is an existential risk when we have more pressing matters that? Really deserve our full attention now. Yeah, so I mean the many answers there one is that like you're gonna supposed to Chew gum and walk at the same time like if we don't we shouldn't like focus on one thing Is it like climate or is it like nuclear like we should really focus on everything because I know we don't exactly know what what might Hit us first So I totally support Like focusing on other other areas as well in fact the Cambridge Center is not about AI risk just it's it's actually Also thinking about things like by risk from synthetic biology and nuclear But I think even then there's like a difference between between different scenarios the difference in impact just like with as Martin Reese said that Like there is a difference are we talking about bad scenario a catastrophic scenario that actually might kill everyone are we talking about merely? catastrophic risk like one Like the person who actually got me involved in this entire thing a laser at Kauski in California He got once really cynical is said that we should not get distracted by catastrophic risks that Are so trivial that they might leave survivors, right? Very positive worldview Let's Let's just assume for a second So I want to play skeptic because I've been talking to people over the last couple of days about AI and AI safety and existential risk and there's been a sentiment amongst the people at least here at slush That we're talking about something that's so far removed That it's just not something that deserves much attention. We're too far from actually seeing it develop in a way that is potentially averse to Humans, so maybe you could just sort of fill us in on the various strands of thought around timelines So, you know, we've just seen some interesting developments with alpha go zero There's a bunch of really smart people working on this to just sort of careening forwards towards the development of a general AI Where do you sit amongst the fray and how soon we can expect something? Generally artificially intelligent to to come about I think a correct Kind of position to take there is that that there's like a lot of uncertainty I Think people who say that like this will like a never happen or this will Not happen in 300 years Like sort of they might not realize that they're making a very very confident prediction there is just not enough evidence to to Confidently say that this will not happen in in 100 years to say nothing about 300 or never So they're just wrong Just to say that Unless they know something that that like people that I talked to a deep mind and open AI etc. Don't so And like the thing is that if you are really uncertain about the situation That doesn't mean that you that gives you a luxury just sit back and and relax Because I can imagine if you're on a plane and now There's an announcement that 40% of People or experts aviation experts that think that the display is going to crash if it takes off but don't worry like 60% think that That it's gonna be okay. I'm not actually using those percentages not coincidentally that was the result of Survey last year or published AI experts and 40% now think that this is a serious issue The super intelligence control problem So like if there's a announcement like that on a plane, you already know what to do. You shouldn't take off people are gonna freak out Yeah, so it's like like it's just like we never will never get to the situation where 100% of AI researchers say that okay This is serious thing Because like yeah, Eliezer Kowsky wrote this wonderful essay recently called there is no fire alarm in AI The the idea is that the point of fire alarm isn't what people think it's not The purpose of fire alarm going off isn't to inform people that there might be fire The purpose of fire alarm going off is to create common knowledge that there might be fire so so it's not just saying that there might be fire but saying that like Making people know that other people know that there might be fire So it's now okay to act as if there was a fire It's likely that we will never get such situation with with AI So but there will always be dissenters who say that like this is never going to happen Sort of like the frog in the boiling water if you throw a frog into boiling water it jumps out because it's hot But if the water boils with a frog in it, it just sort of cooks. Yeah, and like in technology We have had actually similar situations before Like when heavier than air flight the will will work with some brother with some brothers I might the right brothers right brothers exactly if it when heavier than air flight was 500 years away it felt like it's 500 years away when it was five years away It still felt like it's like five five hundred years away when it had already happened many people still thought that it's five hundred years away So so we have experience with things like that. There are nuclear weapons like After after the First test of nuclear weapons the nuclear weapons seemed like an impossibility in fact That's rather for it This missed nuclear weapons as moonshine Just 24 hours before they were invented, right? So so like people Even it doesn't have a great track record saying that something technological doesn't happen Yeah, so prediction is prediction is tough So let's let's take for granted that this is going to happen and let's say that it's going to happen at some point And in in the nearest future can is it fair to say you know a hundred years with a 50% probability? I mean I'm like okay one thing that you have to you know qualify is that Like pro like probability conditional and other disasters because like I do think that there's a there's a significant probability that Humanity will never great AI human level AI, but unfortunately It means that we screw up in some other way that that we destroy civilization or perhaps see species using some other Metal such as synthetic biology, but conditional that we will not do that and the civilization will kind of continue to exist Myself I would give a significant the higher probability During the last during the next century than the 50% my 50% right now is in about 25 years or so Okay, so let's take let's take that as an assumption that we make about when things are gonna happen if that's the case How should we be thinking about this because it seems to me that Before we can start to think about safety around something that's so Abstract we need to essentially have a framework through which to consider it And I think most people tend to think that that framework is a philosophical one So maybe can you talk about how we should be starting this conversation amongst people working in this space? Yeah, so I think that want to there's like a really interesting and fertile intersection between philosophy and Computer science right now people who are kind of thinking about algorithms thinking about agents in general While being informed about the philosophical considerations things like and tropic principle and Like how should we think about multiverse things like that because like like one way of putting it is that Imagine if we still had the understanding of the world That is composed of four elements like fire Earth air and what was the fourth one water water Then like and we try to using that knowledge to create an AI Like we would and try to explain to AI what is a good future that we want to that we wanted to create using those four elements I mean would would just get a random outcome Because like this our ontology is just completely screwed up and more realistically if you hadn't realized that the quantum quantum physics is a thing We still were in Newtonian Under our philosophical understanding is that we are living in a Newtonian world We still would be able to create computers and possibly make them really powerful that if you if you tried to Wire in Basic ontology of Newtonian physics Into an AI that then discovers that wait a minute. We are actually not in a Newtonian world We're gonna have what is known in philosophy as ontological crisis and this is like one of the under like really deep examples I know a friend of mine is doing actually is Philosophy PhD on this topic. That's really relevant to AI, right? Yeah, future The other the other term that gets thrown around which is slightly heady is coherent extrapolated volition Could you you're gonna explain it better than I will so maybe you could just quickly tell us what what that is and why we should be thinking about it right, so So overarching term for what we need to do in order to ensure a good future with AI is called value alignment I kind of Once we admit that we can't probably we can't control something that is super humanly intelligent a friend of mine says that if You are a superhuman AI Waking up in its box that has been built by humans. It's like a human waking up in a prison That has been built by a bunch of blind five-year-olds who know nothing about prisons nor vision, right? So if you don't will not have the ability to control superhuman AI We have to ensure that is so-called value aligned what it means is that the its idea of Good future is identical to what our idea of good future is or more importantly What it should be because we know that that over time humanities has humanity has gone through things like like Moral evolution like we have had like insights like Rals for example last century had these like wonderful insights things like Vale of ignorance for example at the principle of common good so the like We are clearly evolving like so so we shouldn't like fix our current values or our current idea of what the good future is So we should basically take one step back and define our future good future as something that we would want if we were Smarter if we were kind of more coherent more like better. Yeah, like Yeah, more coherent more Consistent in what we want and that's basically is called coherent for extrapolate evolution again another philosophical term, right? So I want to talk about we have a couple of minutes left There's a bit a couple of questions about neural link You gave me a good analogy backstage that essentially trying to create a link between an AI and a human brain in the way that neural link is trying to do is sort of like trying to compare horses to Super cars and saying we just need you know a horse with a stronger rope Yeah, the exact code was by friend of mine who is Economist at George Mason University Robin Hansen. He said that like trying to make Humans Competitive with computers is just like using neural interfaces is like trying to make Horses competitive with cars by developing stronger ropes much much more well said than than I did so we have a minute left What resources should people here be thinking about looking at if they are interested in AI as a field of Study and ensuring that we don't all kill ourselves Yeah, you can go to future of life Institute future life.org for example, and they're like plenty of materials And not just an AI risk, but But other other potential extension risks that you can kind of look at what is the state of the field and in general? I do think there's this going to be like a massive shortage of talent We have gone from like what I call reputation of bottlenecks people were not just taking seriously when they talked about AI and other technological developments as the extension risks Then we had like a period of financial constraints and these are being Largely lifted now. There are more and more interesting in potential funders to to actually invest in these areas And now we're gonna have talent bottleneck. We need as I say we need to make AI safety sexy I had a conversation with an AI developer who said that like okay I'm now completely sold that I need to stop like doing my own current work and start focusing on AI safety But like I get so much recognition when I do AI capabilities research like if I push the boundary a little bit like I get like applause all around and if I just Expect like if I do AI safety research how people will go like, okay That's competent and then they just want so like we need to make a I say to sexy That's that's one got it So the takeaway is if you're you're thinking about AI you're in school. You're doing AI is a startup. You should also be thinking about the safety concerns and And perhaps consider moving into that field So we are out of time. I'd like to thank you all for waking up early and listening to yon and I or yon talk And and and I hope you have a rest a nice rest of your slush. So thank you for joining us. Thank you very much