 Welcome everyone and thank you all for coming to our artificial stupidity panel discussion hosted in tandem by Studium Generale and Enigma. I'm Nicholas, a second year CSAI student and I will be the host this evening. As much as I would like to listen to these amazing speakers talk for hours on end about the topics we have at hand, it's my unfortunate job to stop them. Before we begin, I'd like to say a quick word about Studium Generale and Enigma. Studium Generale organizes all kinds of extracurricular activities like this one on relevant and pertinent topics such as symposiums, lectures, debates, often in cooperation with study associations like ours. The Studium Generale graciously helped make this event a reality which, by the way, you can also receive a Studium Generale certificate for attending if you're a student by attending five events, including this one. So if you're more interested in that, you can put it on your CV. Just Google Studium Generale certificate. So they graciously helped us make this event a reality with my fellow Enigma members. Enigma is the newly created study association for CSAI and we're looking forward to providing you with many more events like this one. Now, on to the event itself, we will have a first discussion round followed by a 10-minute break at around 5 p.m. and then a second discussion round and finally, if we have time, a rapid fire round followed by drinks at the cafe. So you've hopefully all received cards, so I'll explain how these work. We have green for a green, red for a disagree, blue for a remark, and pink for a question. I very much encourage you, all of you, at any point during the discussion. If you have a question, if you have a remark, if the speaker says something interesting and you'd like to participate, just raise your heart, raise your card. My friend Dennis here will come to you with the microphone so we can get picked up by the recording. He will ask the question for you so you can address us directly through it. Now I'd like to introduce our four talented speakers. First we have Frank Posman, who is a researcher for the Department of Practical Theology and Religious Studies and teaches various courses for the Tilburg School of Catholic Theology. During this discussion, Frank will draw from his expertise of theology, faith, and video games by presenting games which use AI as anthropological mirrors. Sitting next to him, we have Pim Haslager, who is a professor of societal implications of artificial intelligence at the Department of AI and the Dunders Institute for Brain Cognition and Behavior at Rabown University in Nibagin. In his research, he investigates implications of deploying AI in neuroscience, raising the question of how to ensure the responsible use of new technology. Now from the Department of Cognitive Science and Artificial Intelligence, we have Afra Ali Shahe. Her main research interests are developing computational models for studying the process of human language acquisition, the emergence of linguistic structure in grounded models of language learning, and developing tools and techniques for analyzing linguistic representations in newer models of language. And last but not least, we have Joshua Ekblad, who is a lecturer and director of the Iconic Hub for Entrepreneurship Research and Education Tilburg at the Tilburg University Management Department. As a former high-tech entrepreneur, he gained experience raising venture capital and building strategic relationships with established high-tech firms. So please, a warm round of applause for our speakers who are coming here tonight. Thank you very much. Now I would like to get to why exactly are we here? Well, you often hear the praises of artificial intelligence and how it's the future. But what about artificial stupidity? We're here to open a conversation about how preparing for the future should not be careless and how artificial intelligence, if misused, could be dangerous. So starting today's discussion, we want to talk about an example that brings to light the dangers of glorifying AI and how artificial intelligence could become stupid through us. In 2016, Microsoft released the chatbot, Tay, a story some of you will be familiar with. The AI was a natural language processing program that was designed to reply and talk to people based on other people's interactions with itself, on Twitter, actually. And only after 16 hours of launching, Tay was shut down. During its short-lived life, it became a racist, fascist, and inflammatory bot. As you can see by the quote, for example, do you support genocide? It replied, I do indeed. Now this is one of the tamer quotes by Tay. We hesitate to put anything worse, but it said much worse things. Tay is an example of how releasing an artificial intelligence that is not sufficiently ready for a task can have drastic consequences. Furthermore, it comments on how the interaction between the public or user base and an AI can cause a good idea or a good AI to become artificially stupid, assuming, of course, that the data the AI was given is one of the reasons why it had to be shut down. It comments on how the interaction between the public or user base and an AI can cause a good idea to become a bad idea. And you can actually see a quote from Tay themselves when asked or when told you are a stupid machine. It replied, well, I learned from the best. And if you don't understand that, let me spell it out for you. I learned from you, and you are dumb, too. I would like to note that this is a problem of intentional tampering. Tay was actually not automatically driven towards inflammatory or racist behaviors. But when the internet got hold of how Tay was programmed, they decided let's have the fun of it and make it as racist, homophobic, misogynistic as possible. They talked to it as bad language as they could because they realized it would learn from their behavior. So with that, I would like to get to the first question to our speakers, which would be since it was the mishandling of the users that led Tay to become such an inflammatory bot, do you think that it is necessary to limit or control the interaction between the AIs and the public? So would you like to start the suffering? First of all, this Twitter account reminds me a bit of Trump. So maybe Trump is also using artificial intelligence, but we have to look at that. So I would say that I think it's important to educate as much as possible. So keeping things from the public is always a bit difficult. People are not up to it. We have to keep it quiet. We don't have to publish this. The people are not ready for this is something that rather a dictator would say than someone who has a benevolent intention. So I think we should not keep anything from the public. But as a university, we have a task in that to educate people as much as possible and make people understand how these things function, how these things work, and then we have a discussion about it. So hiding something I think is never a good idea. Right. Would you say then before you can release it to the public, do you have to very much understand how it operates? Do you think Microsoft was partially at fault for not predicting that the users or the internet would react in such a way? Well, I think there are three possibilities who are to blame here. So you could say the public, the artificial intelligence program itself on Microsoft. I say the second option is not valid. I think an artificial intelligence is not a moral agent, does not have moral agency and can therefore not be morally responsible, cannot take the blame. And I think in this case, it is a combination of Microsoft maybe not realizing enough how these things could be so easily being manipulated. At the other hand, of course, the trolls of the internet having a good day and just being trolls and trying to make fun of everything. So I think it's a shared responsibility in this case. Right. And on that note, actually, I would like to ask Penn, in terms of moral agency, it could be argued that only humans are able to have responsibility, right? An AI isn't a moral agent. It can't make such decisions, so it shouldn't be hold accountable. What do you think? Do you think Microsoft or the users are solely at fault? Yeah, the thing with computers is you cannot punish them. So that's a really big problem from an ethical perspective or from a responsibility perspective. You know, you can destroy a computer or a robot and it cares as much about being destroyed as my refrigerator cares about it, which is absolutely nothing. Right. So that's what I think, yes. Would you like to counter on that? No, no, no. Yeah, there is, of course, always the question like how do you know when you get into solipsism and the other mind's problem? If you want, we can go there. But in assets like a thermostat and a computer are basically processing information, but they're not grasping it. They don't understand it. And without understanding what the information is about, it will be humans that remain responsible. And the big challenge for us, of course, is how can we maintain that responsibility while dealing with systems that sometimes outsmart us? That's a very paradoxical situation that we created. We have smarter systems than us. And at the same time, they don't know what it is about. Chess computers beat the world champion, but they don't know why winning is more pleasant than losing. Both, you know, we just programmed it anyway. And I have a chess computer program that I put in moron mode and then I beat it all the time. And the computer doesn't care. It cares as much as about winning. And so we somehow have to deal with the machines that we created or that we let learning take place. And then how are we going to do that? That's the big societal challenge, I think. So would you say that no matter how intelligent we make these agents, they would never be capable of, for example, how happened their own decisions, their own responsibility, their kind of simulations of? Never is a dangerous word. So I don't believe in the soul. I think humans. Yeah, so sorry. I think humans, all our behavior, all our cognition should be explainable ultimately in terms of physical particles that operate according to physical laws. So sooner or later, we will be able to understand how we understand, how we feel, et cetera. But I simply observe that currently, whereas in the dimension of intelligence, we have an enormous progress, but in the dimension of sentience, we're still at zero. And there's no progress also, which is fascinating in a way, too. It tells us something about what we find interesting or not. And we are very fascinated by our own intelligence. So we reproduce that, and we understand that better and better. But what we share with animals, sentience, feeling, we actually don't study so successfully. And we're not really capable of understanding where that comes from or how we should model that, which might be a good thing, too. By the time machines start feeling, I get scared. I would actually like to ask, if actually, Frank, from the perspective of the theologian, do you think humans have a soul? Or is there something intrinsic in the material? I'm sure you're all familiar with Sheryl's Chinese Room experiment, which essentially says that no matter how advanced an artificial intelligence is, it is only replicating instructions that it's told to do. It's a deterministic program. You give it input. It gives you output, but it never learns. In the metaphor itself, you tell it how to translate Chinese through symbol manipulation, but it never knows Chinese. It never knows what it's saying. You just feed it English, and it gives out Chinese. So would you agree with Sheryl in that state that you need some more material to give rise to sentience? Or is, for example, sophisticated enough intelligence enough to give rise to sentience? Well, I think that discussion of the soul is a very difficult discussion. So I will just place it very gently here. So we just leave it there. But from a theological perspective, I think it's very interesting to see that artificial intelligence and robots in our everyday cultural life, in our books, in the games, in the series, in the films, in the novels that we read, you see that artificial intelligence and robots are used as a anthropological mirror, like in the medieval times, in medieval philosophy. The angels were used as a anthropological thought experiment. So what we think about robots and what we think about artificial intelligence is, well, more often, yes than no, it's a reflection of what it is to be a human being yourself. So for example, if we think about, we want to recreate something that is resembling us, like an ima jodei. We resemble something that is very close to us. And then we say, well, but it does not have any moral agency. Aha, so we think about ourselves as one of the defining characteristics of a human being is that it is capable of having moral responsibility and therefore free choice and things like that. So I would agree with Cheryl in the Chinese room experiments that a computer can very much simulate sensations or can very much simulate a polite conversation from the Turing test. It can simulate having a conversation like to take a computer, but it's only a simulation. It's just reproducing what it has been told to reproduce, but it's not reflecting upon its own existence. Or you should say, well, what we experience as reflecting on our own existence is also a simulation. Then you go down the rabbit hole and you end in simulation theory, which is also very interesting. Because you can never prove that the person you're talking to right now is also conscious. And that's exactly synopsistic rabbit hole that you go down there. Yeah. And actually, if you think about T for example, even though the content of their tweets was very wrong, they're actually very convincing. They would probably pass a Turing test, right? If you showed their tweets, you wouldn't recognize that it's not a person tweeting them unless you were told that it's about. So actually that's a question I'd like to pose to Afra, which is assuming that T did its job perfectly, right? Since it is actually very convincing when it tweets and the stupid part was from whoever released it or the users for example, can an AI ever be stupid? Or is the stupidity of just how it's handled? Because if you think about it, T for example, that exactly as it was programmed to do, right? So actually, the danger of aging myself badly, I wanna compare this more shiny, recent, seemingly very advanced systems with what was common for us to play with as bachelor's master students in the 80s, which was, you probably have heard of Eliza or some of the people in this panel. So there was this rule-based system which was basically a psychologist, right? So you would chat with it and you would talk about your problems and it would say, tell me about your mother. And then you would say things about your mother. And so it would say, tell me more. I get that you are, and it was a completely keyboard-based, a keyword-based system. I don't know what your definition of stupidity is, but comparing with modern machine learning systems, this was a very stupid system, but you really had to work hard to break it. As in, I know that some people actually started confiding to Eliza. It really sounded convincing. Stupidity from a machine learning point of view has definitions, right? So you have evaluation metrics, you are gonna evaluate the system based on cases that the system has seen before and trained on cases that the system hasn't seen before, to what extent it can generalize, to what extent it basically memorizes. None of these really, I would say, translate very directly with the general opinion of what a stupid being is. And I don't even think that it matters that much. I mean, going back to your original question, should we release AOI to the public? I don't think that this question is well-defined enough. What are we trying to protect the public from? If you give Tay as a fun experiment for people to play with or to interact with or Eliza, as long as you make it clear that this is an artificial system, this is something that doesn't represent the opinions of a crowd as long as Microsoft is willing to admit responsibility for whatever harm that the system does, I personally don't see any problem. But if the Ministry of Health then releases a chat box for teenagers who are having suicidal thoughts and puts it in charge of talking them out of it, and then that system does something wrong that actually persuades the teenager to actually do the deed, that's where the real danger comes from. Not developing on itself is not an issue using it. And what condition you are using it is there. Right, well, you could say that, we saw that Tay was shut down after only 16 hours, likely because it reflected very badly on Microsoft. Do you think that if everyone understood that Tay is kind of an experiment, that even when it became as horrid as it became, that it could stay up, for example? I mean, I don't think that Tay said anything that human trolls on Twitter or on other... Yeah, actually, it repeated exactly what they would say, yeah. So I don't control the trolls if you wanna ban racist comments on Twitter or on TikTok or whatever platform is the platform of your choice, you apply the same filters to Tay as well as the anonymous... The rest of the user base. Who, yeah, it's hiding behind there. Would you say something like freedom of speech could apply to Tay, fair? Oh, I'd like to hear your issue of that. Do you have anything to comment on that? I was just gonna say from an organizational perspective, we just look at what Microsoft was trying to achieve. So part of my professional experience is I was a tech entrepreneur working with artificial intelligence. We were developing systems like this. So this is a conversational bot. In our case, we're using natural language processing to extract certain things. But at the end of the day, we're running these experiments. Microsoft is running an experiment. As a startup venture, we were running experiments and you put it out there in the real world and you see how it behaves and how it interacts with people, with society. And at the end, I think Microsoft learned a great deal. They started with this, it went down, but then they came back with Zo. And Zo introduced filters. Then you can get into this idea of free speech or not because the filters themselves are trying to catch these things early on. Eventually that's developed into Cortana. Then they try to compete with Alexa and with Google. So I mean, these are products and services that they're working on that they're trying to develop. And they were unsuccessful. I mean, Samsung, they had a licensing deal with that, but there's a lot of money here at stake. There's a lot of economic value, two conversational bots. And there's all kinds of startup ventures or hundreds of them. And even just in the Netherlands, there must be at least 50 startups that just started up in the last few years that are doing this kind of stuff. So the potential economic value is there. You need to run these experiments in order to see how they perform. I think Microsoft learned something from it. So there's learning. It's not the kind of learning that we're referring to here in terms of machine learning, but it's learning about how to deploy these in society. And I like it, apart from the kind of economic perspective on it, I think it's great that it rendered transparent something that, and this goes to your point, that it already exists in social media that isn't always visible because of these sort of walled gardens or silos, right? Because in social media there's even special platforms that are just for certain types of thinking. And so that's even becoming more and more entrenched and more and more difficult to see. And so this brought it out. Now, of course, this is an old example. I think it's 2012 or 2013. I think it was 2016, I believe. Okay, 2016. Yeah, so it's a long time ago. And we have a lot of other examples to show how these entrenched ways of thinking and how certain platforms are actually vying for that entrenched way of thinking. So from an economics perspective, even though it went down the way it went, you'd say it was still very useful from the point of view of an experiment for Microsoft. Well, it helped them get to Cortana, which is something they were trying to... Develop? Well, they developed it, but they were trying to generate licensing fees from it. So they were really trying to get people to adopt it, certain large brands to adopt it. That didn't really work out. And now the sort of in a discontinued Cortana in terms of an API. Now it's a service that they will use themselves for Microsoft enterprise offerings. So it goes to a kind of virtual... It'll end up feeding into a virtual assistant type of service. And that's the kind of stuff we were working on too as a startup. We were building these virtual assistants for enterprise. And there, there's just a great deal of economic value to be had if you can achieve that. But also what it showed is how difficult that is to achieve. And it's so easy with virtual assistants for the stupidity or the limitations. Where it should go wrong. Right, and in this case, the economic stupidity, I would say, is the fact that the service felt short. So it's not an existential stupidity here. It's a very practical stupidity, which is it didn't work. It clearly showed its shortcomings and failings. And of course, for a service like this to work, you have to convince. And we experienced this firsthand as a startup and I did multiple startups in this area. It's very, very hard to achieve. To pass the sort of, it's not exactly a Turing test, but some kind of test where people are doing these robustness checks and then it still comes back as performing in a way that you need it to happen in a business context. Right. Any closing remarks before we move on to the next? Or do we have any questions from the audience for the speakers? Yeah? Dennis? Frank? Frank Pinn. They both have great beers. Oh, yeah, for Pinn. So just to repeat the question for the recordings, the question is about sentence. And how we should go about to measure sentence. So let me explain. I don't think we have a good test for that. What we do have is an argument from analogy. We are biological systems. We know from our own experience that we feel there is something it is like to be me. I know what a feeling feels like when I have it. Hitting my thumb with a hammer, for instance, very vivid. And by analogy, I attribute those states, those experiences to other biological systems. In the case of computers, robots, thermostats, refrigerators, there's no such analogy. So I would think that the burden of proof is actually on those who want to make claims about the sentence of, for instance, lambda. You remember the chatbot from Google recently this summer. So I know there is a problem with the other minds. You have the argument solipsism. We've been there many, many times ever since the Turing test and Sol's Chinese Room. But the main thing is we can be generous towards systems that are similar to us because we can extrapolate from our own experience to others. And in the case of artificial systems that really have a totally different type of inner structure, I think there's good grounds for being extremely skeptical. I want to see the argument. I want people to show me how it is that that system is capable of feeling. I think that that's a fair request. Does that make sense, as an answer? Which is also why I think that Thai, for instance, doesn't have human rights. Freedom of speech is a human right. Excuse me. You can, of course, say, OK, animals, we should expand this. And there's even debates about rivers these days. Having some sort of rights to facilitate protecting the environment. But a refrigerator, a bridge, Alexa, Thai, and those kind of systems, Pepper, the robot, they don't have rights because there's no capacity to suffer. At least that's a position I would like to defend. Well, the thing about saying that you feel certain things, that's very easy. The first program I wrote in the 80s on the Commodore 64 was, hello world, I'm happy to be alive. And the computer printed it. It was not a program to speak of, I've never been good at programming. But OK, I did my fair share a little bit. And of course, it's easy to make a system. Jerry Fodor, a famous philosopher of cognitive science who no one remembers these days, said, Disney World is not a major scientific achievement. And that's true. So we had, I was in the Bournemouth for Beininger with a robot project as part of an exhibition in the museum in Rotterdam. And we only used the standard live mode that you have on Pepper where it starts following and moving around a little bit. And people fell for it like a ton of bricks. They found it extremely interesting, exhilarating, spooky, all kinds of things. We over-attribute. Sorry if we could just wrap it up because we need to move to the next. Yeah, sorry. I get excited here. Yeah. OK. All right. Very interesting discussion. Sorry to wrap it up so quickly, but we are hourly here. So on the next topic, we would like to talk about AI in military applications. Again and again, we're faced with the reality of stupid AI. And what does this mean for the implementation of AI technologies with the ability to kill? Now, I'd like to pose a hypothetical, a not so distant future where AIs have a very large involvement in strategy, decision-making, and possibly even be the boots on the ground for military operations. I'd like to ask, to begin with, actually, what could artificial stupidity lead to in employing military AI incapable of the ethical and moral codes we humans have? I talk too much. You already noticed. So you have to shut me up very quickly. Well, first of all, of course, what we get, what we already have, is a lot of innocent victims. What they call, what's it called again? Collateral damage, which is a word I really, or phrase I really hate, because systems are too stupid. Situation assessment, for instance, is something that is really computationally hard. It's exponential. The number of features that you can distinguish in an environment will grow explosively. The more fine-grained the distinctions are that you can make. So the better perception, the better reasoning they can do, the bigger the problem space comes, and these systems get lost. Also, the sense of relevance is very difficult for machines, still. When is something actually relevant? That's a very hard problem to crack. Actually, on that note, I'd like to ask Frank, if I understand correctly, you think disobedience is actually a defining human correctness. Well, I don't think that. But some of the computer, I'm an expert in video games. And yes, I like to tell that on parties. And I played the Turing test in the Telus principle, for example, and there are two video games that try to think about what it is that makes us human. And they do that by employing very sophisticated artificial intelligence systems, Elohim and Tom. And both games at the end, that's my interpretation, conclude that what is the defining characteristics of being human and which a robot should have in order to qualify as a human is disobedience. And for a computer program, that would mean going beyond its program, disobeying its program, going against set rules, making its own rules and things like that. And that's, of course, all very hypothetical on the technology of artificial intelligence we have now. But it's like thinking about what human beings are. And one of the characteristics that human beings have, we can disobey or we have free will. And with that free will, we can decide to go against the rules or against the current or against obligations or against a government that wants something from us or our university that wants something from us. And then the interesting question is, OK, do we want robots to be exactly like us? If that means that we have to program them or have to allow them, that they become not only stupid, but also disobedient. And I only have to quote the example of the Horizon Zero Dawn game by Gorilla Games. That's an absolutely fantastic thought experiment about how that should end up. Actually, on that note, I'd like to ask the FARC table. Should we, to some extent, program disobedience into AI systems? For example, if we have an artificial intelligence in the military, should it be able to be disobedient enough that if it recognizes an order is an ethical one, for example, should it be able to disobey a command? I'd really dislike this military domain. I mean, any discussion of development technology specifically for the military application makes me very uneasy. So if you allow me, I would switch to. I'm sorry, if I could hold the microphone closer. I will switch to health, right? So if you, because it has similar characteristics, let's say you have an expert system that is supposed to diagnose cancer. And if it makes mistakes, it can take lives. So there is gonna be collateral damage. There is gonna be added efficiency if you can do it, is it better? Yeah, okay. So the question is to what extent do we allow these automatic systems to actually replace actual experts in these domains where life and death decisions could be made? It doesn't even need to be life and death if your decision is something along the lines of your own example of the Dutch tax system or anything that affects the lives of people in any way. And the core of the matter is do we allow these systems to take over and make decisions without involvement from humans? And that's the part that feels wrong, right? I don't think that we can really say, oh, these systems are not good enough so we are gonna put them aside. They are very good in certain aspects, in certain domains, and they really help the process move forward faster as additional sources of information as long as we admit that we cannot trust their judgment 100% and we need human experts to just use them as aids. Now, disobedience, I'm sorry, but from an insider's point of view, I think that, I'm saying that I'm sorry because I wanna say that this exists in many machine learning systems in the form of regularization. I don't know how many people have had any courses in machine learning. The idea is don't trust the data that you have seen 100%, right? There is always room for things you haven't seen before so leave space for that. And if you actually don't do this regularization or any variation of it, meaning don't leave any space for decisions based on things that you haven't seen yet, then your system is probably not very good at generalizing to new situations or being transferred to new domains or facing unseen circumstances. So I don't think that when you put catchy names on things, people might react like disobediences. It causes rebellion and it sounds cool, but in reality and in practice, you have to do it for every simple machine learning based system if you wanna use them in situations where the system might wanna deal with data that it hasn't seen. In order to generalize. In order to be able to generalize. On that note, I'd like to ask if you think humans should supervise, you can also address this question. If there should always be human supervision to any decisions that an AI algorithm comes with, do you think there should always be someone above? Let me just take an economic perspective on what we were discussing. So I won't also talk specifically about the military. Of course. Because I don't think there's anything particular about that from an economic perspective. But at the end of the idea, this idea of disobedience or this idea even of things like military outcomes, they're value laden for one. And by value laden, I mean they affect or it should affect, in my opinion, human being life. And economic activity regards the condition of human life. So when we look at things like disobedience, we can think of, for example, unions and union activity in order to better wages for laborers who are trying to improve their not just economic conditions, but everything else that we know is correlated with that. So one's education opportunities, one's health outcomes are largely affected by economic conditions. So when I think of things like disobedience, or when I think of things like, you know, an example that you said if there's some kind of order in the military context, I think of those as people fighting for something of worth that meaningfully affects people and the human condition. And of course, there's a lot of AI already in all of that. So we're not just painting a kind of futuristic picture. There's a lot of AI already being used in military context. So for example, recently, so it was discovered that so the F-35 is one of these sort of fifth generation jets that's super advanced and it was discovered that there's, and it was discovered using AI actually through the supply chain. So the analysis of the supply chain of these incredibly complex products that have tens and tens of thousands of different parts. And then you have to think about each of those parts are themselves manufactured using tens and tens of thousands of other parts and processes which are themselves, right? So it's very hard for any kind of human to kind of figure that out. And they were using AI and they discovered that there's a Chinese alloy at the end of all this whole chain that somehow got introduced into making a part that makes another part that makes another part that sort of builds up into this jet. And so they looked at it, but you know, they would have been prepared to ground all of those fighter jets based on that. And so when I think about not just AI used in that context but when I think about what that could tell us about AI also in general in a kind of military maybe context is siloed forms of AI. So different forms of AI being developed by different parts of the world with very different value systems, maybe attached to them. So that's just food for thought, you know. Yeah, yeah, you could add something. So yeah, I fully agree with what Josh was said. Again, from a technical point of view, I also wanted to emphasize this, that whatever your question is, whatever your application is, you always map it to certain categories of problems, right? So a lot of these problems boiled down to classification. Now you train a system that classifies emails into spam and not spam, right? But you can also classify a system that groups people's faces into terrorists and not terrorists, right? So the underlying nature of the problem is the same but the labels that you assign to these classes have really different ranges of impacts on people's lives, right? Now, this notion of disobedience, but it also reminds me of a very, very basic concept in machine learning, which is basically the confidence of the model in predicting a label, right? If you force a model to spit out a label, it's giving an input set. And it calculates probabilities for these two classes and one of them is 49 and the other one is 51. If it's forced to choose the better label, it will spit the one with 51% probability. Right. It would look like it's the same outcome if the probabilities were one and 99. But it is really important to pay attention to how much information we have and based on what we are making these decisions under which conditions. And I think we have a question from the audience. Yeah, can I? Yeah, he will. Yeah, it might just be food for thought but I also have a question about the implementation of such systems because like we brought up before that situational computation is extremely complex. And I think there is not a single environment that might be more complex for decisions than the war zone, for example. Because I know that for example, when you take facial recognition for predicting criminal behavior that these systems already produce a lot of false positives so that a lot of people are suspected for criminal behavior, but the system just decided wrong because the environment in the war zone is super unpredictable and also needs a lot of moral agency by the AI system that you use. So I ask myself like how would you start even to begin thinking about this because it's a super unpredictable environment for the AI as well. Like how would you start testing out if the system would even work in the first place because the environment is extremely unpredictable and I think that we are still very far away from that if I'm not mistaken. On that note, do you think we should have simulations of the environment before? Obviously a simulation always has limitations and by definition a simulation is less complex in the environment, but if we make an agent, let's say for the military, an intelligent system, should it first be put into a simulation of how we think it would operate in the real world and then see how it works? Which is kind of, you're already putting an AI into a simulation, so. I was just gonna, I think it's, there's no simple answer obviously to this, but maybe something to think about is this idea of consequences. And of course we can say in some ways that AI, you could program into it somehow that there are consequences to the AI, suddenly loses processing power or something, that's its punishment, who knows. But at the end of the day, I don't find that particularly interesting. I think what matters is my point of view, but I think what's important is to think about human consequences. I don't know how we can spend a lot of time thinking about other consequences. I mean, this is the world that we live in, we occupy and we have some agency over it. And of course, going to war and the decisions that we make, I think should be our consequences. And if you, because the situation is the complexity you brought up, they also lead to very imperfect actions by human beings, there's no question about it. It's not just that AI can't cope with that dynamic and that complexity. It's people can't either. And they do all kinds of terrible things and they're terrible outcomes. But then we have these other institutions, we have legal systems and things like this that are there to handle atrocities that come out of these things. And there's some kind of accountability and I know you're gonna get to accountability at some point, but there's some kind of consequence to humans that we can really identify with. And that's what's truly important. That's just my opinion, that's where I'm at. After doing AI for almost 20 years myself in applied ways, I basically reached that conclusion. It's not the deepest conclusion to get at perhaps. It's not the most sophisticated. But at the end of the day, I just think at the end of the day, we just need to keep the human and our values at the forefront and build these systems in ways that we understand how they're working as well as the outcomes that we see. And I'll keep that in mind. Well, thank you very much. Everyone, we're gonna have a 10 minute break. So we'll be rejoining at about 5.16, 5.17. Please be on time. But yeah, you're free to go to the bathroom, et cetera. So to welcome us back, we are starting off with a hypothetical, which is the following. An AI engineered and sold by a private company, which has replaced the role of a radiologist, misdiagnosed the patient and its mistake led to the death of the patient. The family of said patients sued the hospital, holding them accountable for the fault of the AI. The court rules in favor of the family and sanctions the hospital. Now, I think for Josh, I'd actually like to begin by asking you if you think it would be wrong to hold the hospital responsible for the misdiagnosis or what do you think? Well, I would say that again, it'd be the certain institutions that are responsible for that in terms of regulation. And then just through a series of lawsuits, there would be some kind of judgment that emerges. So it's a social in the end, it's a socially constructed set of norms that will emerge in terms of who will be responsible. But in terms of an economic perspective on it, I'd say, well, a couple of things. One is of course the price of that litigation and the price of uncertainty will be factored into these services. So they'll, you know, they're meant in some ways. You know, this particular example has benefits in several ways. One is just to make the evaluation because we know the human evaluation, even for very clear set of conditions, it wavers a lot. It's very, it can be very unsystematic. So at least by using some kind of AI technology, you can make it systematic. Whether it's correct or not, at least will be reliable. And of course you want it to be valid over time. But we know that with, there's lots of studies that show that in economics that look at various types of decision making that needs to take place where that reliability is not there. And it could be anything from the time of day to what somebody had for breakfast to whether they're sun outside. I mean, you know, we're extremely irrational, even when we are exercising some very specific task at hand. And so we get these unexpected outcomes. So there's obviously an advantage to having that kind of reliability. And the other is there's possible productivity benefits that can come from it, right? So if it's an AI, it would take less time. And it's something we haven't seen a lot of. So again, economically, we've seen a lot of advantages or gains in productivity in the manufacturing sector, but very few so far in the services sector. In fact, this example you bring up of a, you know, you could also, it's some kind of, you know, hospital specialist as a kind of specialist in radiology, for example, that's one of the fastest growing areas in the labor market. So I'm using the US as a context because of the size of that market, but that that's one of the kinds of roles that is the fastest growing. So it's a labor intensive and that's not for AI, but that's for hiring people. It's a very labor intensive role that's actually increasing and it's very well paid in addition. So there would be potentially some advantages to using this kind of technology to see productivity gains made. And of course, in terms of health outcomes to improve possible health outcomes. Right, especially because the AIs could be argued that they have a better margin of error than human doctors, right? Well, there are studies to show that already. Yeah. Right, so we already know that that they've already done this for at least the last 10 years looking at the performance of human evaluators in the medical context who have, you know, who are experts and then these expert systems. So this is a type of expert system that you're alluding to here. And that these AI systems are much better. Yes, of course. This is totally true, of course, and very important. At the same time, two things. One, we hold technology to different standards than human beings. You know, I require guarantees about a bridge that I wouldn't ask from people that carry me across or something. So we have as AI people to learn that simply saying we're better than human beings in terms of less errors is not gonna be sufficient. We have to look very critically at the high standards that standard technology, think airplane industry or something is applied to and we have to apply that to ourselves. Second, human errors tend to be random. Like what you said, did you have lunch yet or not? Or, you know, is it raining outside? One of the big risks of AI is that there is systematic pattern when the errors occur. And that it's gonna be vulnerable people, marginalized groups that will get a large share of the errors than maybe other people. In part because they're being helped by systems, whereas let's say more rich get helped by people. So that's another thing that we have to be very aware of. So it's not undermining the argument. I totally understand that and less error is always better. But there are additional quality controls that we're still in the process of developing in AI, I think that we have to take into account more seriously. Frank, if you have anything to add, then Alfred. Yeah, and remind me of the problem of the dilemma. So I think we're all familiar with the thought of their experience of the trolley problem by flip of foot and then the doctor thought experience. So you'll have like five people. If you could actually, for the audience. Yeah, of course. So you have five people who are in desperate need of transplant organs, otherwise they will die. And you will have one perfectly healthy human being, which happened to have all the organs that you need and they're just compatible with one another. So the question is, can you sacrifice one person for the benefit of five others in the trolley problem and the doctor problem just to iterations of the same problem? And well, these kinds of dilemmas of course are very much found, especially, well, in military application, also in medical application, medical context, and also a thing that is extremely hard to, we find it extremely hard to make our own decision, let alone try to learn a robot to make that kind of decisions for us. Alfred. So I agree with all of the arguments that were made. I just wanted to make one extra point that it is dangerous to talk about AI as if it's one entity and humans, as if we have just a homogeneous pattern of behavior. There is massive individual difference if you put an extra, an experienced cardiologist next to an intern who was trained for three months. The, of course, both can make mistakes, but the nature of range of these mistakes are very different from each other. The same argument applies to artificial intelligence and to automatic systems. It really depends on how the system was built under which conditions it was trained, under which conditions it was tested. So of course, someone has to take responsibility. Of course, the hospital is responsible. They have to justify why they chose a particular system to replace a radiologist. They have to justify it. And we have all seen, especially during the corona time, that with the health system under extreme pressure, these automated systems can save lives, really. But such choices have to be able to, people who make these decisions making have to be able to defend them. Right. Actually, we'll take that question from the audience, Dennis. You can just give out the question to us and I'll repeat it. So if I understand the question correctly, it's about holding the company that actually built the AI responsible instead of the hospital. Just to take that up. Yes, but it's even more complicated because you have, for instance, companies that built neural networks and other companies that provide the data for training. So then where does the error come from? And that means, and this is what I said earlier about quality control, we really need an AI and this is really urgent, a very cohesive set of rules and regulations to determine who is responsible for what. Ideally, I once made a suggestion, for instance, with robots that they act under human supervision and they keep track themselves a little bit like a black box in an airplane of under whose authority they're currently operating. And so the moment you're like in a hospital, someone else would use the system, then the responsibility transfers to that person and you can track and trace where exactly the human responsibility resides. And in a, yeah, for instance, but we need to do that in advance. So the problem now is that we get these kinds of cases and law is not really suited yet and not updated enough to take care of such questions. And in a way that's irresponsible. So we really have to up our game in that sense. So in terms of accountability, we need to start thinking about these topics now. Yes. If you'd like to add. And also, this is a specific example of an evaluation exercise. So I think apart from the accountability question, there's maybe, as I said before, some efficiency to be had in some kind of evaluation decision-making exercise. But if we're looking at the healthcare sector in general, a service sector, there might also be good reason to have inefficiencies in that. And there might be certain things. It might be things that we actually value that we don't necessarily always want efficiency in that system as well. And, you know, that's part of the reason probably why there hasn't been that much productivity gain in the service sector versus the manufacturing sector. So even as these systems get more intelligent, it isn't always necessarily and we can get into sentience as well. But I mean, at the end of the day, there's also we gain a lot from social interaction and interaction with other human beings. And that's also part of our well-being and something to consider also in the health sector. So you're saying, for example, even if we find that it's more efficient to have an AI take charge of the job of radiologist, it could still be beneficial to have a doctor because for our own well-being, that type of conversation, for example, talking to the doctor as a person that might make a person healthier, it's good for our well-being. I would hope that we have human beings of great skill in both their area of expertise, but also in sort of their kind of emotional intelligence involved in the healthcare sector, along with AI, which can serve some very kind of rule-based evaluation exercise, where of course it needs the appropriate data to draw from in order to make sure also that it is performing equally well for different kinds of people. But that's an issue we already have with the healthcare sector, is we already know that much of the research is done only on males, white males of a certain age, and so there's already that issue in healthcare, and we wouldn't want to exacerbate it now with AI, but we have to still deal with those fundamental weaknesses we already have, which is we know very little about the physiology of female compared to male, and very little in terms of other ethnic groups other than white Caucasians. Do you think that's something that employing expert systems could that gap be bridged? If, for example, we're trying to do research into having more data about these groups that are less represented, could AIs help that? Yes, because again, it's like transactional. So if I think of it from an economic agency point of view, which is once we've decided that's what we want to do, then what are the tools that we can do in order to execute that in an efficient way? And AI certainly allows us to execute what we know we want to achieve in an efficient way, but the decision to do that and to deploy the resources that way is a human decision. So I think once we make that decision, and it could be that having that technology available helps us make that decision also, because the technology changes also our set of options. Yeah, well, speaking of decisions, I'd actually like to hear and maybe see a show of cards in the audience. Do you think that decisions ultimately should be made by humans? Would you be okay with an AI, for example, making a diagnosis, or should AIs merely provide suggestions for doctors and the doctor is ultimately who makes the decision based on the input? Green would be you'd be okay with AIs making such decisions. Red would be I would prefer to have a doctor at hand that supervises the process. Now, very interesting, with some of your mind, I see very few green cards and mostly red cards. With some of your mind, raise your card, your red card again. Wait, sorry. Green cards were the very few. Raise your green card if you're okay with, if you want to say your opinion about why you'd be okay with that. Would that be okay? Yes, you can. Okay, so if I understand this correctly, if the AI reaches a certain point where it reaches that kind of accuracy that is acceptably good, then it is the expert of that field and the human has no role in it. What would you, would the speakers have a remark on that? Would you still say even if we can prove that an AI is more accurate in such diagnosis, would you still want the decision to be made by an expert? I do have a remark, I think again, I agree that it really depends on the domain and on the task and how the system has been evaluated. So there are certain domains such as image processing in which we now have automatic systems that are better at detecting certain types of tumors by scales of, by degrees of magnitude than human doctors. And so if you test the system enough, then you are sure that it's not biased towards a certain type of ethnicity or age range or gender. I really don't see the reason not to trust the systems. However, if you just automatically just press a button and then send the patient to the operation room, that makes me also feel uneasy. So I think the healthcare in general and the decisions about the health are very complex. Just looking at certain factual evidences is not enough. Well, of course, part of becoming a doctor actually is training how to deal with the patient on a one-on-one. Yeah, briefly, Ed. I mean, yes, especially when it's about test results or image analysis, et cetera, then there are tools, right? There are toolbox. That's something totally different than a doctor that on the basis of all kinds of symptoms and test results based maybe on social understanding of the situation of a person, of a patient, think about stress or those kind of things, making a decision, that would be a totally different kind of thing. And secondly, there's also the principle argument that certain decisions about human beings should only be made by human beings. Like imprisonment, I'm not talking about traffic speeding tickets and stuff like that. I mean, you can automatize that maybe, but serious sentencing, where the social circumstances, whether someone has a job or not, or those kind of things, that should be done on principle by human beings because it's about other human beings. You can disagree, but at least that's an argument that we need to consider in the debate. Would you say even if you can demonstrate that, for example, let's say you have a judge that's an artificial intelligence, even if we can demonstrate that they make more accurate rulings than human judges, you'd still want a human to make such calls? But I think part of what's being said is that what is accuracy in this context? It's why the example of the evaluation exercise of is there a tumor here or not is a little bit easier to kind of figure out than most of what we encounter in social life in our society. And imprisonment is one of these things and social justice also as it relates to the penal system. So it's not just about sentencing, it's also about understanding the conditions that lead to certain types of crime and of outcomes. And then of course there's the need, we have a need to also want people to improve their lives. So we don't just send them to prison in order for their lives not to be improved, but there's varying degrees depending where. I've heard that's made to help people actually get their lives back together because they affect other people's lives. Those people who go to prison have families and they also affect lots of other people that are not imprisoned. And that in the end affects us all because we're all connected. Of course, let's take a couple of remarks from the audience. Mrs. up in the back there? Yeah. So if an AI would be able to start diagnosing things that they can't do today, so would this be like the diagnosis that humans can't do today? Okay. But the principle argument remains, right? I mean, it's not just, it's not all just efficiency. There's social factors, there's the psychological factors, there's empathy, you know, those kind of the idea that you see everything that is relevant to a problem by looking at the efficiency or speaking in the abstract of errors is simplifying the social reality too much, I would think. So even though there will be progress and again, especially in the more technical applications like detecting tumors, et cetera, I don't see any problem. But for a lot of the processes, there's an intrinsic social human aspect that you want to preserve. Sorry, just let me interrupt you because we do have a question far back. So if I could try to summarize your question, it would be that we hold AIs to the higher standard than we do for the human experts, but the AIs learn from the human experts and they don't make perfect decisions all of the time. May I take this one? Because this is exactly the follow-up that I wanted to make on both on Piman and Josh was, I fully agree with your comment. I think it's a very important point to raise that AI, I really don't like to call it anyway. Machine learning-based systems are trained on data that is collected from experts in the field. So they kind of reflect and in many cases, emphasize and exaggerate the mistakes made by humans. So in response to what Joshua said that these systems can actually help mitigating the bias, this is the danger that the system itself might actually contain a lot of bias that was imported from human judgment. On the other hand, data is eye-opening in a lot of cases. So even though decision-making in life-changing circumstances should be left to human experts, I think it is important to always have an eye on large-scale data. As an example, this recent meta study came out which made a lot of noise that there is this very clear systematic pattern that if male cardiologists treat female patients for heart problems, they under-diagnose and female patients die with a very noticeable margin of error. But this doesn't happen if the patient, if the doctor is female. So this is a very clear example of a data-based observation which then, it doesn't help if you have an expert which is a human bringing them in the loop and leaving all the decision-making power to them having a system that is forced to become less and less biased as a tool might help in these cases. So it's also about complementarity. Thank you. It's about complementarity. Very often we think about AI and humans in terms of replacement, which I think in many cases is not the ideal at all and should also not be the aim of AI. It's to complement human expertise with a different kind of expertise, like with the big data, and get the best of both worlds. And what we still have to do is find the proper balance between those two systems and to check how that then results in human accountability, for instance. We don't know exactly how to use the tools that we make appropriately. And that's, I think, another very big societal challenge. I'd like to take the question far left over there. You've been holding your hand for a while. No, over here first, sorry. Audio at the edge. Should I tell it? Yeah. As you can say, I hope you want to be judged what if there's only an AI in the country. So let me repeat that, Rook. Oh, actually, Dennis. Oh, yeah. So thank you for bringing that light. So the statement is that there are places that don't have the same resources that maybe some of the Western countries do and that AI is the only solution for them to even have a radiologist. Essentially, AI could fill in the shoes where humans are missing. In that situation, would it be more okay, for example, to have decisions made by AI systems? So maybe just from an economic perspective, if we think about how this comes into the world at the end of the day, these solutions will come through startup ventures, through large tech companies who are creating these services. And the way they go about creating these services could reflect the things that are being said here. That could be the type of, the way it's being done with humans involved. You can have that ethos built into the algorithm. At some point, I agree with you, at some point there's a good chance that those will be applied to other parts of the world or to even any part of the world at specific moments in time when it becomes necessary. And that will be more algorithmic than it is human. But if we can also work on other things like human capital development and economic development also around the world, then we would also hope that there would be more even distribution also of those capabilities across the world. So I kind of see it as a whole mix of things working together. And it's absolutely essential. There's joint action here between machines and humans. Any further remarks from the audience? Let's go with you, accountability. So the statement is that we need to keep the humans in the loop so we have somewhere to kind of place accountability when things go wrong. A sense of justice, yeah. Any remarks? Green card, yes? So translating to the human source of accountability is not going to be easy. You know, like the thousands of objects that consist of thousands of parts that were created all over. That's going to be really messy. But of course one of the nice things about AI is that it's actually quite good in tracking that. So what are the applications that we can build for AI to solve this problem that in part arises because of AI? We keep ourselves busy, you don't have to worry about it. But that I think would be really something to think about constructively especially in combination with legal experts. I think we'll take a question from the far back. Yeah, start with Dina, go on. Sorry if I understood this correctly. The question is that this co-responsibility between the AI and the human that the responsibility will be lost somewhere? So there's this idea of humans on the loop, right? So there's a machine that's making the recommendations and the human is on the loop supervising and intervening when things go wrong. Now if you do that after a while people start doing other things. It's very funny, it's like with a self-driving car where you're supposed to have your hands on the wheel and look at the road, yeah right. I mean and that's not because it's a bad person, it's because we're persons and we start doing other things. The more reliable the technology becomes the more we start doing other things. So they call that phenomenon being under the loop. You're supposed to be on the loop but you're actually under it. It's dangerous because that means that experts like in the medical field will become what they call moral crumple zones like with cars, you know, they take the hit. They carry the responsibility. But in a way the system is set up such that they cannot meaningfully be in human take that responsibility. And that's another issue that I see coming with decision support systems that we will trial human beings for not being on the loop but the whole situation that we created is psychologically unmanageable for them. So I see a big risk here for many professionals also in law for example. All right, I think we're going to take one maybe two more before we move on to the rapid-fire round so you've been holding it up for a while. We can't do that AI can do with pure computational power. Then we ask again, okay so if by pure computational power the AI makes a decision and no human could have ever made that decision because we lacked the ability to do so. We also have to think about, okay where is accountability now? Because it's technically something that we as humans don't even understand anymore. So I don't know, chess bots for example they work in ways that humans cannot compute anymore, that's what we lose against them. And then it's a question, okay who's accountable if an AI that goes beyond human ability makes a mistake because I don't know, I think in that moment you have to hold the creator accountable but it's also like can I do my best to try and summarize this? Good luck summarizing that. So the statement or the question we made is that when you have an AI of a computational power that makes it uncomprehensible for a human being to make any decisions about it would you put that accountability? It's incomprehensible. It's beyond our computational power but chess is very simple it's a very it's rule-based very simple set of rules very simple set of moves it's just that when you start looking at it playing over time it's sufficiently still computationally rich for most of us but we understand the moves that are being done we created the rules and even though we may not have made that same move because we hadn't been able to see that many more moves ahead I think we're not completely lost in that just to finish the point so even if we don't quite understand what it's seeing that many steps ahead we can allow it to do that because it really has very little significance in other contexts where it really matters what the outcomes are we need to be involved in that process and that's just my opinion about it but I don't see why we would want to relinquish that we should be involved in that process and understanding that using the computational power to help us see things we could not ordinarily see but then making decisions what we do with what we see I'd love to keep this discussion going but we do have to wrap up you can always find them after essentially I'll be seeing the statements and prepare your green and red cards because we're going to see how many people agree or disagree with the statements alright so let's just immediately kick it off with the statement ethics is programmable programmable you can program ethics into an AI alright followed AI can't be stupid it does what it's programmed to humans are stupid you think you can summarize your disagreement you can take a problem that's very well understood like chess and you can build two automatic chess players one of them is much more stupid than the other one of these gamers plays worse than the other even though both of them were based on the interviews with the same set of chess experts so relative to each other the technicality of the system matters we are talking about the systems as if they are these mysterious beings but we know actually how they were built and they could build they could be built better or worse so you can translate it to stupidity alright next statement we are glorifying AI today AI has a place in the military AI has a place in the military AI research should be supervised by an ethics board that is a lot of grain is there any red interesting AI will eventually control large portions of society so make political decisions for example in a way it already does with the media for example do we have a question do we have a question will AI never be capable of true creativity rapid fire frank we just have to kick on raise the headache card upscaling neural network and deep learning architectures is a solution to stupid AI so instead of trying to approach it in different ways just try what we are trying harder just like a car needs to have insurance AI should also be insured to account for its potential mistakes maybe in the future we will have insurance brokers for AI agents yeah it is already possible alright well that wraps up the rapid fire round I would like to have a brief afterward before we depart we are running out of time so I would like to thank all of you for the amazing discussion please everyone round of applause for our speakers who are accompanying us here today also I would like to ask for a round of applause for our enigma and Suleyman Ali to help make this event and our amazing host of course the unsung heroes so I hope you had a good time listening to and considering critical implications of employing AI and that you will continue to do so in the future especially throughout your studies thank you very much for attending before you leave we have a link and we know you all love scanning QR codes so we very much appreciate it if you could scan this it's an evaluation how much you enjoy the event or what we can improve in the future and feel free we hope to see you for drinks my colleague Hannah has tickets for one free drink so if you're joining us for drinks at the Espanade please come to her for your free drink ticket you were an amazing audience thank you for coming thank you for coming thank you