 Welcome everyone. Good evening. Hello, hello, hello. Thank you so much for coming. Welcome to the Royal Society, or if you've been here before, welcome back to the Royal Society. My name is Eureke Chowdry. I'm head of policy here for data and digital technologies, and I'm honoured to be your host for the evening. So unless you've been living under a rock for the past year, you will have noticed that AI has become a major political issue. Next week, the UK will host the first global AI safety summit to discuss some of the major challenges and opportunities AI offers the world. And today, the Royal Society has been hosting a series of events to help inform these discussions. In the morning, we held a joint session with the UK government to horizon scan AI safety risks across scientific disciplines, looking at everything from AI bias to AI peer review. This will both feed into the summit discussions and the report the society is publishing next year called Science in the Age of AI. And in the afternoon, building a FFAR 2022 report the online information environment, we held a red teaming exercise with the US non-profit organisation Humane Intelligence, bringing together postgraduate students to test the scientific disinformation guardrails of a large language model. And tonight, we will be exploring a range of potential risks and benefits that AI poses to society. But first, we need to make some housekeeping announcements. So if you go to any organisation in the country, you'll be very unlikely to find any of them that do fire drills on Wednesday night. So if you do hear a fire alarm, it probably is the real deal and at which point you should calmly follow the instructions of all society staff and exit the building. And if you need a bathroom, you really should have gone before the event. But if nature calls, they are outdone down the steps in the basement. And finally, if you're on social media, please make some noise about the event. We had someone write in criticising that we're promoting X, formally Twitter, and just want to make clear that if you would like to make noise on whatever platform you prefer, Mastadon, Facebook, Threads, LinkedIn, TikTok, Tinder, whatever you prefer. And if you want to submit a question, there are two ways to do so. You can submit a question at any time on Slido. You have to go on the, if you just Google Slido, you'll find it. And you have to enter the number A2510. And you can just submit a question at any point during the evening and you can upvote other people's questions that you would like to have heard in the Q&A. The second way you can do it is to wait till our Q&A session at the end where we will have some raving mics. So, I'm delighted to be joined on stage by a truly expert panel. We've got Anna Bakurelli, who's Technology and Human Rights Program Manager, Human Rights Watch. Ben Brooks, Head of Public Policy at Stability AI. Professor Gina Neff, Executive Director at the Mindiru Centre for Technology and Democracy. Louisa Bull, National Officer at Unite the Union. Sophanes Rashid Dia, Founder of Tech Global Institute, which focuses on issues in the global south. And she's come all the way from Bangladesh for this event. So, definitely put some questions towards her at the end, unless she's nervous about it. Finally, Professor John Crocroft, who's a Fellow of the Royal Society and a Professor of Communication Systems at the University of Cambridge. So, first question, let's talk about language. So, one of our reports, I'm going to mention a lot of the voices I reports this evening, by the way, and so, take the hint and read them. One of our reports in 2018 was on AI narratives and it spoke within that report about the effect that language can have on contributing to consequences for AI research, funding and regulation. And there's been lots of phrases recently. AI, large language models, generative AI, foundation models, frontier AI models. So, the first question is what do these terms mean to you? And Ben, let's start with you because I know before the event this is something you actually raised and I thought it was a good point. So, it's stable diffusion, the platform run by stability AI. Is that a frontier AI model? Are these words that you use in stability AI? What are your thoughts on this question? We would like to think it is. But it is an image model, right? And I think if you look at the way frontier risk and frontier AI is described, it's usually in the language model context. And it's usually used in a way that implies a very small number of yet to be developed models. Models that can do, have capabilities that introduce kind of catastrophic risks. It's not a term that we use. It's not a term that guides how we think about our language model or image model or audio model development. And I think it's a challenging way to frame the public conversation about AI risk. Challenging for a few reasons. I think one of those reasons is it implies that these are new problems, frontier risk, frontier regulation and new problems. They're not. I mean fundamentally what we're dealing with are questions of transparency, reliability, predictability, accountability that long predate 2023 and the generative AI cycle. I think the other reason it's challenging is because it focuses on this kind of small basket of speculative risks and ignores this huge number of more immediate, more everyday risks that confronts us today as developers, as deployers of AI, as users of AI. And I think certainly this summer in its focus on catastrophic risk, I think there's a potential risk that we omit some of these other problems as well. And we can perhaps talk about this more tonight, but I think we would like to see a sustained commitment from government and from industry to addressing all of those risks. Information, disinformation, fraud, abusive content, product safety, not just these speculative existential risk kind of problems which have dominated the headlines for the past six to nine months. John, you're a computer scientist. Do these words mean anything to you? Are these words that you use in your own research? What are your thoughts? Not really. No, I think I completely concur with everything you just said. That was very, very succinct. The risks have always been there, and I'm going to cite Reinhardt Rogoff. Anyone know Reinhardt Rogoff? That was a spreadsheet error and it predicted what you should do when there's an economic downturn and said you should do austerity and then a grad student noticed there was an exactly wrong error in the spreadsheet too late for many countries that implemented austerity. So the output of the spreadsheet fed to a decision, so it was a decision support system that helped governments make a decision to implement austerity rather than borrow a lot of money and spend it on infrastructure. You did not need an AI. It was a spreadsheet. It literally excelled. I mean that's like not even complicated statistics. And you can go and see the error. It's like public. So I think the question there is do you verify something which has led to important decisions? Do you have transparency agency? All those things. I think you just listed all of them. So I'm not going to go through them all again. But in terms of for me the challenge with the new hype, which I personally think is rather strange that a bunch of governments have jumped into doing a summit about it, we've had these issues for at least five years with the pervasiveness of AI being used in lots of ways, whether it is a spreadsheet or more intelligent ways of doing sentencing, all the things we know that there are horror stories about, but also some good things in medical diagnosis and so on. All those things have been around. This is masking that, but it's also masking in a bad way, which is there are two aspects of the current hype technology, which are not true of most of the previous technologies we've had. For example, even relatively large convolutional neural nets are explainable. DeepMind did a really nice project with Morpheus Eye Hospital doing retina eye scans and doing diagnosis, a feature literally classifying what was wrong with the person based on 50,000 images. They had a really good precision and recall, but they could also explain what the diagnosis was, which cases in the training data led to that. Many of the systems we've had for since Jeff Hinton started on this 30 years ago or more, and other systems, Bayesian modelling, all kinds of other systems are open to explainability, interpretability, quantifying their uncertainty, all these things that are really essential if you put them in any place where they lead to important decisions or they even more crucially actuate something in the physical world potentially, where they move something that opens floodgates or turns left over a cliff or whatever it happens to be. I think this bunch of terminology is hiding behind some superficial technical things that can be done which look groovy. Actually, the image stuff is much sweeter. The stable diffusion is probably much more amenable to some kind of explanation of what happened in there, but some of the future super large training data set out, high dimensionality system, how are you going to make those interpretable and quantify uncertainty? Probably not doable in any sustainable way and it's not generally obvious what they're useful for. If we get away from all that noise and say what are the things we're using, then we find we can use a bunch of the techniques which would then make them far more attractive and avoid all these risks. If we can have the conversation about existential risk from AI, but I think this is mostly driven by misreading science fiction film scripts and not noticing that actually Arnie actually saved the humans and Skynet lost. It's really important to remember that. Most of the end is the AI loses, but let's not have that. We can talk about that in a bar later. Gina, you've been very involved in some of the pre-summit activities. How have you heard these words being used? What does it mean to you? I want to bring our attention to the word foundation model because when our language does matter and the choice of words matter and when we use the words foundation model we really are thinking about the platforms on which other kinds of goods and services are built. I think John's example of a spreadsheet error being used for 30 years to justify an economic policy is a signifier of the kinds of choices that get embedded into the things other people build upon. My team at the Menderoo Centre for Technology and Democracy along with the Bennett Institute of Public Policy at Cambridge and AI at CAM, which is a Cambridge-wide AI initiative, released a report last week about what the UK's policy and generative AI should be. We really drilled down on this idea of the foundation model because when you think about some of the questions and some of the risks that we're looking at, and I don't mean existential risks, I mean what's being built today is the infrastructure for companies to use tomorrow. Okay, well, what's that infrastructure? Who's accountable for that? And are we building new kinds of monopolies? Are we building new kinds of systems that simply will power the digital societies of the future? And if we're doing that, that's the foundation on which we will live our digital lives and other companies will, I'm not going to use the verb forced, but other companies will be guided into making their own decisions, then we are building the foundations of the digital future and that then brings up these questions about power. Those are questions that aren't on the table at the summit. Those are questions that are not the kinds of things that are motivating bringing people together, countries together in Letchley Park next week. But it is something that I think is very interesting to people in the room. It's very interesting to the audiences I talk to. So it's this kind of notion that there are questions of how we are remaking our economies, how we are remaking our societies. Those questions are not the questions we're talking about, but it's the reason why the public is so interested in what we're looking at. So it's not maybe the bad science fiction, right? It's the science fiction where there's one or two or five powerful companies who dictate the terms of the economy going forward. Let's go to Dia. So Dia, you're here representing the 4 billion people on the global south. A statistic I got from chat GPT, so you might want to verify it on google.com. In the global south, are we having similar conversations like this? Is this a concern amongst the public in the same way it's been in the UK and the US and Europe? And do they use these kinds of terms like foundation models? Is that translated into other parts of the world? Yeah, I mean, I think AI has become a concern for governments around the world. We work extensively with governments in Asia, sub-Saharan Africa and America, and across the board AI has come up in different ways. But I think the challenges and what we're seeing also in the UK is that these terms are being interchangeably, right? So using foundation models versus generative AI versus frontier AI, it's almost being used as the same thing in many cases without clear definitions. And I think that's doing a disservice to what you're actually trying to regulate. Are you trying to regulate foundation models? Are you trying to regulate generative AI? Like I think there's a... And it's really important for different kinds of policy makers across the board to align on these definitions, because otherwise it gets really hard wanting to define what you're trying to regulate, second to actually have that regulate be interoperable, because the kinds that you're trying to regulate are global in nature, they're across national borders. So you can't have different definitions and therefore different kinds of provisions, and then finally regulation that are impossible to actually implement. And that's kind of the challenge that we're seeing in different parts of the world. I think the other aspect of that is, you know, because these terms are being used interchangeably and there is this almost like a gold rush to regulate. So at different points without really having clear alignment around definitions, around the risks, around the harms, everybody, every government around the world is putting together a proposal on the table. And these proposals are not talking to each other. And so you are in a situation where, you know, when we are going and speaking with somebody in Brazil versus somebody in India, they have the same basic idea of what the risks are, but they're doing it in very different ways and therefore if you are a company trying to comply in two different geographies, you just can't, because you're not sort of agreeing on the same premise and therefore it's quite impossible for you to figure out what to do. And I think that does a huge disservice to the public because a government's job is to govern systems to make it expandable, reliable, trustworthy, but when you can't agree on the basics, then you're essentially giving the, you're basically not doing your duty to the public. So Louisa, in their trade union movement, so often this is talked about in terms of automation, has there been any sort of change in thinking over the past year with chat GPT and all the other conversations around AI? Yeah, I mean, I guess what everyone's been saying, the workers' perspective on that to me seems to be fundamentally the most important. What does it mean to us as workers? What does it mean to us as citizens? And what it's called doesn't particularly bother me because I'm not going to understand it anyway. I need to understand what is that going to do to somebody in the workplace? Or is that what's that going to mean to me on my mobile phone? How is that invading my life and my members' lives? And what is owned? Automation, innovation is always here. It's been here forever. I worked in printing 30 years ago and innovation came and went and moved on. So trade unionists are not frightened of innovation, but they want collaboration and transparency, safeguards around that. You know, we're not even invited to the summit. Civil Society and trade unions have not got a voice in that room, and that is to me a big mistake. But, you know, we will work with employers, we will work with industry and we will work with everybody that we need to to ensure that our agenda is in that room or on the table. But I accept what you're saying. It's got to be global. We're a global federation. We talk at European level. I meet my colleagues at European level to discuss this. We have a global agenda for workers, but we don't seem to have a global agenda across business and governments, and that's what's what's needed. And I've left it with a very difficult task here, which is to add anything on what's been said or think about, you know, has this changed the sort of language or thinking around human rights issues in AI? Has there anything particularly novel that's come out in the past year that's changed your your thinking about human rights watch? Yeah, absolutely. I mean, on the language question, I think, you know, consistency is key, just so that, you know, we're debating the same terms. I think getting into the specifics is actually a welcome change from talking about AI as an umbrella term, because we can actually dig into the nuts and bolts of what does this do. But the one thing we need to kind of, I think, look towards more is context. So, you know, how are these models applied and, you know, developed in terms of like products or, you know, how do they land in people's everyday lives? Without context, I think it's very, very difficult to talk about regulation that is, you know, that matters to people, that is accountable to people. And yeah, on the human rights side, I mean, we've been documenting the impact of AI on, you know, various sectors and societies for the last, I don't know, eight years or so. I think in terms of a change in the past year, what we're seeing with particularly generative AI is, I think, I mean, on our side, we have a sort of like verification and kind of documentation of human rights abuses that is really suffering with generative AI, just in terms of sort of gathering evidence and kind of documentation of abuses. Obviously, the crisis in Israel and Gaza is, you know, the most kind of recent example of that. And I think in future, that's, you know, really something that we're thinking about in the human rights movement is, you know, how do we preserve and verify information. Great. Okay. So, next question. So, the more society today has been focusing a little bit on disinformation because of our Twin Cities to report the online information environment. Don't forget to read it. Because that's far as that's an area of research that most closely fits into the AI safety theme. So, other than disinformation, what do you consider to be the key AI safety risks in these shorts to medium term? And let's start with you, John. In terms of generating disinformation. No, sorry. Everything other than disinformation. Oh. Yes. Trying to make you think here. Trying to make you think hard. The other problems with generative AI. Or just AI, as you see, and what do you consider to be AI safety risks from your perspective? I guess that's the question. I think there's one of the problems is the public understanding of what's happening. And so what's beneath the hood is getting more complex. It's always been complex. I mean, you could have a PhD in microelectronics and know how your mobile phone works, right? So it's like, you know, that's pretty astonishing. And actually, I think one of the first things that's real AI was Google Maps, you know, where you have a GPS system, and you have a phone with a assisted GPS receiver, and you have this map data. It's astonishing the number of algorithms that are going into making that thing work. And I actually do know how most of that works, but it's, you know, it's most people probably don't. And that's not being patronised. And they've got more interesting things to do, like what's the next episode of Loki or whatever it happens to be. Sorry, I'm a science fiction freak. So I think, I think that for me, just sort of going back to answers, just going back to actually the previous speaker, a really nice use of LLMs. I came across with the UN was working on simplifying legal documents for refugees getting into Ethiopia, Yemen, Afghanistan, various countries. And one person with, I think it was a hugging face, a llama, Mr Al-Sevan, was running this and it took in multiple languages with the Mozilla Open Voice Protocol, was able to have people speak the facts. It would then fill in the documents, speak the whole document back to a human scene there was a lawyer would check it. And to the person in the simplified form that they would check in Swahili or Ethiopia or Arabic or whatever. That's a really cool use of AI. And notice what I said there though, that there were two humans, there was an expert and the subject, the user. And this is very attractive use of that technology. But the bit that's important there was that there were two humans. It was intensely oriented around the user, it was centred around them including voice interface because they might not be able to read and write. And having another person who would assist with checking, so the verification not requiring to something which at the moment is very hard to do with systems I already talked about systems like that. In other worlds where we connect large complex AI systems to physical systems, cyber physical systems been around for a long time, things like autopilots have done this for a long time. We'll notice that Boeing had a major incident when they didn't include users in the redesign of an autopilot and they changed the user interface without retraining people. But actually autopilots are generally a piece of software super safe because they're verified but they're also put into wrappers and we do this with trading software and stock markets and you can even have trusted third parties who will run a number of algorithms in a secure environment and check they don't interact in bad ways. This is also done by medical agencies in France, they did this with drugs and side effects and they ran a whole bunch of experiments with virtual versions of everything to see what interactions were. So these are all things that we should be doing more of because they are doable, they don't cost a lot more, they get rid of all the problems of later being found liable, they give you mechanisms for redress when you see something does go wrong, they do all those things. So for me that's there's a whole bunch of sort of feed in some positive story here. I think there's a lot of good things we could do, there are some good things being done and they usually are good because they include these factors which when people ignore them then it's at their peril and I'm going to do a Neil Lawrence here, he used a 3D printed widget in House Lords but I'm going to quote Shakespeare, he said, there's no art to find the mind's construction in a face. So he's talking about humans and misrepresenting things but when you use an AI you have no side channel information about what's happening, you have no back channels information to who created the AI, what they used to train it, all those things, you can create all those things. So you could put a face on the AI and in my example with the UN refugee problem there are actually multiple faces, voices put on this system which was very very cool because then you got to get two people look at each other and a person getting help from a lawyer was like wow that's really useful, that just did the thing, right? So I think there are ways forward which involve more clever thinking about technology for researchers like me, that's great because we'll have a job for life, right? But we'll also be able to say it was useful too. Anna, so do you have any thoughts on that positive uses in your experience and again also your views on what you consider to be the key AI safety risks? Yeah, no I'm not anti-tech. The positive uses are absolutely fantastic, you know we've used machine learning at Human Rights Watch, previously I was at Amnesty, we used machine learning there, you know both kind of like sort of image like satellite sort of imagery scraping as well as kind of LLMs and some of our you know kind of background kind of data gathering, absolutely not anti-tech. I think you know we just want safeguards in place and obviously when it's used in particularly high stakes scenarios then that's what we really care about and at Human Rights Watch we've been documenting the impact of the welfare state for some time and automation in the welfare state and so a couple of case studies I can cite, one is universal credit here in the UK and second is TACAful in Jordan and in both cases you have an algorithmic system that is gathering lots of data points in the case of TACAful it was like 57 different data points assessing benefits claimants for their eligibility and then you know making a decision and you know they don't know what that is and what criteria has gone into that and then you know either issuing benefits or not. In both cases we found a lot of similarities in terms of problems so just a kind of very crude reading of information that doesn't account for the complexity and kind of changing circumstances of someone's life in the case of universal credit kind of famously it didn't sort of anyone that was paid twice in a month they were counted as having a double payment so your universal credit was you know significantly decreased your payment account because it was sort of assessed on a monthly basis it doesn't matter whether people were sort of paid on an informal basis or ad hoc which is obviously kind of more likely to be a kind of lower social economic status but or if someone's payment had been brought forward from a month say because it was back holiday or there's weekend for example and so that just kind of crude assessment I know that it's it's been rectified in some cases and if you are monthly salaried then that is fine but it but still if you have multiple payments coming in and you are not on a monthly salary your UC will be docked or you know impacted based on that decision and just doesn't account for real life the same with Jordan you know most recently we found that people that owned a car for example were being penalised in this in this sort of collection of data so that didn't obviously sort of negatively impact to people in rural areas it didn't account for if you haven't used the car because you can't afford to run it you know if you need a car because you you know live miles from anywhere so on so just all of this kind of like crude data which is being fed into a system which impacts people's lives and they have no real sense of like how decisions are being made how to challenge them as well and also the kind of like the digital literacy that's involved in these systems and you know it really kind of disproportionately negatively impacts certain parts of the population in Jordan people needed to often pay for a bus fare to go to a shop and to then pay to use a municipal computer or a smartphone for example to even apply for the benefit in the first place so that's like a down payment they may not get it back so all of this kind of thing is like incredibly mundane it is not what is going to be discussed at the summit next week I imagine but this is the kind of nuts and bolts of life that impacts real people and this is how AI is being deployed whether it's across welfare whether it's across policing whether it's across you know asylum applications that is you know the impact on people's lives and it's happening now as well so you know I think I'm glad that we're talking about AI and accountability I mean it feels to me somewhat overdue and it's great that it's in the spotlight I just really want to kind of shift the spotlight to something that's a bit more realistic so a couple of themes coming out explainability accountability Gina you don't you do a lot of work on these areas do you have any thoughts on potential potential remedies to these questions and again also your views on what you consider to be the key AI safety risks people are the key AI safety risk let's just name it right let's name it what it is and I I can say that a little blibbly forgive me but I truly mean it right we can have a summit about the risks of models but we have to remember these models are people all the way down right they're built they're run they're profiting they're they're run on people right so when we talk about the risks of these systems you know one of the pain points one of the weak links in cyber security terms is always the human in the system so let me give a couple of examples our friends at data and society research institute along with Duke medical ran a series of studies about a new sepsis detection tool that was rolled out in a US research hospital and this was done by an internal innovation team this sepsis any of you know is a terrible condition it kills people in hospitals and it's treatable if caught early but the challenges are often really busy emergency rooms any's busy hospitals don't have the resources for that early detection so they rolled out machine learning tool to help identify and classify patients as at risk of sepsis but they didn't call it AI why because the nurses in charge would not understand what they needed to do with their own expertise if it were named AI they needed the nurses to still be there to advocate for each and every one of their patients and to have their own professional judgment in the system they were worried about nurses saying oh the AI has told us this patient's not at risk so now my professional judgment is not necessary that's a real safety risk in an everyday situation that we're seeing right now in ophthalmology in retinal scanning that's happening in this country anyone who gets their wonderful eyes scanned at wonderful boots yay boots opticians um you know that we have a wealth of retinal image data in this country that it can be used to help early detection of a whole host of diseases right but we are at risk of diminishing the professional expertise and training of systems let me give you another example robotic surgery great fabulous advances robotic surgery you know we can think of surgery by robots but really think about wonderful kind of more precise tools that experienced surgeons use right great wonderful how do they train those tools well they train them on the simple cases right they train them on the straightforward kinds of procedures and processes that junior doctors used to do junior surgeons used to do and in a paper by our colleagues in Cornell they found that those junior surgeons are having to fight for the training opportunities in hospitals they're like wait a second guys that easy stuff that you've turned over to training the robotic surgery arm um we used to do that now where are we going to get our training and expertise and so the organizations that i talk to from public sector organizations to government regulators to cell phone users that's what happens when you give a computer scientist a phone this is what i'm saying people are the weak link okay you know so from all of these organizations they're really struggling about how they build their capacity right now in workshops we have with regulators regulators don't quite know how we're going to roll out the uk's own approach proportionate sensible yes sectoral yes by each regulator yes probably the best way go team uk probably the best way to regulate a ai they don't know how they're going to step fit so we've got a lot of work to do to building up the social human capacity it's both people understanding what's going on underneath the hood yes but frankly i don't understand how my car works but i know i'm licensed to drive i know what guard rails are and i know when that check engine light goes on what i am supposed to do and right now the check engine lights for some of our complicated models are not apparent and transparent to those of us who are going to be making our professional decisions based on their output so we've got a lot of work to do to build up that capacity and frankly that's the conversation that i would love to see taken up at the summit right where are we having the conversation about building out that capacity so we can as societies be safer the check engine light on my car is permanently on that's a good segue to louisa so um you know another theme here is training and people how are you thinking about this at unite and i know you're also involved in the trade union congress work in ai can you talk a little bit about that and then again also your view on what you consider to be the key ai safety risks yes so the um the tuc have obviously been working on this for a few years and about their own ai task force led by great one mary towers but we've had our manifesto and that talks about all the areas where we think these issues are going to impact workers in in the workplace and you just talked about some of the you know higher technical um skills and roles but you know i'm going to take you down to uh a guy or a girl working in a warehouse with a headset on picking and packing now that is all been automated for many years the decision on how quickly they need to pick where they need to pick and what they need to pick is all told to them on the on the phone they've got no power no control and they are automatically disciplined if they miss the target doesn't matter what size shape health they've got if they've got to pick up a box of coca cola or a box of paper everyone has to pick it up at the same time from the same spot and get it back to the lorry by the same moment and the fact that they've not got any power within that and no ability to change that is common place in in work and that is having a real impact on people's health and mental well well-being right so that is a very simplistic example of where the trade union movement have got our manifesto what we believe we need to do in collaboration with our employers bringing in these systems is great they're helpful they're useful but we need to know that there's a human in control and also our worker can go to a human to appeal that process if they need to and that's not happening right now the systems are coming in under the ground we're not consulted we've got no right to ask we've got no information and consultation rules around that and that needs to change you have the right in this country to bargain you have the right to bargain on pay hours and holiday and that's what the trade union movement do but we need to be able to bargain on these systems on data and we need to know what's going into a company how it impacts us as workers and we need the right to ask the right to know and the right to agree to that before it happens so ben um in some people's eyes a i safety risks are stuff that's happening on platforms like stable diffusion sort of harmful content um is that is that what kind of thinking goes on with instability when it comes to addressing some of those harms what are your views on it and then also what do you consider to be the the key ai safety risks well i think both parts of that question are the same answer which is um there are many kinds of misuse that industry government the public are having to grapple with that don't fall into this category of existential frontier risk um there's a lot we can do layers of mitigation that we can put in place for that misuse as a model developer as an application deployer as users um as as the rest of the information ecosystem the social media platforms there's a lot that can be done there to prevent the misuse of these systems for abusive misleading fraudulent harmful purposes but there isn't a silver bullet and i think sometimes there's a tendency particularly this year to look at ai as if it's this big kind of monolithic technology and and and and you miss um you obscure the complex supply chain that sits behind that and if we're going to seriously mitigate those kind of risks that kind of misuse we need accountability and we need regulation and we need mitigation right across that supply chain it's not just going to happen in the models it's not just going to happen in the compute it's not just going to be laws governing users it's going to be some combination of all of these things um so that's on the misuse piece and and we do a lot and I can talk to that as well in terms of dealing with some very immediate risks that that we see I think there's another risk um or a kind of short term medium term harm which is product safety you know when someone builds the web md chatbot or when a consumer bank starts using language models to make investment decisions with deposits what safe and fair means in those contexts will be very different we need regulators who are well resourced and have a mandate to start developing performance requirements for those use cases in those domains and it's it's not as sexy as existential risk and frontier risk and all the rest of it but very very quickly we need to have in place guardrails for what product safety means in these very very different very diverse environments the final piece I'll add because it doesn't get any airtime um is the risk of of a collapse of competition in this space and Gina actually made this point earlier you know AI as a technology has the potential to centralize economic value creation in a way that almost no other technology has of the past 20 years and we have lived for 20 25 years through a digital economy that has one search engine two or three social media platforms three or four cloud compute providers and there's a serious risk that we're going to repeat some of these mistakes again in AI unless we think about competition as a policy priority and what does that mean in practice well it means for one thing that AI won't just be two or three bay area labs producing models that the rest of the economy depends on for the next 10 or 20 years it's going to be open source you will have open source capabilities open source technologies that any public sector agency any small business any research for a university can take from github they can adapt it they can customize it they can make it better they can make it safer and they can figure out how to deploy it safely in a real world environment that's the future that we want to see and so we release our models openly our language models have been downloaded over seven or eight million times since I think a may of this year these aren't users these are developers who are trying to figure out what to do with it and they're trying to build capabilities in a way that doesn't depend on a third party big tech company so I think I think that's a serious risk as well and when it comes to when we you know frontier risk existential risk in that kind of conversation there's a risk that we fear manga railway into a kind of statutory duopoly where only two or three companies have the license in some cases a statutory license to build this technology and that is just as scary to me as a world of skynet and terminator so I think I think we need to think very carefully about if this frontier risk existential risk conversation leads to legislative and regulatory action how do we make sure that that supports a diverse thriving AI ecosystem with corporate labs as well as grassroots developers close source technology open source technology big models small models and everything in between and move away from this worldview that has kind of taken hold over the past six months of AI being two or three big labs threatening to destroy the world and asking for kind of statutory licences and other things to to prevent harm so I think I think that's another risk as well that doesn't get a lot of their time there's an interesting report I think by the internet watch foundation today which is talking about AI generated pedophilia content of celebrities made me think about novel risks have you come across any sort of novel human rights risks that have come out come out in the past year and again also what do you consider to be the key AI safety risks I mean I think in terms of novel risks we've probably talked about gender to the eye if it's the past year and I would say the big worry on our mind and our sort of within human rights watch and I think also from speaking to you know focus and tech companies is what happens around the elections and there's a big bunch of them looming next year so I think you know that's and then you can go into what is the novel risk there is it a chat bot pretending to be a representative and you could look at different applications I mean I think to understand the human rights impact of something you've really got to look at a use case and so for me you know that the foundation and sort of a model layer is is all well and good but like what happens when that's applied in practice and all AI you know ends up in a product of some kind and so that's the stuff that we're kind of really interested in is like you know yes is the technical safety components but you know I'm sitting here with a human rights hat on so it's like how how is that going to impact potentially my access to rights and then also you know we mentioned supply chain what is the what's going into that and the creation of said product and you know particularly this year I think there's been a welcome focus on the the employment rights of like data labors and you know everyone kind of in that sort of life cycle that was I think formally sort of confined to the sidelines they weren't thought of as tech workers you know there's a there's a real difference between someone in say Nairobi like labelling data or like moderating like the worst parts of the internet you know kind of taken down graphic content or whatever and say you know white collar tech workers in Silicon Valley's but I think now we are thinking of that a little bit more as a supply chain so I'd really encourage us to also think of the kind of like employment rights at the very sort of start of that as well sorry what was there was that yeah that was a good comment so um so during this morning's session on horizon scanning we looked at um AI being trained on weird populations was my phrase of the week which is western educated industrialized something rich rich and democratized you know populations and um you know there's a lot of interesting conversations there about what that means for the global south what it means for um different populations if these programs are rolled out globally and also just touching on on that point on data labelling do you have any thoughts on those issues do you because those have been in the news a lot especially when they're the labelling of this content and also your views on what you think the the key AI safety risks are yeah um I think um and I you can raise really good point around and it goes back to your earlier question on definitions right I think AI as a system has existed for quite a long time so I think we need to really start by thinking about automated system I think that's the starting point of the conversation when we go into generative AI frontier models you kind of lose the crux of the problem and then you're not really working around solving that sort of issue so if you really take automated systems as the crux of it I mean the global south has been disproportionately impacted by it for a very long time the Jordan example is a really good one but in the social credit scoring employment choices um potentially how they're being represented in other kinds of media I mean the global south has historically been excluded and therefore what you end up with through automated systems is a very reductions to view of the global south and you know it was a joke at the beginning that I'm representing four billion people but essentially that's what it is you have one image that represents everybody in the world right and I think there was a really interesting report that was done by rest of world recently that looked at um image generation not stable diffusion others as well but in general all kind of image genera- sort of image generative AI systems and it was trying to put questions like you know average Nigerian man um average Indian man and like it really kind of produced images and content and um responses that really boiled down to the worst stereotypes about those populations right and that's really really harmful because in the long run you know and it's the same as sometimes I'm like the google search like when you're googling when you're searching something on google which is also not an automated system you're essentially only finding one version of the story because that's coming up on page one and page two you know and you're missing out on the entire sort of whole range of other kinds of information so with systems that are that rapid and trained based on on biased data you're essentially getting responses and images that again goes back into the worst stereotypes only a singular you know narrative about the entire global south which is really harmful but I think ben can have touched on something in terms of the AI safety risk which is really kind of taking it to the supply chain right and really recognizing that models don't exist in vacuum they exist because people design them and make decisions behind them there are people who are deciding what data goes in what data goes out how you're going to train a model how you're going to design it how you're going to deploy it how you're going to govern it those are human made decisions so I think the biggest AI safety risk in many ways are the people behind it because they're not they are making a deliberate choice to exclude certain parts of the world because it's it's you know it's more profitable to do it or because you know it's just an easier thing to do and then you're releasing models into the world that gives really factually wrong and historically inaccurate information so I think really recognizing that entire spectrum is really important and really going to the going down to supply chain it's not just the application layer but also like the data layer the supply chains um understanding you know how the labor practices are being impacted because of data labeling content moderation how you're sourcing the data once you have the data then how you're actually designing the program and I think just having human rights due diligence across the entire supply chain and the and the complexities around it is really critical because when there's so much focus at the application layer you're kind of not really solving the problem at all it's going to be very superficial it's going to be not sustainable and that's going to be a challenge and I think the other sort of I mean in terms of the systemic um sort of the safety risk is a bit around you know how it's being regulated because again because that commission is entirely being dominated by a couple of companies um in the west uh predominantly in the US and because the legislative environments that they exist in are are more mature you're essentially extrapolating the same legislative instruments or legal instruments across the world and so you have a situation where you have patents for example they're exclusive in the US and then if anybody's trying to develop models on the basis of it they can because it's a patent violation right or you have a situation where you're bring where you're kind of copy pasting legal instruments to regulate at the model level and then you have a situation where local companies in Nigeria or Kenya or Bangladesh can't even um comply with those requirements and therefore they're being killed and so you're actually killing competition in that way so I think this this idea that you have this very sanitised um you know one country view of how AI is being designed, deployed, what you dare you're collecting is essentially creating a lot of ripple effects across the world which is essentially excluding an entire um population and I would say the majority of the population because that's like what four billion people out of seven billion people in the world so that's that's really problematic just a reminder that four billion is from chat gpt so please yes there's a really important point in there around we've become accustomed to thinking of trust and safety and tech as being something that you you hire up two or four hundred trust and safety people in your big tech company and then that's how you deal with trust and safety. The reality with AI and a whole bunch of other algorithmic technology is it's going to be deployed by tens or hundreds of thousands of businesses and organisations around the globe and we need all of them to be resourced with the standards, the research, the tooling to implement this stuff safely so what we really don't want to see coming out of this summit or the task force or any of these other initiatives is a world in which trust and safety is kind of exclusive to upstream companies making the models we want to see resourcing so that everyone across that supply chain and everyone across that ecosystem can implement this stuff safely and we haven't really seen a commitment from any of the large jurisdictions to help make that happen and that's going to be critical if we want to see AI deployed safely not just in the san francisco bay area but to see it safely deployed across the global south across small businesses here at home. Yes but it's risky for small medium enterprises for startups to build on technologies that they don't have clear legal regulatory IP oversight in terms of what's going on so we call for in our report on generative AI for UK policy UK policy on generative AI sorry uh you know we call for getting that clarity in the UK jurisdiction around ensuring that companies know what is coming for them so for example let's imagine there is a model that there may be challenges on the intellectual property issues of the inputs that were used to build that if I'm a company and I'm using that to build my products and services what's my legal liability right now we don't know the answer to that we're not going to unlock that innovation in this country that is possible we're not going to see the kinds of growth in small startup companies that we do so well in the UK based on these foundation models until we get some clarity around what that legal regulatory environment looks like on the underlying foundational technologies that they're that they're relying on. It can't just be their responsibility. A couple of things in this by-chain you're also actually had an event not that long ago with a number of companies and the BBC talking about provenance of data and particularly in the world of generative AI. There's an excellent report on that. Excellent report but unfortunately unfortunately it was exactly in this world where the usual suspects were the companies who were going to build the technology for the provenance and track it and do the audit trailing. On the other hand the ICO a few years ago wrote a fantastic report about data trusts which actually talked all about what might happen with ownership and a good example in the UK would be NHS patient record data which should really be a trust which we are all shareholders all subjects and we own and when Google DeepMind builds an AI based on it given the software is about 0.01% of the effort and the data and particularly labelled data labelled by surgeons and experts is the high value then at UK NHS PLC should own 99.999% of that application and make lots of money when we sell it to US healthcare private companies or whoever and that's that the inequity of what's happening both I've just given two examples of just that that's just not what's happening which is terrible and any market with an invisible hand would not arrive at that point right so this is if you want the regulator of choice who should be at the summit next week it will be market regulation people I think several people here have alluded to that being an aspect and civil society and unions as representing work courses and exactly the same argument applies I think that's already happening to the UK's largest retinal image database so millions of retinal scans now sit in a in a public trust and and the first licenses are being negotiated and it is the community oversight board who are helping to evaluate what those proposals are and who are interested in understanding retinal scans it'd be great if maybe a pharmaceutical company that's building new solutions for diabetes had access to millions of retinal scans in a country to be able to to train their models they're the ones who are going to pay not million not 10 but maybe potentially 100 million or more to be able to access that data so you know we have the models we have the models in place and we have them here right now I was going to say I think I mean yes and I think I'm like an improv session right now but anyway yes and on the legal instrument piece I think a big question to ask is who's defined those legal instruments right because if you are you know in the last 10 years at least if you take the experience of social media and like use that to kind of understand what could possibly happen in the AI space as well in social media for the last 10 years IP has been regulated by the DMCA which is an American legislation the problem with the DMC is is that because it was lobbied by US tech companies largely and sort of the US broadcasting companies it predominantly is skewed against people from other parts of the world so the DMCA gets weaponized by authority and governments by you know using sort of wrongful IPs and then taking down political content their DMC has been weaponized like you know use things like for example google's right to be forgotten so there's a there's a bunch of implications of how legal instruments designed in one jurisdiction has negative ripple effects in other parts of the world so it's really important and I think coming back to the question on definitions it's really important to really lay out those definitions up front and have really global cooperation around those because otherwise I think what what at least our fear is from a global stock perspective is you know you have a few legal instruments from a few sort of countries and then these companies which are predominantly based in this kind is trying to kind of roll out global products using that same legal instrument everywhere and then essentially you have people in other part in these markets who we can't really actually enforce on those legal instruments or there's or they're basically disadvantaged against it so like that's a model and the and so the other other flip side of that is if you're saying that okay fine these models exist you know if you if we're trying to kind of take something like a GDPR and then implement implemented in a different market for example the markets infrastructure is not designed for those kinds of legal instruments either so you don't have the maturity you don't have the infrastructure and therefore even if you were trying to create your own legal instruments the question becomes sort of is this contextual is this relevant because all of the ideas all of the narratives all of the academics are sitting in a different part of the world and so essentially even if you were to take something like a GDPR in a different part of the world you don't have the maturity within that ecosystem to actually implement the GDPR and then you have a situation where the few handful of companies that have the resources to actually implement something as complex as a GDPR they survive and you push out the local competition and they they are killed because they are just non-compliant so you have you have an instance where either you have one legal instrument around the world implemented unilaterally or you have a copy paste mechanism where again it's defined in a certain part of the world and then having doing a disservice to other parts of the world which I think is a problem as you're thinking about the legal environment and can I speak up briefly it's also about enforcement as well because you know so many countries do not have a data protection authority and you know when you're talking about the absolute basics here I think you know it's just if something is going to be sort of applied globally or you know certainly beyond one jurisdiction then we really have to think about the nuts and bolts behind that yeah I'm going to move us on to a slightly different question so something I didn't mention at the beginning is the name of this event science times science times air safety expecting the unexpected and part of the reason for the name is thinking about how AI might evolve in future and in particular thinking about where risk might lead to in future the sort of unknown unknowns and unlike other technologies this is a very general purpose and therefore there could be all sorts of ways that this leads to complicated new risks so this next question is about how we think about risk and how we predict risks in future because this will be the key question I think that comes after the after the summit and I want to start with with you Louisa because the main thing that most people probably care about is their jobs and for you it'd be interesting to know how you think about this at night like you know what training is needed what jobs are going to be automated how can you predict any of that you know how do you go about that sort of risk assessment and in the trade union movement so I mean we've obviously got the data impact assessment but it's not a bigotry on employers to complete it and so we are struggling without any regulation at the moment to even be able to find out from employers what in fact they are using and most of them don't actually know what sits behind the system that they're using anyway so for instance if you if you apply for a job then that automatic automated process is going to weed out the first level of applicants now we don't know how they do that we don't know what unconscious bias is in that system like you said we don't know who wrote that system there's no appeal to that system so that's the kind of area where we need to have some transparency and some dialogue with with the employers about what they are putting into their system I mean most people if you work for an employer they will have put their HR systems onto your phone well they own your data or somewhere someone owns your data it might be that they've sold it to a third party but you didn't have a choice you did maybe not even add a work phone it might be your private phone and it's all done in the name of you know ease and access for you to book your holidays when you're at home on a Saturday night watching tv or check your pay slip online all of that system is what is really what we're talking about at that workplace level and we have no conversation or right to a conversation about that right now so part of what the tea you see in our manifesto is about people power and technology so that we know what is going in what it's being used for and the right to say no as to who owns our data it is our data we are not an AI lab walking around with a mobile phone sharing everything we put on it with the world we don't mean to do that you know but we are doing that and we're actually doing it at often at the request of our employer and our personal phone and our work phone are mixing up and so that whole kind of you know description of us as being collected that profile of each and every one of us as a citizen and as a worker is just is just sitting out there and we're actually being encouraged to do that and in many cases obliged to do that by our employer and when you ask the question they don't know because they actually don't know and we have no legal obligation to inform our members of what it is because we have no right to get that information from the employer and that's the level of conversation and debate within the trade union movement about how we regulate now what we do about that people around this room have all got different reasons but from a workers perspective we have the right to ask and to know and receive and say no to that issues of our data instability AI what is the risk function what does it look like is there a risk register you know what's on that risk register how far into the future does it go can you tell us a little bit about how you're thinking about these sort of long-term risks and in your company the focus as a model developer for us is really around performance and behavioural risk in the model again because it's a model that can be used in so many different scenarios so many different use cases and it's open source so it can be used by so many different people we um we want to understand the kind of breadth of capabilities in the model the potential for misuse behaviours that are undesirable or unsafe or unlawful and then as far as possible in the part of the tech stack that we own the model and in some cases the API and the application we want to then put in place mitigations so for us you know we do have a robust kind of pre-release process particularly with powerful models like our language and image models where we're thinking through what is the data that is going into the training we filter it to take our unsafe content which should make it harder for the model to produce unsafe content we fine tune the model for certain kinds of behavioural bias and certain kinds of undesirable behaviour and then we evaluate it you know and a good example of what external evaluation might look like going forward was DEFCON earlier this year and we were one of I think seven or eight companies that fielded a model we had about 2,000 researchers come in and essentially attack the model to try to elicit undesirable or unlawful responses and and that's a great example of how you can use external independent evaluation to test the kind of full envelope of the model's capabilities that's one piece of it the other piece is internal red teaming where we our research has probed the model themselves and then there's standardized evaluation and I think this this is one area that is both underdeveloped but going to be very important going forward for existential risks as well as for kind of behind the frontier risks you know we have benchmarks today that look at you know take a language model for example look at comprehension and reasoning and fluency and and other things like that they're fairly crude benchmarks in many ways you know a lot of work's gone into them it's it's really useful as a kind of first assessment of the model but as we start trying to probe for these more novel or more exotic risks we need more specialised frameworks that really try to capture capture that and a kind of battery of tests that a developer can run the model through before they release it this will become very important when you look at when people think of a model developer they're typically thinking of a big corporate lab but what you're now seeing is you can take a base model you download it from github or hugging face and then thousands of people can now fine-tune the model in ways that are generally very good that generally makes it safer more performant but you can theoretically fine-tune those models for malicious purposes and so how can we put in place evaluation frameworks for those downstream developers and downstream application employers so that they understand the risk that they're seeing in their fine-tuned model or in their fine-tuned AI capability that's you know an area for further inquiry that's something that we're certainly supporting and and we're working with civil society and governments and a number of other organisations to think about what does that look like what does the future of evaluation look like for the kind of existential risks but also these more everyday kind of risk as well did you want to come in John yeah i'm just going to bring it back to science and safety so um e equals mc squared tells you about energy mass equivalence lets you understand how you might do something interesting lets you build nuclear weapons we had an organisation called pugwash which was a voluntary for nuclear physicists to not work on weapons dna structure dna quick and watson and another very important person anyway um um very good but you could do recombinant dna and you can do really bad things with recombinant dna experiments so we had asem on asem on over which is like a conference which like annually decides what we should not do so i think you know there is an existential risk which i could find hard to quantify but in the society had a report about this 10 years ago on governance of climate change and one of the areas they were interested in was geoengineering which is now coming around again yw geoengineering is coming around again is we're hitting 50 degrees wet bulb temperature in some parts of the world for more than a day where basically if this happens for a week you'll have 100 million people die and countries with launch capability will just stick sulfur in the upper atmosphere immediately because they can do that and that will drop temperature by one and a half degrees you can that's known science but it might also move them on soon 3000 kilometers away and six months away which will cause a load of people to starve and no food so we need to quantify the risk of geoengineering but it's so complex because it's adding experiments to climate change you're actually having to do probe which means we need to run climate models what if at scale 100 to 1000 times faster than they're currently running and we need to have uncertainty quantification of the output of those models otherwise we'll make the wrong decision and this is not a thing where a person loses his job we only have one planet right we only get to get this right once or wrong one so so that's an area where when people quantify risk we really really better do it well and that is a science modeling thing and the two other examples I gave were you know from science models the models came from you know biology and physics this is climate modeling it's actually you know fluid flow models and all kinds of interesting things but but the only way we could scale our scientists to use AI and to make the AI safe for some of these things we really have to up our game and the nice thing about upping our game on those is of course if we were able to on those things at scale we would do all the other things we're asking us to do that there are other people have been you know when these smaller systems would be quite a lot easier because we had solved the problem in a really really hard space so that's my last thing I want to say about there is actually one existential risk post to the human race right now and we could make it worse if we use AI wrong um so I just reminded that you can submit questions on the slide though the details are on the screen 82510 we'll go to questions in about 15 minutes um Anna so you've been in the space for a long time in the human rights space and obviously human rights watch is focused on documenting uh abuses but I'm I'm guessing you must have thought a lot about prevention long term and I'm interested to know how you might approach sort of long term risk assessment for a technology that is is very hard to predict how it goes yeah that's true and I think you know there is a component of sort of forecasting in the human rights world which is you know part of human rights due diligence um which uh do you reference which the human rights impact assessments and again kind of going back to use case we would always you know recommend that human rights impact assessment was used as part of you know social economic kind of impact assessments and I think yeah there's a sort of you know in terms of thinking about the the impact of rights on individuals and the duty bearers responsibilities and so on and so forth um I think context is very important and so um there was a great paper out last week I think from um Deep Mind's ethics research team about social techno social technical uh impact assessment something along those lines in terms of like forecasting the risk of generative EI and it's really about sort of situating it in context of what happens you know what are what are the kind of live components of that and you know who's impacted and building on you know the work of human rights communities and you know sort of uh yeah the kind of whole human rights advocacy movement that is based on participatory methodology around talking to people who the systems are ultimately going to impact and so on and crucially creating means for remedy in redress which is a human right you know you need to be able to challenge decisions when you feel that you've been sort of maltreated and you need to be able to compensate it adequately I would say that if you can't if you're building a system or you can't imagine a sort of adequate way to compensate someone that system should not be built frankly okay so we've we've obviously referenced the summit um but let's talk about it more directly um and everyone in this room they will probably have opinions on the summit and some might be critical of it some might be supportive of it everyone will be focusing on it next week um so interested to know what your views are on the global AI safety summit and what you hope will be achieved at it I'll start with you Gina do the unexpected so um I you know this research center that I run we call ourselves radical re-thinkers so I'm going to say actually something that is supportive of the government here yes that's quite unexpected um listen we're going to play the summit we've been dealt so much about this AI safety summit has nothing to do with the problems of AI and digital technologies and building just digital futures that I work on right just not in my remit but here in London there will be five days of packed full of events that are bringing together the community around justice around international concerns around regional concerns around AI around civil society businesses basically the entire agenda of what AI could be will not be discussed in Bletchley it'll be discussed in other places and so we've got to play that card that we've been dealt but I want to defend the government's framing of the summit because we're in a particular geopolitical context we're in a moment where tensions are high around the world you may have noticed and you're not going to get the US and China at a Kumbaya let's make AI work for the planet AI summit you're just not going to do it and so if you think about the goals of the AI safety summit as let's create global regulation for AI it's going to fail that's not its goal and it's not the goal that the government this government has set forth this government has set forth let's talk about this narrow slice that can get some of the most powerful actors in AI to formally sit down together and do the kind of horizon scanning that both John and Ben have said we should be doing that's going to happen next week and if you've looked at the agenda which they've published the first day is basically multidisciplinary tables round tables with 100 or so experts not it's a very small group and they're going to do this kind of horizon scanning and I'm a critic but that doesn't sound like such a bad idea now is global regulation going to come out of that discussion absolutely not are we going to normalize east west relations and somehow get a thought in the coming a second cold war that is that is happening between the US and China over AI also probably not but we're going to get the summit that we've got and I think there are a lot of really interesting opportunities that can come out of this this moment so let me make one more plug the summit's happening in the UK it's not happening anywhere else if you haven't noticed and that's playing into a certain narrative that we talk about here and it's playing into a certain narrative of this government in this particular political context where an election will be called in this country within the next 12 months that said you know we can say the UK we know the UK is third in the world in terms of the investment and development and research on AI not so bad right so now if you narrow that focus and you say let's look at responsible trustworthy and safe AI let's look just at this focus around the questions and concerns of developing and building out privacy enhancing technologies in in investment in responsible and trustworthy AI from a science funder and perspective in in terms of cultivating small and medium enterprises that are working in that exact space the UK is punching above that number three spot arguably it's either number two or number one if you drill just in that responsible trustworthy and safe AI bit so now we have a very different geopolitical map of what's happening at Bletchley right the UK gets to assert it's um truly world leading position in a slice it gets to play with the with the with the biggest players in the space and it gets to convene and lay the table for a conversation that cannot officially happen so when I talk to my colleagues at the State Department they like hump hump hump we don't understand why China's on the invite list harump they get to save face last week China you may have noticed released um uh an announcement where they said they're spinning up their own global AI safety institute that will be available to their allies in the belt and road initiative harump harump everyone gets to save face because they're going to go to Bletchley and have a great time that's a goal of this summit for the rest of us we've got work to do we have five plus days of events around everything else from human rights labor rights the global south yes research networks industry building industry models we've got so much work to be done and it's a chance for us to use this opportunity play the summit we've been dealt and use it to the advantage to furthering the conversations that we want to have very good answer um should also add the agent smith our president will be at the summit uh next week and we will hopefully feed in some of these uh thoughts uh dear so obviously this is the global AI safety summit have you heard much about it in the sort of countries that you engage with what are you hoping will come out of it um either at the summit itself or in the conversations that surround it yeah i think i want to build a bit on what Gina said i think you know it's it's helpful when you kind of set in that geopolitical context because that's really critical but i guess the larger question from from sort of broader global perspective is that there are just so many competing proposals on the table there is the china approach the us approach the rusher approach the india approach the east africa approach the one that is own national like legislation around ai there's just a lot of competing proposals on the table so like the question that i'm thinking of and stickers that i'm talking to a thinking office how does it all kind of come together does it juxtapose against each other um what is of the common sort of threat and of course this summit is not going to give us a global regulation on AI but is it sufficiently intentional about tying these competing proposals into some sort of a narrative into some sort of inclusion and what would that actually look like because um because without that being in place given the technologies are global they are interoperable they are designed predominantly by american and chinese companies so that's for the larger ones it's going to be really hard to get any teeth in any of these proposals so i think that's uh that's a critical question that at least my stakeholders are thinking a lot about i think the other aspect of that and i want to kind of talk about just the just the trust deficit deficit that exists now between sort of the north and south right because of you know the most recent one is the pandemic where sort of vaccine deployment was it was a colossal mess right there was a lot of distrust that was reintroduced in an existing inequitable situation where the global south is now leaning towards partners like BRICS towards the bulletin road initiative because they just don't think uh you know partner allies or partners like the uk the us are really sort of you know putting their money where they're making their commitments right so there's a there's a lot of trust deficit that now exists across the global south so the question is going to be how does the uk navigate that particular trust deficit and kind of getting to where they need to get to and the third piece of that is around who is in the room i think you know it's great to have a hundred people i would love to know who these hundred people are from a world of seven billion people i would love to figure that out because i think more and more um you know you can't have a handful of nation states and like a couple of private sector companies deciding what's going to happen to everybody in the world and i think that's fundamentally flawed and and we've seen more and more how multilateral models when it doesn't have equitable participation fails so a good example of that is the multilateral agreement and investments uh the anti counter counterfeiting trade agreement uh some of the cobit agreements all of that has failed because a couple of people are a couple of countries who are sort of leading in these spaces are making choices without getting the uh support and the buying of other parts of the world and then when it comes to actually implement scale globally become a global leader they're just sort of caught off guard because they just don't they just don't have the buy from anybody else in the world so i think i'll be very curious to figure out um how you're thinking about um in inclusiveness how you're thinking about equitability and more importantly how do how do sort of we start thinking more about multi stakeholder as a model as opposed to multilateralism because i think in the last decade the reason technology regulations has largely has been a bit of a messy spot is because it has followed a very multilateral model which just doesn't work in something like the erinat or ai which is so much of a public good so how do you really evolve from a multilateral to a multi stakeholder model as you're thinking about this summit is going to be a critical question i'll let ben have the final say before we go into questions so stability ai is one of the companies that will be at the summit i think you yourself will be at the summit so what are your expectations going in what do you hope will come out of it we'll make us up with what we hope um we hope that um in a very tactical sense there'll be a sustained commitment from every government industry civil society representative there to support safety across the ecosystem and across the supply chain not just in a cluster of big models from big labs um what does that mean in practice that means research it means evaluation and testing resources it means standards making it potentially means regulation but for that to account for this diverse ecosystem and not just focus on one very very narrow part of it one very narrow part of it that doesn't need the money frankly um there is a academic and grassroots research community that brought ai to this point and um needs to be resourced and nourished going forward as well i think in a broader sense we would hope that people come away from the summit with a a kind of critical optimism that powerful technology can be developed and deployed safely even if it is frontier even if it is powerful um i think the way that frontier and ai have kind of been used as rhetorical crutches over the past year has led us to a place where a lot of people are afraid of technology because they think of the kind of skynet or or paperclip examples and they're not thinking about the circumstances in which you can deploy it you can develop it safely and so we hope that that is one thing that that everyone comes away from that summit is sense that this can be done and it can be done safely um and and that you can have an ecosystem that is both open source and closed source not one or the other which is how it is tended to be framed i think in terms of our actual expectations going into it um we deal a lot with washington and brussels and london and and singapore and tokyo and and governments around the world they're all tackling different problems i think this is sometimes lost in in the public commentary on this right but what the ai act is attempting to address in europe is uh transparency and disclosure and application layer obligations what certain senators in the u.s. are tackling a existential risk and national security risks what the u k has committed to in the white paper is a kind of product safety approach where each regulator deals with ai in its own specific domain um and so i think our expectation going into this is that each jurisdiction is tackling a different problem has different things to show for it legislation or funding commitments or whatever else um and i think we are still unsure about what exactly the consensus will be coming out of that hopefully resourcing for a safety institute hopefully a commitment to harmonising uh evaluation research for example but beyond that it's it's unclear because each country each regional bloc is dealing with a very different problem great i just to add on ai regulation is a field divided by a common language right so the same words even in the u.s. and the u.k. the same words are actually meaning and being used in different ways and i think that's something you know if that's if that's the only thing that happens um over the next week that'll that'll be useful for going forward absolutely 100% like i think governments around the world even when we talk to them about explainability it means so different across different contexts that there is just no common definition common understanding and therefore to Ben's point we're seeing using the same terminology tackling entirely different problems having entire different proposals on the table which i think from a pure privacy perspective there's a massive compliance burden so how do you actually streamline that it's going to be it's going to be a critical question and i hope the summit gives us some direction on that okay let's go to questions there's some really great ones on the slider um i'll go to the slider one first and then if you think of a question put your hand up and we'll have we'll send a mic to you so the most upvoted question is from James phillips and the question is why do you key figures of ai like hinton, bengio, altman, hasabis, leg and altman again um unless it's a different one i don't know about you have driven the cutting edge take frontier risk much more seriously than you so we start with john that's an interesting question the the hinton interview uh with ang from intel um does not go that far and i think some of these people have actually taken a sort of position that's for performance reasons you know and um maybe to boost the value of their company or maybe because they want to distance themselves from where they used to work or whatever but i i i um i don't think there's some magic great thing about people having 50 000 citations on paper in europe's that means that they have a greater insight into the existential risk and fundamental agi as it's nor normally known uh that's going to come out of lm's which is not going to happen um this is just simply you know not the case and i think that consensus technically is that those statements are not part of a peer reviewed scientific process frankly uh and that the community is i think what we've expressed is where things are at um if you want a science fiction reference then dune had the balyrian jihad against agi 50 000 years before anyway we can discuss that some other point but that was about agi and these guys are making we're making uh in implications that are just not born out by the reality um by anyone who looked at the work in symbolic ai and neuroscience informed 45 years ago such as university or you know 30 years ago at edinburgh or whatever it's just uh incorrect so i'm sorry um i you know so that's my position on on their their positions and you know they've lost people have heard of them much more than me so that's fine i've read a bilateral drinking game um at uh social science poo which is a gathering that happens at facebook uh cosponsored by facebook and orially media in silcon valley and it brings people from the uk and somehow i count as one of those people from the uk it brings people from the uk together with people in the us to think about computational social science and so we um have both pre and post pandemic run this drinking game um called first calling bs on ai and um where someone had to stand up and actually say i call bs and they could use foul language we of course in the uk don't use foul language um uh those americans um so you know you could get up and you could say this thing and then and then and then you would call out you know what you're seeing as an ai failure and we ran it in in february um on generative ai and i you know chathamhouse rule but you know the um science fiction right he has given permission to say you know science fiction writer kim stanley robinson was in the room and he was like i call bs on ai because it's bad science fiction and i can say that because i'm a really popular science fiction writer okay you sure i'll i'll take that you know meredith brisard and an artificial unintelligence in her book um traces this kind of first wave of of ai pioneers and and their complicated relationship to a to science fiction and you know the idea that we're scared of these films and we're scared of these books and we're and we're talking about the capabilities of the models we have today and and the problems you know it's just it's just not helpful for how we think about policy and if i can just ride one more of my hobby horses here you know in the last few days i've seen very well intentioned op eds from really bright people right who are very good at making technology and they've been proposing what kinds of policies we need to have and they've been saying you know it'd be great if we could just have an international atomic energy agency for ai i'm like dude that's my lane and let me tell you how many treaties and how many years it took to get to this place and i don't know if you've read any news about you know atomic capability in states and the fraught nature of those conversations or that we actually have treaty obligations that that countries have signed up to that that agency is the official assurance agency for but i don't think you actually want to make that proposal right now so i think we have to be really careful when we reach for these metaphors right without looking at the really hard work of the how how are we going to implement these things how are we going to make it happen who's going to make it happen what frameworks are we going to use let's just start with what benchmarks do we have what horizon scanning can we do what kind of shared language do we have if we get countries together on a little bit of agreement or at least agree to talk about those things then this summit will be a success so not the ministry of the future that that that that would be an optimistic look right that would be an optimistic look let's bring him to royal society in the next few years wouldn't that be great i think the other thing is that there's actually a lot of nuance in these positions that the kind of frontier risk people have taken and often it's kind of caricatured through the media and by industry as to what their position actually is there is much more consensus on frontier risk i think than people suspect right i mean the idea that if you have a system with very high consequence risks risks that are potentially catastrophic to the entire community and there are very few countervailing benefits to this system yeah like don't build that thing or you know if you do do it in a very controlled environment that's sort of unobjectionable to anyone in this in this space the question is empirically where is that line in the sand there are many people who think that lama too for a matter is a frontier model it's not a frontier model in the sense that they describe frontier lama too can do many things it can do many bad things but it is not going to destroy the planet and the idea that a company or a research lab building gpt 5 is going to need to obtain kind of sanctioned approval from the u.s. state department to do so is terrifying we don't do that in software normally we have we have some very narrow instances of technologies and software uh zero day exploits for example high precision gps things that you can turn into a missile that are subject to this kind of control regime lama too is not that and and the idea of of only giving a few organizations the authority to develop these things is is unsettling it comes back to this question of the how right it's absolutely that there is a potential risk there absolutely we need to be thinking about what evaluation and mitigation look like for those risks but the idea of jumping in 2023 to some of the legislative proposals that we've seen in the united states and elsewhere is it's premature it's not consistent with how we regulate almost any other technology and it's and those proposals are often coming from people who uh are very vocal on frontier risk have potentially worked for a long time in computer science but um clearly have not spent a lot of time looking at how we legislate and regulate for these things in a practical sense and so you know I think I think that's challenging I think so to the question we take it seriously we've certainly signed up to a few of the statements saying let's be very careful as we develop technologies that have unforeseen unpredictable capabilities um but that is different to saying where exactly is that line in the sand and on the line in the sand question you have uh very very diverse perspectives any questions on the floor yes uh niki in the middle and someone take a mic also if you could um just say your name and the affiliation and try to keep it brief if possible okay uh hey my name is niki I'm with Chatham House uh thank you very much for the very insightful panel um I have the first question is about you know the solutions for AI safety risk because a lot of times um you know regulators mention about this human in the loop approach but that's based on the you know condition that we believe a human is capable of telling what is right or what is wrong from the decisions AI is making but nowadays we're dealing with a word like AI is shaping the society like Gina mentioned some junior doctors already fighting for training opportunities if AI is taking over the junior jobs and also also the learning opportunities from human workers how can we build this experience to be that good to be able to tell whether human AI decisions is right or wrong if we are not having this capacity as a as a human is this some sort of like AI safety risks that concerns the panel thank you I don't want to come in with this dear perhaps human people talk about human in the loop often is people sometimes in the global south do you have any thoughts on this or anyone else on the panel yeah I'm getting a start I mean I think the first part of your question was about what kind of solutions could potentially exist and I think um it really goes down back to the basics because it goes back to the initial definitions of AI versus automated versus all that right so if you take the very basic of it how do you really safeguard against those kinds of risks so that's better data protection laws better consumer protection laws better IP protections better labor protections better environment laws those things already exist to some capacity so the question is how do you kind of take these existing legislative models and then think about some of the things you would have to make in order for this to also apply in those particular contexts because and I think the viewpoint that least I take is you know possibly news really expensive and and especially if you are a resource strapped government or a resource strapped state you have a bunch of things you have to really worry about too like I think Anna's point earlier so if you are trying to also mitigate those kinds of safety risks you're also worried about that then how do you leverage your existing frameworks to then be also able to provide certain kinds of safeguards against AI models or automated systems so I think that's a really big part of it and I think the other aspect of that there's a there's a lot of focus on human in the loop and I think it for me it kind of goes back to the old argument around if you had hired 100 million content moderators all the child sexual pornography would have been gone like it really doesn't work that way so I think the question is it is really important to have a human in the loop but the broader question is sort of how do you think of it at the design level to ensure that you're building systems that are safe or you're kind of drawing that line and if it's if it's just bad if it's a system that's about predicting social credit scores just don't build it that's pretty that's as straightforward as that. I mean I'm kind of reassured by what Penn's been saying about the whole framework of the safe debate you know but in terms of in terms of the systems they're there they're not going away we have to we have to work with them and we have to embrace change and innovation we have to ensure that people are trained correctly and we need to make sure that as we develop new new roles or new systems we've got workers that are trying to go with it but we're not you know we're not we can't stop it innovation is something we only embrace and therefore you know this is going to move on and we need to move on with it but the frameworks and the regulations that will come during the debate will help us get there right but when when do you take a human out of the loop then that you have to find something else to guard it and it might I think there always has to be a human intervention where you've got people where you've got workers but a lot of this won't need human intervention it will need protections and guards and frameworks around it so I think there's two different debates going on one about how we grow the infrastructure and the machinery and then how we guard against the worker but where where we grow the machinery and that we develop the AI I think we are very supportive and really welcoming that I think as a as a kind of citizen we're scared we don't know what it means we listen to the media we listen to the you know the debate but we actually aren't getting those reassurances the kind of conversation that Ben's having about how it's got to be safe how it's got to be protected that needs to be a debate with people so that we're not having a scientific discussion and a government discussion but but an understanding in society of what's coming at us you know we hear about oh it's going to be a bus with no person it's going to be well yeah okay and as you said before you know do they really fly the planes have they been flying them for a long time or you feel safe when you're in the plane anyway so it's this kind of mixed message you know we embrace technology we work with innovation we just need to make sure that we understand it's transparent and explained properly so that people will embrace not be frightened and actually just to there was a point in there around that the people aspect of this is interesting because a safe outcome or a good outcome with AI means very different things in different contexts right so if you're saying does the NHS AI chatbot tell you to drink bleach to fix your COVID infection right like that's a kind of basic product safety problem in many ways and regulators know how to deal with that right you look at the reliability of the system how often does it give you a harmful response how often does it give you a helpful response but then you look at other issues like the the doctor's example you gave another example of this is kind of the sag aftra strike in the in the United States and um and actors who are concerned about systems AI or not that um are able to replicate their physical likeness their performance right and uh and what does that do to background actors extras what does it do to pathways into the acting industry in Hollywood the problem there is less a how do we evaluate the system problem right the problem there is a kind of bargaining imbalance problem between in this case actors and studios and how is technology going to affect that bargaining imbalance um and so you know in a funny way and and you know certainly have to be sensitive about how we talk about this from the tech side of the lecture but the wga's approach the writing the writing guilds approach to dealing with this and the negotiations that they held with the the producers and the studios was really interesting it was a really interesting look at how how are you going to address some of these problems in a non scientific way and a non regulatory environment you know where does bargaining and other systems kind of come into play there um so it's something we're following with with interest what that looks like you know we certainly as a model developer we want to build tools that help with tasks right push a button get an image push a button get a poem is kind of a caricature for how these systems are going to be used like fundamentally we're seeing Broadway designers use them to mock up new concepts for the stage you know we're seeing people using them for analysis and classification tasks medical research teams using stable diffusion to take MRI scans and see if they can visualise what the patient was seeing and what could that do to medical diagnostic approaches for different neurological disorders right there there are functional use cases for all of this that aren't um a kind of wholesale replacement for a person or a job and that's the kind of thing that we want to build but we need to have this conversation with uh with users and with creators and with workers who may be affected directly or indirectly to make sure that we're kind of keeping to that mission um I'm going to move us on to another question because there's a lot of good ones we're running out of time um so on the slide we've got one from Zisgy from Wikimedia uh a question that I think we're interested a lot of people in um the political world so what guardrails or coordination is essential to put in place around the use of genitive AI tools ahead of a year with over 40 countries holding elections and Anna I'd like to start with you God um no it's funny I was at a um round table with a bunch of tech companies a couple of weeks ago discussing exactly this and I genuinely don't think that anyone has has a good answer for it you know I think like really it's a sort of wait and see in some ways um yeah I mean I think there are there are things you can do in terms of like guardrails you know sort of um putting you know sort of mitigating factors in place whether that's um all of the kind of technical aspects um that Ben ran through but then at the end of the day I think you know just sort of we're going to need more investment and more resources when it comes to content moderation when it comes to you know enforcement of policies and so on and that's something that kind of concerns me particularly with regards to the discrepancies between one country and another and where the resources are going sort of globally is not equally I think you know one I think sort of if I look at something like the DSA or the DMA um cynically I think that the tech companies are going to put a lot of resource into um sort of showing kind of transparency um measures that they're making in Europe because it's you know it's going to be public and I think if we could have that level of transparency globally I mean my kind of answer to a lot of these questions is sort of you know let's shine some light on it and we need transparency you know if anything from sort of algorithmic registers around kind of public use of of AI to um yeah kind of like sort of uh understanding the resources that are going in from tech companies in specific countries um I think particularly with regards to elections that would be a great help don't know if anyone else let's actually move on because I want to try and squeeze in some more questions before we close any more questions on the floor yes a gentleman in the jumper and the shirt clearly I would never get a job on BBC question time yeah thanks for that um good evening I'm Sidhan Charrergy from Holistic AI my question is for Dya and Ben specifically um increasingly the trust and safety ecosystem is utilising and leveraging generative AI to create synthetic training data to train um you know content moderation classifiers hashes etc my question is how do you ensure that that synthetic training data in itself is representative is bias free or you know risk to bias are reasonably mitigated what are the considerations that can be used to underpin that how does that happen in practice and more generally do you think regulation has a play here Ben it's not yeah absolutely um so I think it's important to acknowledge especially given this headlines around this issue that synthetic data can be used in a number of ways as part of developing one of these models I think the most interesting example of this is um actually from the kind of the driverless car world right where you have vehicles that are trained obviously on a lot of real world data but then you're also using synthetic data to expose the model to scenarios that are very important but it's very unlikely to encounter in the real world right the car needs to drive on the real world to understand the nuance of pedestrians and cats and soccer balls and things like that but you also need the vehicle to know what happens if an oil tanker drives onto the wrong side of the road and that's not going to happen every day um the same is true in generative AI you use you can use synthetic data to expose the model to edge cases that it wouldn't otherwise encounter you can also use synthetic data to correct for certain biases that you otherwise can't correct for in real world data um we do a lot of work with our image and language models thinking through how can we make them more representative and correct for many of the biases that exist in for example the internet um you know one of our language models is currently the most capable open source Japanese language model our image models have gone through a whole bunch of work to fine tune and correct for some of these behaviors in some cases using um synthetic forms of of data so I think it's it's not I think media has spoken about this in the context of a um a kind of doom loop where AI sort of feeds on AI contents and it just becomes kind of cyclical and recurring and eventually everything implodes the reality is synthetic data has been used in much more limited ways and most of those applications at least in our experience have been around correcting for bias and making the model more representative yeah I think just two parts to it I think you know a few months ago there was in a hackathon where we tried to kind of lean on synthetic media to be able to train better trust and safety models because to Ben's point you're not going to get right now the way harm is determined is when it actually happens on a platform and then you do something about it which is very reactive so if you were trying to take a more proactive approach to it could you then use synthetic media to be able to predict as many harm scenarios as possible qualified across as many cultures and other kinds of diversity indicators as possible and then understand how that harm is being happening so I think there is a positive use case to that and I think that hackathon was kind of trying to explain that but I think a big part of that is in terms of representation is that the existing data system they don't really have that so you have to be quite intentional about how you're um in when you're trying to even correct it how you're kind of being very representative about it and so I I think there is some really good work happening for example in southern Africa where they're like trying to build out more training data to be able to inform synthetic data and then to be able to then create more representative models I think there's a good work happening in South Asia on that as well but I do think it really comes down to at the again that the data collection level are you one getting diverse data sets representative data sets two I think in terms of just the different modes of data collection that you're doing so for example right now it's predominant in English are you getting multiple languages global sub populations converse largely in audio video and not actually text so are you then using the sort of different ways of collecting different forms of data collection so I think there's different aspects to certain cultures that needs to be baked into the data collection phase that then essentially informs the design phase so it's a bit more of the supply chain level as opposed to the application level I think being very intentional about it from the very beginning is potentially the only way to address some of the synthetic media question that you're asking I want to me I want to the next question and so one of the reasons for doing synthetic data is you can give synthetic data to other people but in the process you have to do differential privacy which means you usually reduce the precision about rare cases which may be the interesting ones there's recent work in the Turing Institute with the Financial Conduct Authority and O&S in fixing that problem to a large degree so you need to look at literally the last years New Europe's papers in this space because things have improved at the representativeness of the data in the tailors of the distribution which matter if you care about because there are the lots of other uses isn't it I'm glad you didn't say that actually because you set me up nicely to mention that the Royal Society has two reports on privacy enhancing technologies which also covers synthetic data um I'm going to take another question from Slido which is from Spicy Takes and Lloria hopes because it's not really a spicy take this question this is a very normal question so um what is the role of universities I'm going to give this to Gina to answer what is the role of universities in supporting AI for science given that they are behind the cutting edge and do you not have access to data, compute and high salaries It's not a spicy take is it? So at the Royal Society today we actually spent the morning working on horizon scanning on AI for science um yeah you've named all the problems um I could add 10 more when you have the head of university the president of the university here in the UK saying yes in the sector it's going to collapse that's another problem um listen academic research has a lot of things going for it I don't know how I'm going to finish the sentence time is one of the things we do have and and academic researchers are empowered by curiosity they're empowered to work on problems where they can have time to discover some of the research that I've done on how teams take on AI and other technologies to to do new kinds of collaboration show that sometimes just bringing advanced tools into the lab allow people to ask new kinds of questions and not to say that doesn't happen in industry but in industry yes there's more money and more compute and more power perhaps but time is not the resource they have and they don't have the resource to be curiosity driven in most cases right there's a product there's a bottom line there's a manager and there's really smart people and interesting problems and you know my colleagues in industry love their work so I don't want to belittle that so so yes um we need a lot to keep universities competitive and I'm going to make one more plug we need a lot in this country to keep our universities competitive um it's it's not simply about compute it's not simply about the salaries it truly is how are we investing in people and how are those people being empowered in universities to continue focusing on curiosity we came up with the conclusion at the table that Arik and I were at the round table we were at today really that you know people want in universities scientists want to do good and they want to do good work it's a huge motivation for those of us so can we please get what the basics we need to ensure that our science in this country does not collapse really could talk about this all night but unfortunately is 8 30 which means that brings us to the close of our event so thank you so much for joining us this evening I hope you learned something new and as you may have noticed I mentioned many policy reports you should read them if you really want to understand these topics well you have to read you can't just listen to a podcast and you know we're better to read than a raw society publication finally come over round of applause for x and pan on everyone who helped put together this event and the whole day today thank you