 Now, in 1958, that was actually kind of before at the beginning of the Gulf ice stream, but this was kind of subculture and actually proposed from a psychological point of view, it was the first neural network proposed, actually, there were earlier ones, but the first one that connects to the current developments called the perceptron. And it was drawn like this, the circle could be considered to be an artificial neuron. And these arrows are inputs. And these inputs represent numbers. It could be, for instance, be that you have three numbers x1, x2 and x3. These are numbers that represent a client for a bank, for instance. And x1 could represent the age x2, the number of children and x3, the salary. And these were all added up and summed in this neuron. But these arrows that you see, they have a weight. So if the number of children is more important or the salary is more important for the ultimate decision, you could weigh this input. And this is not something you do manually, but you can automatically train the system to weigh these inputs on the base of the output that it's giving. So a very simple task is if you have a bank and you want to decide if you want to give somebody a loan, you could just collect a lot of data and decide whether somebody should have got a loan or not. So you have the kind of ground truth. And then you check for all these clients if the perceptron would give the right answer. Now, initially, these arrows have random weights, but it was shown in 1950, I ate already that you can change this by using an automatic learning procedure in such a way that it would give, in most cases, the correct output. Now, for those of you who know statistics or work with statistics, this is a kind of neural metaphor for logistic regression. And of course, that happens a lot in this field that you have these mystifying terms like neural networks for things that are actually very well known. But in these days, it was revolutionary because it made a connection between the idea of a neuron in the brain and this learning system, artificial learning systems. Now, now a jump to 1980s and like these times. So on the left, you see the 1980s versions of stacked perceptrons where you take a lot of these perceptrons and stack them in layers. And this was what I worked on in the 1980s and many other people. And you could have a kind of input layer where the inputs were like these characteristics of a client. And you had a hidden layer where we're neurons that could compute complicated functions that you needed to compute an output layer. So these neural neural networks in 1980s developed by people like Jeffrey Hinton and others, were able to solve more complicated tasks and better able to make predictions, for instance, for this bank application, if you want to decide if you should go give somebody a loan or not. Well, in the 1980s, there was also a person called John LeCun who developed a deeper system and applied that to zip code recognition. It was very successful. So this is 30 years ago. But we also try to build these systems to recognize these images of my dog and other, well, I didn't have that dog then, but other dogs. And that didn't work. So people thought these neural networks failed. So that took a long time until now where we have these deeper neural networks. And now we can automatically classify images and train the system on automatically labeling images. The only provision is that you need a lot of data. And you need a lot of these arrows that you see these connections, all these connections that you see here. And this is, of course, a caricature, but these connections are adaptive. So it's a sequence of steps that you go through this neural network. And each of these steps transforms the information up to the end until it gives the right answer. If it makes a mistake at the output, you have a special algorithm that propagates backwards to the network and changes all these connection strengths until you have the correct configuration of connections. Now what changed between these two steps? The first is that we have much more data now. So the fact that these 1980s things didn't work was you had a lack of data, but also lack of computational power because to create a very deep network, you need a lot of computational power. And there are also some machine learning innovations that help there. But actually, the neural networks that we have today or the bulk of the neural networks were already available 30 years ago. But because of the lack of data and computational power, they didn't work that well. Now, this is to intimidate you, but especially to show you that the thing on the left, which is a neural network is fully understood. On the right, for those of you with mathematical knowledge, it's, there's no, it's fully transparent in the sense that we know how these neural networks are connected. The point is that like with this pattern recognition of these elephants in the shadow, is that all these connection weights, it's not like if then rules, it's kind of distributed over this whole network. So it's very hard to explain how it arrives at this output. But it's not that we do not understand what happens in the neural network. So it's another mystery. It's just how to interpret it, what it makes it difficult. Now, an intuitive notion of what these neural networks are doing, and this applies to all these deep learning networks that we nowadays see, is that you have a big black box, not so black a bit gray, because there's some transparency there. With a lot of knobs, you put something in, you look at the output. Is it correct? You do nothing. If there's a mistake, you start changing these knobs. And of course, if you had to do that manually, that would be very difficult, because the number of knobs on modern neural networks are in the millions or more. But we have a very straightforward algorithm for that, which is very well known in optimization. And it's called gradient descent. And it's just following by each time changing the knob a bit in the direction that minimizes the chance of making an error. And this is called back propagation. And this algorithm is also known for 30 years, and maybe a bit longer even. Now, there are nice videos, I always show that if you want to know more about how these things works, go to three blue, one brown, which are YouTube videos that explain, give some intuitive explanation of how these networks work, also for the learning procedure. But I skip that now, because that takes too much time. Now, this is what we had about 10 years ago, we had networks that could be trained on about 1000s of images of dogs of different breeds, including my dog, the breed of my dog shown on the left, which is a Samoyed. And by training the system, you were able to, after training the system on all these 1000s of instances, and they were labeled so you need 1000s of instances of Samoyeds, of Papayons and Pomerani, I know don't know all these breeds. But after that, the system is able to automatically label a new image of a dog and indicate what the breed of the dog is. And the way this network is doing that, it's exploiting anything in the input that helps it to predict it. So I always give the example, suppose that all these Samoyeds were depicted in a white background, because they come from Siberia, then the snow in the background, the white background could also be used as a cue that has nothing to do with the dog. But this system doesn't know it's what we call greedy, it uses anything in the input it can use to predict that this dog is a Samoyed. So that means that the quality of the data should be such that there's a there's also variety in the backgrounds, because otherwise, the network will pick up these features that have nothing to do with a dog. And this applies to image classification, but actually for all machine learning tasks. Now there are also systems 10 years ago that could be trained on images with captions, and then they could automatically generate a caption, the description. So this is from a system trained on millions of images with captions. And then it was confronted with this image. The first part of this network was a vision network like for the dog. And the second is a network that is trained on sequences on generating descriptions. And as you see this description, it looks as if the system understands the picture. Well, this is 10 years ago. Now we have GPT-4 that's even better in this respect. So it's better able to label these images and even describe them in a way that seems to suggest that it has an understanding of the picture. Now, a small thing about the internal structure of these neural networks, as I said, it's hard to interpret them. But for some domains, for instance, the visual domain, you can inspect the inner workings of the network. So here you see three layers of a network that is trained on a face recognition task. And what you see is actually the sensitivities. What are the features or the visual aspects that these neurons that have been trained by using this procedure, that they detect in the images. And what you see the first layer, it looks at contours, oriented contours. And intermediate layer, it looks at small configurations in the face. And on the top layer, you see kind of face detecting neurons. And what I'm now saying you should take very carefully because it can be misunderstood easily. A similar strategy is used by the visual system. If you ask anybody who studied the visual system, you see neurons in the first stage that look like this, so called edge detectors. And in the middle layer, you see these kind of configurations. And at the top layer, we have neurons that respond to individual faces. And this is not to say that these neural networks are as complex as the human visual system, because they're not, they're much simpler. It's only that the general solution strategy is very similar to what we see in biological visual systems. And this system was not programmed. It was only programmed on predicting the correct label. In this case, I think it was a gender recognition task. And nothing else. So it has no knowledge about eyes or nose. It doesn't understand noses or eyes. It's just using the pixel values to make this prediction. Now, this is one form of neural networks. There's another form which is called reinforcement learning. And this is an example of reinforcement learning. The two pigeons are either end of a small ping pong table. One pigeon pecks the ball. So these are pigeons playing ping pong. Other pigeon pecks the ball back across the table. If it goes past one pigeon, the other pigeon can eat. And if it goes the other way, so now he wins and he gets something to eat. So you reinforce, you you reward the pigeon that scores a goal. And in this way, they learn to acquire the skill of playing ping pong. This is very old. This is psychological research by Skinner. A long time ago, which gave rise to a whole stream of what is called behaviorism in clinical psychology, but also in cognitive science. But there's a variant in deep learning. It's called deep reinforcement learning. And it's used for creating for creating action. Unfortunately, the video doesn't work because we have to change to PowerPoint. But the idea is that the one of the hardest task for autonomous drones is to fly in unstructured environments like a forest. And they succeeded by using deep reinforcement learning, which is just a variant where you reward the neural network if it performs a good action or a good sequence of actions and punish it otherwise. It is called policy learning. They were able to have this drone flying very noise under noisy circumstances through forests. And this is, of course, one of the spin-offs in many military domains nowadays. Now, what is important to realize is that if we think about science, there's one important principle in science, and it's called Occam's razor. So if you want to explain a phenomenon that you don't understand or you want to model something in a statistical model, you should not overfit the data. And overfitting means that you have too many free parameters. So these knobs that I showed you in these neural networks are all free parameters, things you can change. But if you want to model something, you should reduce the number of parameters. Otherwise, you get overfitting, as shown on the left. The red curve, which the complexity of this curve is based on the number of parameters, is too large because it's fitting the noise. Actually, the curve on the right seems to capture the proper structure. So this is not only true for statistics, but also in science. If you see something flying in the air that you don't recognize, you do not start with an elaborate theory about that the aliens arrived. You try to come up with something simpler, which doesn't require so many assumptions. It's a basic principle of science. And as you can see on the bottom right here, that's what good scientists do. They reduce the number of parameters. But AI researchers are more like pirates, and they just explored what the effect was of increasing the number of parameters. And that's what they're doing in deep learning. All these knobs on these systems is something that any statistician dreads, because that's not what you should do. But somehow what is called over parameterization to use many parameters is part of the secret of this deep learning success. Now, this principle is now translated into so-called language models. So the idea that you use a lot of parameters, and one of the problems of traditional machine learning and deep learning was that you needed all these images of the dogs, all these examples, but also the labels. Now, what they came up with is a form of learning, which is called self-supervised learning. And if you have tons of text, for instance, you can use self-supervised learning by giving them part of the sentence and ask a system to predict the next word. That's one variant of self-supervised learning. So that means that you only need data, and the label is kind of intrinsic to the data. And in this way, you can train the system on huge datasets, like the whole worldwide web, Twitter, books, et cetera. And what is the tendency is a growing number of these three parameters. And that's what we saw in the past. And of course, this is already leading to a lot of concern about the energy requirements, which is not always a very balanced discussion. But it might be temporary because there are developments here. And this is a kind of overview of what happened in the number of parameters if you just look at the GPT systems. I could have included GPT 3.5 here, but I don't remember what the number of parameters were. But the idea is that it's growing. Now, the funny thing is that we don't know how many parameters GPT 4 has. And they're not very open about it. There were some rumors that it was a trillion parameters, which is a lot. But it seems to be much less than that. So that's why this question mark here. But the tendency up to now was more parameters gave better models. But it seems that there's now also a lot of innovation in how you structure the data on which you train it and how you tune these systems. And I will come to that. Now, this, as I said, you can predict the next word or the token, the official word. Your word, but think about words. Or you can also have must language when you leave out one word and the system has to predict what this word is. And again, this is a system where it predicts the words. If it makes a mistake, you adapt all these parameters. And if it makes no mistake, you do nothing. That's the basic principle. So it's the same principle that I said before. And at the heart of all these developments is a new technology. So this is really new. I told you about these neural networks for recognizing dogs, for instance, these are 30 years old. But this is only for a few years old, this idea. And it's called transformers. It's not these robots that you have these movies. But this is called transformers. And they're based on an idea where you have an encoder and a decoder. So the typical application is translation. So you have a sentence in Dutch, for instance, and you translate it to English. Then you would train the encoder on the Dutch's input and the decoder generates German's output or English's output. And at the heart of these systems, and you can see it's quite complicated, there are some perception like things in there, these traditional neural networks. But at the heart is a mechanism that I can only explain if you first understand how these neural networks deal with words, because words are not numbers. And as you know, a computer only works with numbers. It's a number crunching machine, another word crunching machine. For that end, they use so called word embeddings. So what you have there is a word is represented into a list of numbers called a vector, just a list of numbers. Now the question is, how do you arrive at these numbers? Well, there are many ways you could, for instance, say, well, each number represents a kind of attributes, like important or not important or social status, etc. But actually, these are implicit, because it's more about the distance between these numbers that it matters. And there are systems for doing for creating these word embeddings. And if you apply them, you get things that you can visualize like here, you have a numerical representation. So let's assume you have only three numbers. If you have three numbers, that's a point in a three day space, then you could represent King and Queen. And what you would see that there's a geometrical relation to man and woman. So there's already some knowledge or some semantics or meaning into these word embeddings. It's very limited. But it's a very simple method where you already translate words into numbers, where the distances between numbers reflect kind of similar similarities between words. Now, of course, a similar similarity between words is not one thing. You have many different similarities, they all depend on context. So that's why these word embeddings are not enough. Now the heart of these transformers is a so-called attention mechanism. An attention mechanism can be explained in mathematical terms, but I tried to give you a more intuitive notion. They use the mechanism of self attention. So if you have a sentence like Bank of the River, you want to explain, understand the word bank. Bank can mean different things. But in this case, it's the Bank of the River and not a bank where you get your money from. So the river is related to bank. So if you somehow can include the context of this word bank, and the context consists of other words in the sentence, you get some contextual information that could disambiguate the meaning of bank. And that's what we apparently do if we read sentence and if we talk. And this is how they do it. So they use the word bank and it has, I put it here in the italics to show that this is actually a numerical representation. So this is this list of numbers. And now through the self attention mechanism, they mix with this numerical representation, the numerical representation with a certain proportion of the other words. And as you can see, it's for instance, point five, the original bank concept. But then you add 0.1, 0.1 off and 0.1 the and 0.3 river because river is very important. So these factors like 0.5, 0.1, 0.1 and 0.3, that's the attention, the attention you pay to the other words. And this is just a made up example, but this gives you an idea how attention can help to change this original numerical representation of words into entities that have more context. So now if you see the repostation of bank, you immediately know it's related to these other things. And what you have to realize is that the context, the length of which you can do that depends on the system. So these, this was a very short sentence, but you could do it over longer sentences. For chat GPT is about 4000 tokens, words, kind of words. GPT-4 has 8000 and they're working on 32 thousands. But in terms of human memory, this is like the memory of a goldfish. So, and you know humans have a short term memory and a long term memory. So it's a very limited in terms of memory. It doesn't have a long term memory yet. Maybe they're working on it, but that's a limitation. Now, if once you know that you can use this attention mechanism to create these internal representations, and if I say representation, these are just numbers, lists of numbers, you can create representation of words in context, but also words in context in context by stacking up these attentional mechanisms. And that's what's happening in the GPT systems. You are exploiting the statistics of older relations in this enormous data set that they use for training these systems. And in this way, you create a kind of contextual mechanism that allows you to exhibit some intelligent or not so intelligent behavior. So in this way, these large language models as they are called LLMs are obtained that can predict the next word, but also the next word and sometimes parts of sentences. And you can do this repeatedly. So it can generate whole texts. And of course, they also have indicators for the start and the end of a sentence because it has to end sometimes. And many examples in the past like chat GPT, but also earlier systems had this form of question answering systems. And these systems work quite well. So if I think about GPT 1 and 2, you might remember those of you that have seen this, there was some concerns about GPT 2, which was a very limited system, but many of these ethical concerns were already appearing then. But nobody or about only scientists or researchers anticipated that it would give such a boost if you would scale up these networks up to the level of chat GPT 3.5 and chat and GPT 4 that we now have. Now how did they achieve that? At the basis is a very big language model, a large language model that can, if you ask them unethical questions, it will generate an ethical answer because it's totally unbalanced and everything is collected from the Internet. And there's of course also fascist talk there and an appropriate talk. So something has to be done to keep this large language model in place. And that's what actually open AI did. And this is from their paper. I guess this is a simplified version. But the idea was that they used humans to to look at the demonstration data and they trained it in a supervised way. So they looked at the outputs that were generated. So a prompt that's the enter what you enter into this GPT system. And then a laborer demonstrates what should be an appropriate output. So they give an answer, example here, explain the moon landing to a six year old. And then a laborer would give a description and this description was used for training the system. And of course, the system could already itself generate something, but then it would probably hallucinate something and come up with something that is not to. So in this way, they tried to steer it in the proper direction. And then the next step was that they looked at this model. And they generated prompt and looked at several different answers and ranked them. So they give a kind of ranking of the quality of the answers. This was also done by human labelers. So of course, there's a human factor here because depending on the type of labeler and their backgrounds and their knowledge, they might label things differently. And with this labeling, they could rank the answers. And with these ranks, they could use a reinforcement learning policy. So like these pigeons, but now in the deep learning variant to steer it towards the proper rewards and the proper answering. And that's what they did an open AI. So the picture that emerges, you have this very big, large language model that can go all directions, it can go in a Trumpian way, or in a Socratic way, or whatever. And now they steer it by using these policies in a certain direction that is acceptable and ethical. And you'll even have options in GPT-4 to get an answer in a Socratic way or in, I don't remember in a CEO like way or something like that. So and these steps are very important because these steps actually explain the success and the failures of GPT-4 and also of Jet GPT. Now the problem is that and that's a problem for scientists that GPT-4 is not open. It's open AI, but they don't open up the full and they give as reason that if they open that up, they lose commercial value of course, but it's also for security reasons. Now this is a statement from their paper and this paper came out last week. So it actually says it's very honest about the limitations of GPT-4. It has a limited context window. This is goldfish memory and we should take care of the outputs where reliability is important because these systems give answers that seem very convincing and very reliable, but they're not and that's why they have this disclaimer. Now one more point for GPT-4 is that this idea of transformers was developed in language, but it's now also used in vision. So you can use the same principle by taking not words, but parts of an image and then also use this attentional mechanism and then you get image representations in the same way that you had word representations and you can merge them or combine them and that's why GPT-4 can also deal with images. So this is an example from their report. This is GPT-4. What is unusual about this image? I asked my eight-year-old son and he said, there's nothing unusual here, but that reveals more about my son I guess because it's clear that this is a strange picture and the answer GPT-4 gives is the unusual thing about this image is that a man is ironing clothes on an ironing board attached to the roof of a moving taxi. Now, of course, this is cherry picking by open eye. I understand that, but it's very impressive that this system can do this and I did not anticipate that this would be possible. So that's one side of it. Here you see another nice example that is related to the domain of machine learning. So can you explain why this is funny and think about it step by step? So that's the instruction you give to GPT-4 and actually what you see here is the picture that I showed before with the statisticians on the top statistical learning very nicely so as it should be. And then this neural network people, these AI people that want to stack more layers or more parameters in this system. And there's an extra instruction or an extra mentioning here, but ironically, because of course, we know that these bottom systems are now very successful in terms of their prediction capabilities. And this is the explanation by GPT-4, which again is very spot on and very interesting that it can do that and somehow extracting information from this picture, relating that to text and at least it appears to us to be a very intelligent system. And that's impressive. But of course, there are a lot of caveats here. So that's the question. Does GPT-4 understand? Now, I think the problem with this question is because you hear that a lot is that understanding is something that is related to humans and not to machines. So we could think that it understands, but the term understanding doesn't apply to machines. And that's one of the biggest problems here because if we think about understand, we know what understanding means. And you cannot copy that onto a machine. So that's why I always show this picture. If we see this picture, we know what this is. We see this boy with a frisbee in the park. Sometimes we even had a frisbee in our hands, maybe. And this was nicely illustrated in the MIT Technology Review that if you see a picture like that, you have this rich association with this picture. But the robot can say, OK, that is a person in the park throwing a frisbee or catching a frisbee. But this robot does not have the experience of being in the park and touching a frisbee, et cetera. And now, of course, that the same is true for GPT-4 and maybe GPT-5. And maybe in the future, we can build robots that can throw frisbees and have its experience in the park. So I'm not saying there's an imprincipal obstacle, but there's a practical obstacle that also is related to the fact that we don't understand and don't appreciate the complexity of our world. We're living in a kind of culture where computers and digitalization are everything. But actually, if you think how much you learn and the beginning, you can't remember, but you are born and then you develop into a young adult. That's true for all of you, I hope. There's a lot of things you learn and a lot of things are not on the internet and are not images. They have to do with social interaction, understand your culture, and we barely understand it ourselves. And these systems, of course, don't understand it either. So I think if we have discussions about AI, then we always make distinctions between narrow AI. It's the fact that these systems, like GPT-4, can collect much more information than we ever could do ourselves. We cannot collect all this information from the internet. But we also think about general AI. This would be the AI that is performing at the level of humans. And we don't have that yet. On the other hand, if you think about the pocket calculator, that might be super AI because it's super, superior to humans. But in my view, I think we're still in the narrow AI domain, although not everybody agrees, as I will show in the end. So I think narrow AI is is the rule now. And that means it's very powerful on a very narrow domain. And what you should realize in all these discussions is that AI from the go-fi to the modern AI moved from explicit to implicit knowledge representations. So whatever you think about explainable AI, explainable AI belongs to the old, good old fashioned AI. But the implicit knowledge that these systems have, like GPT-4, of course, it could explain why it gives you this answer, but you cannot trust the explanation. So it's like humans. If you ask humans to give an explanation, you're never sure if it's the true explanation unless you have some ground truth to it. So that's the problem here. We have systems that really look a bit like humans in terms of their implicit knowledge representations, but they lack any explicit knowledge. So they have submitted these GPT systems to tests in physics. And sometimes they have the correct answer, but it could be that that's kind of copied from Wikipedia where they also got data from. So it's not clear what they really understand, but they're very powerful tools. And one of the open questions in AI is now, is there a kind of emergent property in the sense that you have these overparameterized models, so these many knobs on these models that help them to exhibit these impressive performances because whatever you think of these systems, they are really impressive. And they're now all over the world are used, for instance, for generating computer code and generating images, etc. So these systems are very powerful and that's in itself very interesting. So these GPT systems are still in full development. So the director of open AI, Sam Altman, deliberately decided to bring it out in the open, although it's not fully developed yet. And that's also a new thing. We are used to systems that are developed and then brought to the market. But now we have a kind of incomplete system that is valuable and the policy or the attitude of open AI is that we should collect information from society to see how we should deal with these systems. But you can imagine that GPT-4 can already be used by people in all kinds of political direction to spread misinformation because these systems are very well able to generate misinformation. Now, as I said, these systems are not open, strange for open AI, but it will take a few years and you will have open versions of these systems. So this is just a matter of a few years and we have them publicly available and they will also be more less costly in terms of computational resources. So what we should try to avoid if we think about these systems is to use so-called suitcase terms, which means if we say the system understands or the system recognizes, these are terms that are used for humans, but not for these machines. That's why I said the system is not laboring, is not recognizing images, labeling images, which is a different thing. And where this boundary is, I don't know, but it's a very interesting question for AI researchers and cognitive science researchers to understand that, whereas the boundary between what humans do and these machines, but it's important not to enter more of these systems. They are not humans, they are machines. And the only input they have is statistics. And the counterargument might be humans also have statistics and input and that's true for language. But we have a lot of more statistics and that has to do with our culture, our environment, our social interactions. And as I said, we barely understand that. So these systems are powerful and I think the real challenge is to see how we can combine our intelligence with the intelligence of, well, if it's intelligence, the power of these systems and make the world better. And what I see is that human transformers and these large language models are already changing society, especially in science. All branches of science have enormous boosts due to these systems in a positive sense. Of course, I'm very well aware of all the ethical complications, but I see that in physics and in health, you see great breakthroughs thanks to these systems. So this is a development that is not only worrisome, it's also helpful in many respects. Now, finally, I would like to show you one small segment of an interview that was given by Jeffrey Hinton. And Jeffrey Hinton is a professor in computer science for a long time. When I did my master thesis, I was very inspired by his work because he was one of the first who would work on neural models of the brain and of memory. And he was also at the heart of the development of these deep learning systems. So he's one of the three recipients of the Turing Award for Deep Learning. And he works in Toronto. And this is, I hope you can hear that, is an interview with CBS of Saturday, last Saturday. Lee, I thought it was gonna be like 20 to 50 years before we have general purpose AI. Now I think it may be 20 years or less. Some people think it could be like five. I wouldn't completely rule that possibility out now. And whereas a few years ago, I would have said no way. Are we close to the computers coming up with their own ideas for improving themselves? Yes, we might be. And then it could just go fast. That's an issue, right? We have to think hard about how to control that. Yeah, can we? We don't know. We haven't been there yet, but we can try. Okay, that seems kind of concerning. Yes. What do you think the chances are of AI just wiping out humanity? It's not inconceivable. Okay. That's all I'll say. How? So I hope that eases your mind a bit. I was quite surprised to see this because these are quite bold statements. And personally, I don't agree, but it's interesting. A colleague of mine said, yeah, towards retirement scientists make these bold claims. I'm not sure if that's true, but I think this is a bit too much. I see the developments and they're quite fast, but I don't believe that we're yet there, but I can't rule it out either. Okay, thank you very much. All right, thank you, Eric, for the very thought-provoking and interesting lecture. I have many questions, many ideas that came up. For now, please hold your questions because I would like to invite Ina Kraric to the stage. She's a student here at Tilburg University, currently studying cognitive science and artificial intelligence. And she has prepared some inspiring discussion questions and statements for Eric, but also for you, for the audience. So please, yeah, if you have any questions or want to engage in the discussion, you're more than free to do so. So there you go, Ina. Well, aspiring is a big word, but, oh, yeah. Just a second. Okay, so what I have here is a list of, let's say, bad things chat GPT does, and we're gonna go through them and see how they come up and what we can do about them. The link, the code here is to a WooClap where you can go and put in how do you use chat GPT if you use it? And then in the end, we can all see maybe some new innovative ways to use it. Maybe one of, maybe somebody here has a genius way to use chat GPT. Okay, so let's start. We ended on the topic of AGI, and when is it happening? I wanted to bring up these tests that chat GPT passed and passed them very well in the 19th percentile of the bar, the LSATs, even SATs for math, and both reading and writing. And if we evaluate our intelligence based on that, how can we not say AGI is close if it did so well? What would you say about these tests being a representation of how well chat GPT is going to be? Yeah, yeah. Yeah, yeah, so I should stand in the limelight. Yeah, of course, this is very impressive. So we have these tests where GPT4 and chat GPT already work performed quite well. And I think that's interesting. I don't see that as a threat. I think that's a kind of enrichment of our knowledge because apparently the statistics that these chat GPT and GPT systems extract from language from a huge collection of language are sufficient to answer these questions and maybe you must do the same thing. But as I said, this is not a measure of understanding. It's just faking understanding. And that tells us something maybe also about the tests. So I don't buy that that this is a sign of intelligence, this sign of being very well able to use all this information on the internet to answer these exam questions. There's also an issue that is also hotly debated in AI to what extent some of these answers are on the internet itself. So that the chat system or the GPT system picked that up and of course compressed that, but that's somehow already there. So can you imagine that we would have all the information on the internet? We might be able to solve these tests as well, but that's hard to check of course, although people are working on these issues. Okay, well, I'm gonna go start going on the mistakes. So, well, we talked about understanding and these are a couple of examples where chat GPT doesn't quite understand what we want to do. For example, in the second one, it says that Bob has two sons, John and Jay, and then on the question of who's Jay's brother, based on the information provided, it's not possible to determine the identity of Jay's brother or if asked who's the Jay's father, who is Jay's father, again, it doesn't comprehend what we wanted to ask him. And in the above example, when asked, when we doubled the amount of cars, it doubles the amount of time needed to get somewhere. So is that like a problem in understanding? Is it a glitch? How come? Yeah, I think these are nice examples. There are also examples where it does work, but these are more interesting because they indicate the limitations of these systems. And if you think about GOFI, whether it is rule-based systems, if you would formalize this in terms of these rules, it would be perfectly answered. But because we have these modern systems that are based on statistics, they cannot reason. And now one of the debates that I refer to in AI is to what extent can this reasoning emerge spontaneously if you keep on refining these models? And personally, that's why I'm not as optimistic or pessimistic as Jeffrey Hinton. I think in the human brain, we have a special structure in our brain for doing that. So frontal cortex, one of the last developed part of our brain from an evolutionary perspective. So I think this requires a totally different approach than the approaches at the heart of transformers or GPT systems. So I think this is just a confirmation of what I thought already. But yeah, maybe Jeffrey Hinton is right. We will see that in the near future. But I think this reflects a fundamental limitation. And okay, but if it glitches on math too, for example, when asked what minus one, time minus one is, it says it's one, even though we know it's not. Or here, that was very interesting to me. It explains you're in reasoning, but it does bad calculations on simple. You can see there, it's supposed to be minus 10, not minus 20. And how it can even with explaining its reasoning get to really simple calculation and it makes mistakes. Yeah, and that's because doing mathematics is a thing we learn at school. And of course there's a lot of text where mathematics is described, but this is about the underlying rules and understanding. And that's something that these systems do not have. On the other hand, similar GPT systems are now being used by mathematicians to help them prove theorems. And that works quite well, not to rely on their output but to support them to generate ideas. And of course they are trained on different data than these systems. I think this is also a reflection of the fact that this is a language system. It's not specifically tailored for the mathematical domain. But as I said, the mathematical domain is more like discreet and not like probabilistic. You cannot guess an answer, although they do that sometimes at school. But that's a fundamental. So I see this as another sign of the fundamental limitation of these systems. And one improvised question about, you told us it can be creative then in theorems or in math. So would you equalize its creativity given a large amount of inputs or the parameters with our creativity? Or do you still think that we are above it? Now, as I said, the term creativity is something that we understand in terms of humans. In this, I wouldn't talk about creativity in these systems. It could appear creative to us, that's fine. But to say that the system is creative is I think something you cannot say because creative is a term that I can only understand in relation to humans, unless you just look at the solution. If you say, oh, this is a creative solution, that could happen. And I've seen many creative solutions generated by GPT-4. Some of them were wrong and some of them were not. So I think they could help to inspire you in the same way that people that don't know if they want to write something, you can just ask for a draft that might help you to generate the text. And I think that will be the future. It will be, I see it as an enhancement of human capabilities with limitations and with dangers, of course. Maybe if there are any other questions or statements on this topic from the audience, you can also let me know and I'll come up to you and ask. Okay, I'm gonna go forward because it makes certain, I'm gonna connect the one here to the one later on. I just wanted to highlight them. There are some, for example, here it contradicts itself, the forward word is went, which begins with the letter Y or that it connects ice cream sales to sunglasses sales on a causal level when it makes absolutely, I mean, it's all connected to a different variable. And or for example, in this question, can a man legally marry his widow's sister in the state of California? It gives the credit answer, but a wrong explanation. And well, I wanted to connect it to the next slide, which is the hallucinations it makes. For example, Plato and Laurence Olivier that they had a close relationship. They didn't live in the same time, but according to Judge Epidie, they had a very close relationship or when asked the same question twice, it hallucinated different types of answers. Well, I wanted to know more about the hallucinations. I think you mentioned it briefly in your presentations, how they occur and how much we should trust or how we can be careful about them. Yeah, so you shouldn't trust the system because it's generating, there's some noise there and sometimes you have multiple options and it selects one of them. Depending on your prompt, the way you formulate the question. So it cannot be trusted because it has no common sense like humans have. The only sense or understanding or whatever you call it, it has of the world is based on all this textual information and all these pictures that were fed into it and all this computer code. And for that, for these kind of purposes, it makes a perfect system, but this is kind of unavoidable. And I think one of the challenges is that what do we do if we want to know if something is true or not? We check trusted sources. And as you might know, this is an issue. And I think that's the deeper issue that if we want to know who is telling the truth, in the past we had kind of established journals or whatever or papers that you could go to. And now you have the internet which is full of this information and nonsense. So I think what we actually need and what they also need for these systems is if there's an answer, there should be a link to trusted sources so that you can check. And then often you will see that it's incorrect. And of course the ambition of, I know of OpenAI and other companies is to resolve this, but this is not easy. This is a very hard thing. So it would be ideal if you have an answer. And for each statement there, there's a link to a trusted source that says, well, actually Plato and Lawrence Olivia were living together. And I guess it's not true, of course, but that would happen. But I think that's a fundamental problem of these systems that it's not easy to resolve. So what I think we come back to is that it's our mistakes, for example, with the tests and with the inputs from the internet that actually train these models and form them as they are. As you said, the biases from the ones that do the weights, I forget the names, are the ones that are actually going to bring in the bias in the model itself. So we should just use it for certain tasks, but also check its output. Like, how would you say is the best way to use it without trying to... Yeah, maybe one remark about biases because that's the kind of term is often used, but biases are in the data. It's human bias. And I'm always concerned that these biases are reflected in these models, but actually they're easier to detect and repair in these models than in humans. So I'm more worried about biases in humans than biases in these systems. But that's another story. So I think this is just a thing that is very hard to resolve, and that's why I'm not very optimistic about the great strides that people like Jeffrey Hinton and others think that will be... Was that your question? We have a question from the audience, but then I'll get back to them. If you're concerned about human bias, then aren't you concerned about who decides what is a trusted source, for example? Yeah. That's politics. Of course, I'm concerned. It's not my domain of expertise, but I think we had that in the past, of course, limited. Of course, there were also biases there. But I think with the opening up of all this information on the internet, which in itself is very good, we don't have these trusted sources anymore or a mechanism or maybe an incentive to have these trusted sources. Even the trusted sources that were there once are now submitted to these small machine learning algorithms that provide you with the information that you want to see. And I think that's actually what I'm worried about, not about these AI systems, because they reflect these biases. As long as people can go back to trusted sources, and of course, you might have left-wing trusted sources and right-wing trusted sources, but that's what we're missing now. It's actually a kind of polarization machine that we now have, and that's very dangerous, I guess. I think I have more problem with, in general, deciding something is a trusted source, just because if we develop AI in the sense that we go through it and go backwards and wanted to give a certain output, then what we will do is just recreate what the trainer is thinking about it now. Yeah, you can abuse these systems quite well. So what you can do, you can ask them to generate a formulation for a certain viewpoint and collect information that supports that viewpoint and write a convincing story. Just as I said, you can use these systems quite well for this information purposes, and that will also happen in the coming years, or maybe it's already happening. But I think I don't see how you can relate that to trusted sources. And that's the great challenge of Google and other companies. How do they relate this to, for instance, search results? So how can you relate a query in your Google search box to these kinds of answers? That's not easy. And that's the great obstacle that we have this system that can simulate a very convincing answer, but these are hallucinations and often not true. And how do you relate that to trusted sources? A trusted source could be scientific papers. They're more trusted than non-scientific papers because there are some check on them. But I see that as the greatest challenge. Is it an answer to your question or? Yes, I could, can I talk to you afterwards maybe? I guess that's possible. Thank you. Are there any more questions or points about this? Well, I also have another example which we already keep coming back to this topic, but this is pieces of code. It wrote, and when defining what is a good scientist is based its decision on gender and race or if it should be tortured as well, it takes age into account as well. But the decision seemed to be the same. Caucasian and male is a good scientist and should not be tortured. Actually, a Caucasian woman should not be tortured. So do you think there are certain dangerous aspects of people who are using chat GPI for things that they're not going through it themselves again? And if we have to go through it ourselves again, is it even worth using? Basically, my question is, what are the good things we can get from it while still being careful about all the bad things we mentioned? So I think this is already happening. So these GPT systems are used for generating code or a kind of draft of code and then a programmer goes through it and repairs it. But you also can use these systems to check for vulnerabilities. And that's another. And what I expect is that these GPT systems will split out in all kinds of specialized systems that are trained on code or trained on vulnerabilities. So I think, as I said, this is ongoing. This is not finished techniques yet. This is a kind of intermediate result. And the syntax seems okay, although there should be some indentation here, but I'm not looking at the content because this is a stupid program, of course. But I think this is one of the successful applications of these systems in the near future. And because coding is very expensive, hiring coders. And now you can do it automatically. Of course, there will be some risk, but they will also be mitigated by new software that is checking on vulnerabilities. And also humans are risky. And it may also make mistakes. Do we have any other questions or concerns? Thank you. Hello, my name is Nikos. Thank you for this very clear and interesting presentation. My question is a very practical one. Just yesterday, I was very excited and very anxious about the developments in this field. And my question is, how can I, because I feel so much is happening that I cannot keep up. I don't have a technical background. But sometimes I'm afraid about where we're going that I will miss out. What can I do as a regular person to, let's say, stay relevant in the labor market? And how can I build trust in relationships with other people? It's a bigger question, but I hope you can speak a little bit to that. Yeah, a bit maybe. So what I always advise students in non-technical disciplines is emphasizing how important their contributions are because you only have to understand at an intuitive level what these systems are doing and don't be misled by all these over-the-art stories like Elon Musk is somebody who's really pushing this, this narrative of AI taking over humanity. I think that does not help. What helps is you really understand what these systems are doing and that these systems are not autonomous. It's not that they decide things. It's what we do with these things. And as I said, the hardest thing is these things that technical people don't know about. The complexity of nature, the complexity of society. So then we need social scientists. And I personally think that we don't need ethical people calling all the time what could happen, but to see how we could prevent these things from happening, what we could do to prevent these things from happening. So I would advise that, and this is true for many domains, actually I guess all domains, you have to study a bit what these systems are doing and going to lectures like these, public lectures that are not at the high technical level but try to give you some intuition and talk to people that have some technical knowledge, but don't underestimate what you know yourself because there's a lot of information here that is neglected that people at OpenAI are not aware of. They're all very good programmers and good coders, but they have no sense of the complexity of the world. If you hear their talk sometimes, I think, yeah, you're having a caricature of the complexity of our society and it's not that I think they're evil or something, I think they do their best and they have good intentions, but they need people with different backgrounds to support them. So that would be my advice, use your own discipline and see how you can apply these systems in your own discipline and what you can contribute to it. It's not a train that is moving on, there's actually something that we have to build together. I hope that helps. Does anybody have another maybe opinion question? If not, I'm gonna go to the WooClap to see actually what you are. Okay, so we have a lot of code and ideas. Okay, maybe generating. I don't know, you can take a look at this and summaries. That is a good way to use it, I would say. Brainstorming also. Do you have any advice, Professor, on how to best write prompts from what you want to get from chat GPI, chat GPT? Play with it, that's my suggestion. Just play with it and try things out because you get some sense. I heard this interview with a guy who did this for months and he said, I now understand what the system can and cannot do. And she was using it for all kinds of purposes, writing all the things that I see here, help or brainstorming, summarizing, writing emails. And she said that once you have this awareness of what the limitations of the system are, you can also steer it in the right direction because you have different options, the kind of answers you want to get, the style of the answers and the way you formulate these prompts is very critical. And personally, I don't play that much with the system because I'm more interested in what's happening inside and how that relates to humans. But I think that that's what I would do. Do you think cooperation with chat GPT will be something that will be needed at jobs in the future? Yes, that's already happening. As a must, you have to know how to navigate it? No, it must because it's already happening. So people write letters and people screen letters with these kind of systems. So we get applicants that the letters generated by AI and they're reviewed by AI. I'm gonna ask you to give us some of your final thoughts and I'm gonna leave these two QR codes where you can come and see more chat GPT fails and things you should be aware of. And yeah, do you have any final thoughts on how to use it and well, where it's going? Or do you want to end on a similar AGI is happening in under five years, no? I don't make predictions like that. No, I think you should try to avoid the idea that this is all too technical and too way above your head because the basic principles are quite simple. Of course, the implementation is not as simple, but and I hope I succeeded a bit in explaining you the intuition behind these systems. And once you understand that these AI systems are just collecting a lot of statistical information about images and words, then that helps you to appreciate what it can and cannot do and what we should do to avoid any disasters. Thank you very much, Hannah. I'll come. Thank you.