 David is assistant professor in the Department of Cognition and Communication in Tollwork and his research also concerns research reality. So today we're fortunate because we have two speakers who do research in the same environment or with very different research. Because what David does is he conducts experiments in combination with other techniques to extract information about the user of virtual reality applications from motion tracking, eye tracking, but also fmi and EEG, so looking also at brain signals. He's interested in, for example, various non-verbal and verbal cues in communication. I believe that this is primary focus and he's actually quite famous. He's so famous that there is a particular group of replication studies that are referred to as paid rep. So if you're interested in that, I suggest you have a look under this hashtag. David has been very successful in obtaining funding for his research. He is, for instance, he has conducted his research in the context of a Lainey grant and he's also a member of the Tollwork Young Academy, which is an independent platform for talented young researchers here in Tollwork. And in that context, you came up with a campaign for priceless ideas spelled with a Z. David, we would be happy to hear a little bit more about that. And now what I wanted to share about David is that he's very much against rankings and he's been quite vocal about that and not just university rankings, evaluating institutions, but also scientific rankings, evaluating us as researchers. And again, I invite you to share your thoughts about it with us today. Perhaps before or after you talk about your own experimental work. David, take care. Thank you very much for this very nice introduction. Let me maybe first talk about the science and then if there's some time left. Let's also go into university rankings and competition and all sorts of environmental factors that could be quite detrimental for our health in academia. So in terms of the science, I'll talk about VR as a tool to study psychology of language and communication. If you think of the world as it is today, you may be quite tempted to look for a different reality. And virtual reality to some extent can offer a different reality. You place this headset on your head. You no longer experience the real world, but also there's this digital environment that sort of immerses you into a different environment that can be tailored to your own preferences. VR is not just used as we have seen before today for distraction and entertainment purposes only. But it's also being studied, for instance, in the social sciences, for instance, psychology, science, communication sciences. And you could distinguish basically three ways in which virtual reality is used in social sciences. It has been used as an object of study itself. So you can basically study as a researcher how people experience VR. For instance, you could study how different variables influence whether people have a nice experience in a virtual environment, how this compares to their behavior, their experiences. With the real world, you could study what factors this contributes to whether people become cyber sick or not. So VR here is really the object of study itself. Since we heard about this before already today, this is not what I will be talking about. A second way in which VR has been studied has been used in the social sciences as an application. So VR can be used to change people's behavior in the real world. For instance, there are lots of applications that could teach you a second language, a foreign language, in a foreign digital environment, for instance. So you can practice in this foreign language whenever you like. Just place this headset onto your head. Get immersed in this foreign environment and learn a language there. Of course, it can be studied how successful those applications are, what factors make for a successful application. Another example here is that there are many applications that look at phobia treatment. So people are afraid of all sorts of things, spiders, big rooms, height. There are now many types of exposure therapy in virtual reality. And of course, you could study how successful those kinds of applications are in treating people's phobia. But this is also not something I'll be talking about. I'll be talking about VR as a method. So really the study of real-world phenomena through a virtual simulation. And often in those cases, there are fundamental curiosity-driven scientific questions that we're trying to answer using VR. So you could answer those questions with a variety of different methods. And VR is a relatively novel way of doing so, especially in the study of language and communication. So in this more specific field in the social sciences, so the psychology of language, the psychology of communication, we're typically interested in those situations here. So people are communicating as humans. We are capable of communicating using speech, using hand gestures, using facial expressions. Also, as signals, we use to communicate. And in this field of study, we typically study what kind of cognitive processes allow us to communicate. But the mental operations, what kind of mental representations allow us as humans to communicate. And there are many different, we think, important fundamental questions that are being tried to be answered in this field. For instance, as I said, what kind of cognitive mechanisms allow us to produce language, allow us to comprehend or understand language. Also another quite odd question, a topic at the moment, is to what extent do we predict upcoming words when we listen to someone? So if you're listening to me, are you perhaps actively predicting, without even realizing it, the sort of words that I will be saying in a bit? And maybe that will allow you to more easily understand what I'm saying. There are different questions like how do people integrate verbal and nonverbal signals? When you're listening to me, you hear speech, but there are hand gestures that I'm making. I have a certain facial expression. My body is moving. If you really want to grasp what I'm communicating, maybe you should integrate, combine all those different signals to really get at the message that I'm conveying. How does this work? What kind of cognitive mechanisms allow us to do this? I get another question, for instance, in the field, in a more specific field of bilingualism. How do bilinguals manage to switch from one language to another? If you know, if you must, or several languages, sometimes you're speaking one language, that one language, at another moment you're speaking another language. How do you manage to express your thoughts in one language or another? To the extent that this is context, to the extent that the person that's listening to you will really understand you. So just a couple of questions here that are being studied in the study of the psychology language, in many different states, in many different ways. In this field, what we're typically trying to do is we try to build theories, we try to build cognitive models of those processes that are taking place in the mind, in the brain of people when they're communicating. You see a conceptual framework of many of the processes that are going on when people are communicating. We try to really get a better understanding of what happens in their lives. Of course, and also we can look at their brains. We can scan their brains while their friends are producing speech, while they're listening to somebody to see what regions, what networks of brain regions are involved, when we're communicating, when we're talking, when we're listening, when we're writing, when we're reading, really trying to get a better fundamental understanding of what kind of brain networks allow us to communicate as humans. Many of those questions that I was talking about, and many of those theoretical models, neural models that we have, are actually based on very standard psychology experiments. And those experiments typically look a bit like this. So you may have participated in those yourself. You're invited to come to a certain psychology lab. You're placed inside a soundproof booth, typically in front of a computer screen, and you carry out a certain task in this lab environment, by yourself, sitting there by yourself, and data is being collected. So your responses are collected, your response times, perhaps your eye movements can be recorded, your brain activity can be recorded while you're performing this computer task. But typically, those computer tasks are quite static and repetitive. And typically, they are non-communicative and that you're sitting there by yourself, just doing a simple task on a computer, stimuli are presented in 2D. And those experiments in the psychology of language have often been so-called uni-modal. So there's just auditory information like speech, or just visual information presented on a screen. As you can imagine, those kind of lab setups, those lab situations, are nothing like the situations in which we use language and communication in everyday life. So those situations look like this, right? So they're typically very rich in dynamic situations. Their situations are communicative. They're in 3D. There's auditory information and visual information all at the same time. Communication is multimodal. We use speech as one modality. We use hand gestures as a visual modality. We use facial expressions. So it's all there at the same time. Very different from those very isolated lab experiments that are being conducted to learn more about actually, hopefully, what's going on in everyday labs. So there's quite a gap between what we're actually interested in, which is everyday. In the end, we're interested in building theories of this everyday situation where people use language where they communicate. There's quite a gap with sort of situations in which we test our theories, because those are quite isolated static, repetitive lab kind of sets. Of course, over the past decade or so, more and more researchers are starting to realize that this may be problematic. So people have said, for instance, extrapolating those findings from those restrained lab environments to real life may be much less obvious than we previously thought. Well, a few of language comprehension is situated in a rich context. Seems very plausible and appealing. This is not what has shaped our theories. Our theories are not based on those rich dynamic setups. They're actually based on more static lab experiments. So maybe our theories don't really generalize to the thing we're actually interested in, which is people's everyday behavior. And some researchers have even said what we're actually doing, what we have been doing is passive spectator science. So those participants, they come to the lab, but they're just being passive spectators doing quite repetitive computer tasks. First, of course, in everyday life when we're communicating, we're not passive spectators. We're actually active doing all sorts of things at the same time. Very different from this lab set. Now you could say, why are you then doing those experiments? Why have you been doing them? For the typical answer that you will get as well, we need to collect reliable data. Therefore, we need a lot of experimental control. So we need to have full control over what people are doing in the lab, such that our data are very reliable. For instance, such that we can isolate one specific cognitive process and see what role this plays in the bigger role of human communication. If we would do some of this experimental control, we could make those experiments a bit more ecologically valid. So they might look more like the things we're actually interested in everyday situations. Actually, we can do that because we actually need the experimental control to collect reliable eye movement data or reliable reaction time. There shouldn't be too much noise in the paradigm that we're using. So for decades, I think this would have been the standard answer that you would have received if you would ask a researcher in this field, why are you then doing those experiments if they don't really resemble everyday life? But I actually think what virtual reality is showing is that this continuum with those two constructs at those two ends is actually false. Rather, it feels like they should be two different orthogonal factors, such that it should be possible to combine eye ecological capability with eye experimental control. I think virtual reality, if it's used properly, can be used to sort of have the best of both worlds and combine those two constructs in a nice way. Back to those questions, let me illustrate this more abstract point by focusing on one of those questions, namely the last one. How do bilinguals manage to switch from one language to the other? How do they manage to switch between the languages that they... In a traditional study, a question like this, actually this question, would be answered by getting some participants into the lab. For instance, in a classic study, they had 16 unbalanced bilinguals come through the app. Unbalanced bilinguals are bilinguals that have one dominant native language, and at some point, they also learn the second language, but they're more dominant in the language that they had learned for a longer time. So, for instance, for many of those, we have a certain native language. At some point, we also started learning English as a second language, and in this field, then you're also called a bilingual or probably unbalanced bilingual, because you're probably a bit more proficient in your native language than in your second language language. Well, those participants were placed in front of a computer screen, and there they were asked to perform a very simple task, namely, name the numerals that you see on the screen, either in your first language, your native language, or in your second language, as a function of a color cube. So, if there's the blue color cube, blue rectangle around this numeral, name it in your first language in L1. If there's a yellow color cube around this numeral, name it in your second language. So, if your first language is Dutch, in this example, first you would see the numeral 4, so you would name it in your native language, Dutch, so you would say 4. Next trial, again, it would be blue frame, so here you would say nege, but then on the next trial, there would be a yellow cube, so there you would use your second language, English, so you would say 5. So, what you then get in this kind of setup is so-called non-switch trials, where you use the same language twice in a room, or there could be a switch from your first language to your second language, or the other way around, a switch from your second language. So, your first language, of course, that could also be non-switch trials in your second. So, this is very classic two-by-two design. If you have people who perform this task, you can measure their response times. Does it take longer for you to switch languages, for instance, compared to when you don't switch languages? Is it more difficult to switch from your native language to your second language compared to switching the other way around? It's a very clean design, very clean data, reaction times. You could also perform this task while collecting people's brain activity, for instance, via EEG, and then afterwards, as a researcher, you can analyze whether switching comes at a cost, whether switching from one language to the other comes at a higher cost than switching the other way around. And we sort of can build theories based on those findings from those computer tasks, from those isolated environments where people are naming those numerals in front of a computer screen into a microphone. One more recent way of answering those questions is by using virtual reality. And a nice advantage of virtual reality is that now participants no longer have to speak for themselves into a microphone in an isolated environment, speaking basically for nobody, for a recording. In virtual reality, you can actually have them speak to virtual, life-size listeners. So here you see a cave environment where an experimental participant meets a virtual agent, life-size, and this participant can speak in one language or another to this virtual agent. And again, we can measure things like response times. As you can see here in the picture, we can also measure things like EEG, so they can wear this EEG cap and we can record their brain activity while they're performing this task. Or we can have them meet several virtual agents, several virtual listeners at the same time and have them switch from one virtual listener to another within the same language or across different languages. Those virtual agents can introduce themselves to the participants, explain what language they actually master, and then if one virtual agent, one virtual listener looks at the participant, they may respond in one language. If another virtual agent looks at them, they may respond in a different language and again we can measure in a bit more naturalistic environment in 3D whether this language switching comes at a cost or not. We can even take this to the next level by having participants become immersed in again this cave environment where we can tell them for now on in this environment, you're a shop owner, you own a market stand in this marketplace, and please speak to the people that come up to you in the language that they master. So before the experiment, we tell them there are those market visitors who speak, for instance, Dutch, but those others only understand English. And again, we can switch between languages in an environment that looks much more like an everyday environment compared to those previous computer tasks that researchers have been using. And our participants may look like this, so they're wearing those 3D glasses, such as everything that you just saw in 2D. They will see in 3D, so they really have the feeling that they're immersed in this virtual environment. And as you can see here, we can record things like brain movements, but also eye tracking, eye movements, and also response times, if we would like to. One final thing related to language switching is that since we started using this relatively new methods, virtual reality, we also started asking new questions. In this previous traditional computer task approach, participants would simply be speaking into a microphone for nobody. When we tried to build a virtual reality equivalent of this experimental paradigm, we realized it would be good to actually have participants speak to somebody as they would do in natural environments. So a new thing we now all of a sudden can measure is, for instance, whether it takes longer to switch between different listeners compared to to switch between different listeners, and at the same time also switch between languages. So there are those new questions popping up. And when we look at brain activity recorded while people switch between listeners and between languages, we also find new findings, new results. So here, for instance, one thing we found is that there are very large similarities if you look at the brain activity that you find when people switch listeners within the same language. So switching listeners, switching from one listener to another, comes at a certain cost at the brain level. It actually looks very similar to when people are switching between the different languages that they know. So apparently, there should be some quite generic switching mechanism that's used, that is subserving, switching between different people, and also switching between different languages. All right, so here's the take home message to close this presentation. I hope to have convinced you that ecological validity and experimental control are not necessarily two extremes on a continuum. You can have the best of both worlds, high experimental control and high ecological validity. For instance, by using immersive virtual reality. It offers unique opportunities for fundamental experimental research. And in this field, in the psychology and study of the psychology of language, I think it's really leading to a shift of focus towards really interaction between different communicative modalities, speech, gesture, eye gaze, facial expressions, and also towards interaction with the non-linguistic environment. Because now all of a sudden, people are immersed in this non-linguistic rich environment that was not there in this experimental lab setup when they were just sitting in front of an interview. This presentation has been quite methodological in focus. Since we have a quite broad audience, I thought it would be good to not go too much into very specifically linguistic theories. But if you're interested in those, of course, there are many papers you can read about this, for instance, via my website. And also feel free to reach out if you have any questions or suggestions. So thank you very much for your attention. Thank you, David. So I have a question that kind of follows your talk. And I think we would all agree that ecological validity is a good thing in our experimental studies, particularly if they concern communication. Can you give an example of an experimental result where you really saw the impact, for instance, in terms of neurological data, of conducting the same kind of study but translated into a virtual reality setting where you could simulate more truthfully the natural environment that you communicate with? Yeah. Yeah, very good question. I think in general, before I give my actual answer, it is good to mention that ecological validity to some extent is always a sort of assumption. We assume that the immersive VR setup that we have is more ecologically valid than this kind of computer setup. But it's hard to actually test this very thoroughly, which ideally you would want to directly compare to the real world. But if that would have been possible, we could have just as well done the experiment in the real world. So we wouldn't have needed VR. I think in general, a nice approach is to look at the phenomenon, the real-life phenomenon that you're interested in. For instance, just make a simple list of really the basic elements of this everyday environment, everyday behavior that are really central to what you're interested in. And then have as many aspects of that list in your experimental setup as possible to mimic it as closely as possible. Of course, the risk is, and I'm actually getting to the answer to your question, is that you set up this whole VR study this VR experiment which takes more time, more resources than a standard computer task. And then you find the same thing. And then on the one hand, you could say this is nice because it sort of confronts the theories that we have in a richer way. But on the other hand, you could also say, well, maybe it was a bit in vain then that you put all this effort into this VR experiment that took so much time to set up, et cetera. I think the most promising way of, if I look at the studies I've conducted in VR so far, of using VR is to actually test things that are hard to test in traditional computer experiments and really get at new questions. So for instance, one study I didn't talk about, but that we also conducted is a study into so-called indirect requests. So in everyday communication, we often say things with very indirect ways. So for instance, you can imagine that if you're sitting in a restaurant and your soup is cold, you want some new soup, you could say in a very indirect way to start on that later. My soup is cold. So if you don't mean to say it's your soup, what you mean to say is get me some new soup. It's a very indirect way of communicating. Those kind of situations were very hard to test in traditional experiments. What we can do now is we build a virtual restaurant, we tell participants you're now a staff member in this restaurant, you will meet certain visitors in this restaurant and go speak to them. And then the visitors, virtual visitors, say things like my soup is cold, and we can then measure what happens in people's brains while they listen to this. So I think the strongest way of using VR is really getting into old questions in a nice and new way or asking completely new questions that sort of pop up that weren't even asked before. Thank you. We have a little bit of time for more questions. So what I was interested in is you now link VR to many different things. I also understood that you are researching the links between virtual reality and arts. Can you say a little bit about that? It interests me enormously. Yeah, excuse me, say a bit about that. Yeah, so what I've been talking about is mostly the fundamental studies. So big questions about human cognition and communication. It's not so applied to the things we find and not directly for society. Or we don't build based on those studies, we don't, in those studies, we don't build applications that a broad audience can use. That's different for what you're referring to, which is this, we are now also collaborating with Princess with the next door library, the local library in Tilburg and also the National Royal Library. And then we try to have a broad audience, become more interested in language in general and literature and poetry more specifically by offering them experiences where they almost literally step into a poem. So you can imagine that typically, if you would read a novel or read a poem, you would do so from a printed book or on a screen perhaps. What VR allows is now to build in a very nice 3D virtual environment that people can get immersed in. And if you really can more or less literally step into a poem. So they would listen to a poem and at the same time see a 3D environment that more or less matches what they are listening to. And we've been testing this with library visitors, having them listen to poetry in those 3D virtual environments that create a sort of aesthetic experience, combining visual with auditory information. And in general, they seem to like this quite a bit. So we're also looking into whether we can use these secondary schools to induce students into secondary schools for language and literature and literature education a bit more than traditional education programs can offer them. So I think this is a very nice and exciting development that's also in the field of literature, literature, poetry, and are those new works in which all sorts of different realities are being combined to really get people interested in, for instance, language, literature, aesthetic ways of using language to impress oneself. Thank you, David. So I have a question actually for both of our speakers. And that is a question that's based on anecdotal evidence. We sometimes notice in our experiments that kids seem to suffer less from cyber sickness than adults. Now, you're an expert on cyber sickness, Phil, and now it works with different age groups. Can you confirm this? And if so, is that a good explanation for that? Do you want to start? I think specifically from the work that I've looked at, it's not necessarily just younger age groups, but also older age groups as well. You find this great middle ground with people, maybe all of our ages in here, where we will actually get sick. Again, it'll be completely dependent, but you can get sick actually a lot easier. But then you'll find it anecdotally, then with much younger people, they will actually be able to withstand it much more from the work that I found. I'm not sure if you found anything similar or if you found the opposite. In general, I've been very careful in working with kids in really immersive environments. So there are some recommendations from the financial VR lab in California who say, don't immerse young people in VR environments for more minutes than their age in years. So for instance, if you're five years old, don't immerse them for more than five minutes at a time in a very rich treaty, virtual environment. So in that sense, I've been a bit reluctant to really take VR into very young populations. I just feel like there are large individual differences. So in the work I've been doing, we have typically been testing, pre-testing all our simulations on a staff member who already gets sick when sitting on a bus. We like to really easily get sick. And then if he manages to survive the virtual experience, then we're sort of sure that most people will actually be able to deal with it. And on the other hand, there seem to be people who can read static text in VR and move around and everything's moving while they're not moving and they're moving one way, but different jobs. I was not adapting to where you're looking and there's no issues at all. So I feel like within different age groups there, in general, a huge individual difference system of people responding to this. And I think one thing further, I can add to that as well. In the literature that I've read specifically with younger age groups, a lot of it's been within clinical environments, but you'll find a much higher uptake of people willing to actually bypass that phase of the symptoms beside the sickness and motion sickness because of the benefit which they're receiving. Again, this is normally from more of a clinical medical sort of side of things. So you can imagine that if there is an escape to something from the sort of bad thing that's happening to them, they're much more willing to endure that side of things. But as well, and again, I'm not too experienced with this, but I'd imagine especially for the measurement side of things, trying to indicate how a young child is feeling, for instance, compared to an adult. An example, with pain, use completely different pain scales. You would use ones with smiley faces on them and ask them to indicate where on those scale they feel. And I imagine you'd have to give them your approach especially from a psychoscientist perspective. So it would be certainly interesting to explore further. So either of you suffer from psychoscientism? I don't personally, but you should ask Wendy Powell when you next see how she suffers. She's absolutely dreadful. I can say that. I also don't know. Yeah, I typically like to see whether I can break the system. So if there's a virtual wall to walk through it or if there's a virtual cliff to jump down. So far, no sickness experience. I'm not perfect. I never have to necessarily come out of it, but I can certainly see what people experience. Yeah. What do you think is the limit for the amount of time that people can spend in DR for experiencing negative effects? To pretend that I think on what it is. There's a game that came out about a year ago called Half-Life Alyx, which is they've got lots of different sort of movement techniques in there where you can you can travel constantly or you there's like a blink movement system where you can move in steps almost and they will close the eyes almost and mimic that. I was able to play that for hours, for instance, as we'll go. My partner, for instance, she did it for 20 minutes. So it is very tech. Awesome. So when you're using a VR headset, this facial input is really, really in front of your eyes, right? I think biologically we're not really used to always having this visual right in front of your eyes. We also use it every now and then. Look at the distance at things. So I also noticed this is a large individual difference. I also know some elderly people actually who managed gaming in VR. So they retired and then they started gaming in VR. You can do this for hours. Wearing a headset. I'm not sure whether it's, yeah, whether it's a very healthy thing to do. Try to stay on the safe side and take a break every now and then. Of course, for many of us, we're already looking at the screen for most of our day. So especially during the pandemic, I felt like at the end of the day, I just don't want to do it anymore because during the entire day, I was already looking at the screen. I don't want it in my free time and also have some visual input right in front of me. So yeah, it's maybe during work times. So it's kind of, she's doing that. I think we have to close down on this, although we could probably talk about VR for a lot longer. The next edition of the Titanic City Talks will be the 21st of April. So we hope that many of you will be understanding. Anything we need to add? No, just thank you to the speakers for my interesting talks and thank you to the audience for coming to us. Thank you.