 Hello and welcome to the OIST podcast, bringing you the latest in science and tech from the Okinawa Institute of Science and Technology Graduate University. Today we speak with Professor Tom Froze. Tom is a cognitive scientist with a background in computer science and complex systems, and he's now leading the embodied cognitive science unit here at OIST. Where traditional approaches to studying the mind tend to think about the brain as an isolated computer detached from its environment. The embodied cognitive science unit thinks about cognition in an interactive way. That is, thinking about cognition as a process involving agents interacting with their environment, and studying the role of those interactions in the human mind. In this conversation we talk about the successes and limitations of thinking about brains in isolation before exploring what embodied cognitive science can reveal about the mind. We discuss what this research looks like in a laboratory setting, including what happens when human participants meet in a virtual space. Finally, we consider the implications of this work beyond academia and what it reveals about collectivist cultures like Japan. It was a great conversation, and I look forward to seeing what comes out of the unit once their new laboratory toys are up and running. Enjoy. Tom Froze, thank you for joining us today. Thank you. To begin, on a personal level, what was it that first captured your attention to this research space? When I was a teenager and I used to enjoy playing computer games, back then one of the things that frustrated me was that you could usually trick the AI in the game quite easily. And it would just show that it didn't understand really what the game was about. So I thought, well, why not try to create some better AI? And that basically got me then to university studying computer science, aeronetics, AI, robotics. Through that degree, I came to realize that actually we still don't properly understand natural intelligence either. So that kind of was like the beginning of my interest in research. And natural intelligence, before we get to the embodied cognitive science side of things, what's been the prevailing wisdom until this point? OK, well, studying the mind has a very long tradition, right? In the West, usually we think back to the ancient Greeks and Aristotle or famous philosophers like that. And so to some extent, the mind has been a topic for a long time. But in those years when I was at the university, most people thought about intelligence as manipulating symbols in some kind of clever rule-based way. And that seemed to be true for many of the things we were actually doing in class. You know, so it's funny, we were kind of describing our own research practices to some extent. We were working with number systems, doing equations and so on. But it seemed like to generalize those practices to animals and so on was a bit of a stretch. And so maybe that's where some of the problems lie with trying to generate some of the AI in a naturalistic way. And then moving on from some of those ancient philosophers into the modern day. Now we live in a world with computers. In many ways, that's shaped how we think about the mind and ourselves. And you know, in that long journey that you alluded to, one of the key figures is René Descartes, the French philosopher, who to some extent was also one of the founding fathers of a modern scientific method. And in order for him to get science to work and be accepted, he had to separate the mind from the world and put it in some other realm. You know, that was then the church business to some extent. And so actually the advances of the natural sciences was bought at the cost of losing sight of what mind could mean in such a naturalistic paradigm. And this idea that maybe we're simple manipulators or computers, we would call them now, as you mentioned, has a little bit of heritage of this idea that actually maybe what cognition is, is like, you know, the manipulation of abstract ideas, right? And so that's, you know, removed from the messy stuff that happens in the world, basically. And so the problem there is that, like I said, in the games that I was playing, used to go back to that example, it doesn't seem to be very good at creating systems that understand their interactions in the way that we would or that an animal would. And although they can appear very clever sometimes, usually they are very limited in their domain of applicability. They have to have well-defined rulesets to work with. And if you put them out of that context where quickly they drastically fail. And so that's a bit strange. So organisms are usually much more robust and adaptable than that. Yeah, if you take a human and put them into an unfamiliar environment, for instance, they're not just going to completely collapse. There's some degradation. They can bounce back, they can reorganize, they can learn, they can adapt. And of course, there are limits to this. I mean, if you just take an organism out of its niche and put it somewhere else, it'll probably not thrive as it would in its normal habitat. But let's just say some of the simple tricks that we can fool our AI with by just asking him, what is that? And it'll tell you a given. And then you add a bit of noise in there in the image. And then suddenly it will tell you that it's a nematode worm. Or I don't know, something like that where you basically see that there's no understanding on the side of the system. It's recognizing some patterns that has encountered in the past. But those patterns are totally abstract. They don't have any content. I mean, the system doesn't know what those patterns are about. They'll put it in a different way. And that seems to be the trick that life and mind has solved. We don't only see patterns, we know what those patterns are about. And so far, we have had a hard time replicating that aboutness, if you want to call it, in artificial media. So you're now attempting to move into a space where cognition is thought about in a wider sense? Yeah, that's right. So maybe the problem why we don't find understanding of patterns in artificial systems is because the cognition that they perform is so detached from the world in which they find themselves in. So we have a system that the only thing it can do is manipulate abstract symbols based on a set of rules, do some kind of information processing to put it in general terms. Then it doesn't really have any direct engagement with the world around it. It's just like locked up in some kind of digital world. And so our idea in this unit is to say, well, maybe we can do things differently if we think of cognition more in an interactive way. We think about it as an organism engaging with an environment. And then look at those interactions as that's where cognition is happening. It's not happening, as Descartes would say, in a different realm, with different substance. Now we don't agree with that anymore, but we're still putting it in this pedestal of abstract mathematical formalisms. But what if it's in the messiness of bodies and environment and stuff and materiality and historicity and all of that and maybe that's where cognition can be found. And that's very hard to replicate in AI. And so maybe there's two sides of this coin. On the one hand, it might help us to better understand natural intelligence, but also the failures of artificial intelligence. It seems like a very intuitive idea based on our own experience as people. The idea that cognition is influenced by the environment around you, the other people around you. Why do you think there's resistance to this idea? I think everybody would agree that cognition is influenced by the environment and by other people. But influence can mean many things and influence in particular can mean that it's providing just an extra source of inputs. So for example, many people think about who are agreeing with this idea of information processing. I think that culture provides just extra information to be processed. But to some extent, that means that the brain or whatever is the machinery is running these processes remains separate from and independent from these cultural processes. It's just taking the input and doing something with it, but it's not being transformed by its embeddedness in that social-cultural context. So we want to push that a little bit further. We want to say that the interactions we have with our environment are not just sources of input or external outputs produced by an internal process. But we want to say that the mind really extends into those interactions. They are expressions of mind and that's a little bit of a more radical move. So we want to say that behavior is not just a product of cognition, but is part of the process of cognition. So doing is thinking to some extent in this view. Even that seems fairly intuitive as well. I mean, if this is an example that's relevant, but spouses who've lived together for 50 years where you're quite literally outsourcing parts of your brain to somebody else. I do think that what we're saying to some extent fits much better with everyday human experience. And that's partially why I find it so attractive. You know, much has been made about this gap between science and daily life that, you know, maybe science is on an ivory tower somewhere completely locked off. And what it has to say might be a bit divorced from the experiential reality of people struggling to pay their bills. And to some extent, that's true. And if our idea of the mind is basically that it's just a number cruncher of some kind, then I'm not surprised that some people find that alienating, right? And so we're really trying to recover a more intuitive sense of what mind might mean and cash that out, not only in intuitive terms, but also in scientific terms. And the example you bring up, which is, you know, long term sustained social relationships, is perfect. So I completely agree. And actually part of the work we're going to do here is precisely to look at those interactions and try to see whether we can capture what the interactions between people do in terms of transforming them and allowing them to, as you say, outsource part of their cognitive processes into each other. And so that, you know, people will complete each other's sentences. They will know what the other is going to say before they say it. You can remind them of things that they might have forgotten. You can anticipate things in a, you know, super fluid real time manner. You don't have to consciously reflect about what they might think because you're already somehow attuned to each other. Now, if you wanted to explain all of that just by looking at an isolated brain as a, you know, information processing engine that just receives inputs and produces some outputs, it would be quite difficult. But if you instead look at the interaction as a complex system that has several components, which includes two people, which are reciprocally influencing each other, then we can imply some ideas from complex systems theory that ask what happens when we have two non-linear systems interacting. And, you know, it's not unexpected that in those situations, collective properties will emerge that somehow go beyond what the individuals bring to the situation. And that's the kind of thing that we are looking at in a more systematic way. But it wasn't too long ago that people like Sirle, who's one of the big philosophers of mind, when he writes about collective minds, collective intentionality, was writing that in principle, he could have all the social experience he has, even though he was just a brain in a vat and the entire social world was just a figment of his brain, like a massive hallucination. And so I think, and that was like in the 90s. Less people agree with that. But you can imagine that if the leading figures in the field deduce these kinds of claims from their theories, it should make us wonder about the validity of those theories. And an embodied approach is trying to correct this picture to some extent. It also feels a lot less isolated and a lot less sad, in a way. As you say, when you go in with a preconceived idea that, for instance, brains are just these isolated units performing some kind of computation, the result of that will probably be finding things that just reinforce that idea you had in the first place. But again, that doesn't seem to match human experience. Otherwise, why would we interact at all? That's true. I mean, to some extent, if you looked at isolated brains, you're going to find things that isolated brains do. And it has been fairly successful. And of course, there are also states of the brain that involve less interaction. And so that's a little bit of the challenge to our views. So what do we say about imagination, dreaming, abstract thinking? So some of these capacities will are easier to explain if you assume that it's just the brain doing this and nothing else matters. Okay, so that's also an interesting kind of challenge. But on the other hand, we also have to think that the brain has been already been a top priority for research agencies around the world with billions of dollars being invested. And we're still just to some extent mapping functionality. And it's, you know, nobody really still has a very good idea of how we go from brain activity to mind as such. And maybe part of the problem of this is that we have the wrong approach. You know, maybe we're just, if you're trying to understand the brain, but actually it's just a component of a much larger system, then obviously, a lot of the patterns that you will see in the brain will be hard to make sense of without, you know, taking into account all the other things that are going on. And that has direct implications for things like psychiatry and so on. You know, a lot of people are saying that mental disorders should be thought of as brain disorders. If you think about it that way, that immediately limits the kind of treatments that you might consider as being worthwhile pursuing, right? Then it's just a matter of trying to find the right neurotransmitters or something like that that fix the disease. And we haven't been so successful with that. I mean, depression, for example, some forms are untreatable and so on. So, so maybe that is that a signal of saying that maybe we should broaden our search, you know, maybe maybe we need to consider the mind as a more distributed system than just as a brain based system. At least, you know, there are some interesting clues and these problems that the brain centered approach is running into. So effectively what you're saying is in contrast to the idea that the brain is somehow the top of a hierarchy, you're almost kind of democratizing that hierarchy a little bit and thinking about the brain more in the context of some of these other things with equal weighting. That's right. So or maybe let's say the weighting is up for grabs, you know, and we'll have to work out how the weighting and maybe it depends on the situation, which like when you're sleeping, maybe the brain will have bigger weight in your environment, you know, I don't want to deny that. But when we're waking and we're being in interactions, maybe we should give equal weight, you know, that's that's an interesting thought. I mean, one way of making this is intuitive is that what we're studying is patterns unfolding over time and they can have different media, they can be your arms moving, they can be neurons firing, the things can change in your environment, things can move around. But really what we're studying is change over time. And then if you look into the physical world and look at how different kinds of physical relate to each other when they have different time scales, it's usually the slow ones that predominate. So when, you know, we have the tsunami warning signs all over the place, right? When the tsunami comes in, it subsumes all the smaller waves, even though they might be faster frequency waves, it will just, you know, be dominant and determining the overall shape of the water. Now, in terms of brain activity, it's super fast activity. But that means that it should be susceptible to being enslaved by slower timescale processes. And one of those is behavior, right? So the way in which you're interacting with the environment unfolds on a much slower timescale. But the impact that's happening on your brain is correspondingly much bigger. And in fact, so when we are now with experiments, we're planning with EEG, so looking at the electrical activity. To some extent, we have to be very careful how much movement we allow. Because if the participants move too much, then you get a very messy signal, right? But that's just because exactly there's a very strong link between movement and brain activity. So I'm saying maybe the hierarchy should be reversed, right? So maybe it's actually the brain is smoothing over or adapting to the bigger waves in which we find ourselves involved in. And I guess part of the appeal for historically having the brain being the focus is that it's just easy experimentally to deal with in isolation. At least if you can get people to sit still in a scanner for however long you need them to do that. That's right. With your experimental approaches that are moving beyond simply looking at the brain, how do you begin to bring in these other variables and deal with those in an experimental context? Yeah, that's a good question. And in fact, I think some of the early work was very brain centered, because as you said, they didn't have a lot of technology that would permit them to record all the other things that are happening. But nowadays, we have more and more options of actually recording what happens with the body and with respect to the environment at the same time as we're looking at the brain. And so you can think of like, you know, these suits which have different markers to record the space, you can have cameras and record the motion patterns from the video. The thing that we're doing in our unit is to draw a little bit on our computer science skills and to employ simplified virtual reality environments or human computer interfaces such that the movements and the sensations that people get are captured and given by a simple computer interface. And the advantage of doing it that way is that we can record what the people are doing and we can record what they get back, depending on how they move. And we can also manipulate their environment in which they're interacting in a very finely controlled manner. And so in that way, we can record the brain in interaction with their body and their bodies in interaction with their environment and maybe even with other people and have this kind of holistic approach to recording what goes on when people interact with each other, for example. And what have you found in those studies when you have people kind of interacting with one another in a space? Okay, so one of the exciting findings coming out of this work is that interaction makes a difference. Okay, and this is something that again might seem obvious, but like I said, most of the time interaction was just seen as an external product of cognition that's happening on the inside and was kind of like a, you know, maybe like the exhaust coming from a car, right? It has nothing to do with the functioning of the car. It's just an external output, right? So behavior was delegated a little bit in that. But what we find here is that in moments when people are interacting in a virtual space, and let's say they have a few different objects with which they can interact and they have to find out who's the other person, how do I feel the presence of the other person compared to the other objects. When they have this moment of felt presence that there's another person in the space with them, usually that moment of recognition happens at the same time for both people. So even though they're just interacting through a virtual reality interface and another person's connected somewhere else, when they actually meet at that time, they have a moment of mutual recognition that's synchronized in, you know, in the second scale. So our hypothesis is that during these moments, really they have a short moment of, hey, here we are doing something, right? And this sense of we are doing something is actually realized by both of their brains bodies in interaction. And what basis do we have for believing that the minds of those two participants are connected? That's an important question. And the way we have investigated so far is also in computational terms to get a formal understanding of the space of possible explanations. So what we have done is involve artificial agents in a similar scenario, and looked at what happens in their artificial brains, so to speak, during moments of mutual contact. And it's very interesting. What we find is that their brains start to be able to exhibit patterns of activity. They're so complex that they will be impossible for the agents to realize in isolation. So there's a kind of change in their capacities brought about through the interaction where they seem to enhance each other's degrees of freedom by going through the degrees of freedom of each that each other brings to the table, so to speak. So smashing down the ivory towers that you spoke about earlier. Once we move out of the laboratory setting, what are the implications of this work for society? And in particular, Japanese and Okinawan society where we currently find ourselves? That's a good point. Much as it's been made of the notion of a person in Japan is much more collective than in Western society. And the collective is very highly valued here in Okinawa and in Japan more general. And so I think our research to some extent supports these ideas and says, you know, this is not just a difference in opinion, perhaps that, you know, maybe you can think that collectives are important or not. But we can show in our experiments and in our computer simulations that when we interact with each other, we transform each other and we can produce collectives that to some extent have a life of their own, their dynamics go beyond what the individuals can contribute. That resonates very nicely with how people here think about society. And so I look forward to exploring these ideas more detail, but in this particular context. So what have you got coming up next? Well, life is very exciting. The reason we came here to OIST is to expand on our work and to a more experimental direction. And so we've got some equipment arriving to our unit. New toys. Exactly. New toys. Very nice toys that will allow us to study embodied real time interaction between people. And we're going to record everything. So we're going to look at brains, we're going to look at bodies, hearts, breathing rates, your, you know, how much you're sweating in your hand when you meet the other person in the virtual space. And so we're going to try to plug all of those different sensors and combine it with the virtual technology and really try to have this holistic approach that will try to do justice to the complexities of human social behavior. So before I let you go, we usually like to wrap things up with some quick fire questions. Okay. Can you explain in a few simple words what it is that you do? I'm a cognitive scientist studying the role of interactions in the human mind. Beyond your own, which emerging field in science or technology most excites you? I'm also very interested in artificial life, origins of life, and consciousness science as well. If you could wave a magic wand and erase a common misconception from everybody's collective brains relating to your research area, what would that be? I would erase people's beliefs that they're nothing but their brains. Is there a tool that you use in your work that you're particularly fond of using? Slack. I found that Slack allows us to interact on a much more frequent and personal basis than sending emails, which can sometimes have a too formal taste to it somehow. Do you have a favorite science related joke or fact for us? Science related joke or fact. Well, what about this one? Okay. So sometimes the same data can produce completely opposite theories. And I'll give one example. In the history of life, evolution, the record looks like there's a lot of stable forms and then there's a rapid transition to another stable form. And a lot of times people thought, well, it's just because we haven't found all the fossils, you know? So there should be lots of fossils in the middle of all the transitory forms. Until someone came around and said, well, actually, maybe what's happening is that there's a lot of stasis interspace with rapid change. And there is no continuous slow change. So they were looking at the same fossil records, but they were giving exactly opposite interpretations. So I think we should always keep that in mind when we have our own chair of scientific assumptions. Tom Fros, thank you so much for the time. Thank you for having me. Thank you for listening to the OIST podcast. If you enjoyed this episode, please remember to subscribe, leave a review, and share it with others who you think will enjoy it. You can also get in touch with us on Facebook and Twitter, or by sending an email to media at OIST.JP. Thanks again for listening. We'll see you next time.