 Hello and welcome, everyone. This is Actinflab Livestream number 35.1. It's January 5th, 2022. Welcome to the Active Inference Lab. We are a participatory online lab that is communicating, learning, and practicing applied active inference. You can find us at the links here on this slide. This is a recorded and an archived livestream, so please provide us with feedback so that we can improve our work. The backgrounds and perspectives are welcome here, and we'll be following good video etiquette for livestreams plus playing with our new affordance, thanks to our jitsi repair person, and that'll be fun. Go to ActiveInference.org if you want to learn more about how to participate or get involved. If you're listening to this livestream, live or in replay, allocating your regime of attention this way, then maybe you'd like to get involved a little bit more actively with our projects, including planning the livestream. So today, we are in Active Stream 35.1. We are discussing the paper that Dean and Blue and I did 35.0 on yesterday, and that paper is A Tale of Two Architectures, Free Energy, Its Models, and Modularity by Majeed Benny in December 2021. And we're going to be talking, modifying the slides, asking questions, going into some things that we brought up in the .0, connecting a few dots, and also preparing for, hopefully, Majeed to join in the .2 video. We're going to be discussing this again next week at the same time. So let's begin with some introduction and warm-up. We will each go around and just say hello and introduce ourselves, as well as maybe just add something that we're curious about, what got us really excited to discuss this paper, and what are we looking forward to reducing our uncertainty about in the coming days and weeks. So I'll start. I'm Daniel. I'm a researcher in California, and I think I'm excited to talk about the information encapsulation and what that means, as well as to go into that parable of Alice and Bob and see who here wants to relay that story to us and shine some light on narrative-based understandings of free energy principle. And I'll pass it to Steven. Thank you. Yeah, I'm Steven. I'm here in Toronto. I'm really interested in this ability to step back and look at some of the underlying processes of actually modeling that this talks about and the philosophies of that in quite a practical way. So I'm excited by that potential from this paper. And also what the group here is making of that, because I think it's challenging us to start to think about some foundational stuff in our work. So I'm going to pass this over to Dave. We're going to just skip Dave for some audio. Oh, is Dean around? Yeah. I'm Dean. I'm in Calgary. For me, it's always trying to be able to hold up two things at once. And so for those two things in this paper or the moment of instantiation or the sort of the model slice aspect of what we're looking at and then also to instantiator to modulate. And so those two things together, what does that imply? And that's what I want to talk about a little bit today with Bob and Alice and I'll pass it to Blue. Hi, I'm Blue. I'm a research consultant in New Mexico. And something I'm excited to talk about today is or maybe next week is the idea of modularity in the free energy principle and how that might work in the absence of a brain, where the FEP still applies to brainless systems and how that might be also modular or not. Okay. So there's so many places to jump in, but thankfully we have some questions in the live chat, which we'll begin with. And then as per usual, we will modify our slides and just keep on adding more slides, adding more questions. So if you're watching live, thanks again for all these great questions and I'll first bring them into this slide. So Joseph Clark asks, this is a little bit of a general question and then we'll move to a question that's a bit more specific to the paper. So the general question is there a space on either the discord or live streams to discuss active inference in reference to more traditional, philosophical, psychological, linguistic or social science questions? What would anyone say on that? Well, I suppose that we do have some of those conversations as part of the active inference lab. Some of the social science questions, it can get in the weeds with too many different assumptions in the background, which they themselves could be contentious. So I think it's helpful to do some of the type of work that we're doing now to sort of think about, well, what does it actually mean to model something? What are the sort of assumptions that are even going on before we start to dive into broad sort of popular theoretical frameworks because are we going to come at this from the position of a very low-dimensional modeling approach, which is shed light onto things like autism, things like schizophrenia, things like types of behavior, or are we going to come at this more from a structured, principled approach where we have an ecological niche relating to an agent and a generative model. Some of those things I think are more helpful to start to pass so that we know how we're locating the discussion. But I'm not sure if others agree with that because it's very easy to get caught up in a discussion which is essentially located in, say, the social sciences and trying to work from there with all the implicit assumptions and get back to active inference and it can get quite heady and kind of confusing. It's maybe better work in the other way addressing something from, okay, how do we take a type of modeling approach used around FEP active inference in relation to something? What's useful from that? But that's my feeling and I also believe it's nice to do stuff where you're doing interactive dialogue and working with documents in real time as much as it is through text chat because I think that also can start to get, it can't quite handle the level of complexity at play. Thanks, Steven. So I would push back on Joseph for a specific example of a question that might fall into this category and also, is there a particular paper that might be related to active inference in this field? Because I do feel like we do discuss a lot of philosophical and social science questions. I'm not sure. I guess there's some linguistics also. I can't remember like a stream. It's just not like linking in my brain. But I mean, we just did thinking like a state which is very social science based, I think. And so yeah, I would just love to see what particular questions you would like to see addressed. And if you can find a paper and recommend it that we discuss on the live stream, more than happy to accommodate that. Yeah. Yep. The linguistics guest stream was Elliot Murphy. So I'll respond in terms of affordances. I think there are probably more. So create your own affordance. Modify the acting flab as your niche as you see fit. But there's always the informal discord affordance. Just post in general and say, hey, everyone, I'm going to be curious about these ideas. I don't even know what to know. But I'm going to be curious about these different things, these topics, these three questions. And in two hours, I'll be doing it or in two weeks at this time. So you can always just say when you're going to be in the voice chat on discord. And maybe some people will drop in or you can invite some friends specifically. And then a little bit more formal of an affordance is we can host a guest stream or a panel. So that would be done through the.com's organizational unit. And we would work to plan the timing and then also set up the discussion. So it's not just like, we show up and it's like, okay, well, who is going to prepare? Like we can just to whatever extent is required, get a jam board ready, get some slides ready, some questions that will spark the conversation. And that can also invite people outside of the active community. So it can be whatever you like. So by asking the question, you've signaled that you understand that such an affordance could be important. And so looking forward to seeing how you follow through. So let's go to Joseph's second question, which brings us, I think a little bit closer towards the paper. It will zoom in towards it. So how do you stop overfitting on one layer of the model? For example, leg movements to avoid compromising the larger goal that an agent or person wants to do like kicking a football. I feel this question may also help put some of the debates around modularity versus hierarchical information segregation in an analogous context. So it's a great question. We talked a lot about hierarchical modeling and about that lateral modularity, kind of the collective behavior element and then the nested modularity, which is the multi-scale systems perspective and in active inference, as we explored with the mental action paper and deep active inference, we sometimes have these nested models that are expanding over multiple timescales so that the action at one time step isn't just greedily pursuing the overall preference, but rather the preference can be realized or uncertainty about realizing the preference can be achieved through policy selection over a given time horizon even when the micro movements might not be intuitive. So sort of taking one step back so that you can go a better path. What would anybody add to that or we can start to diagram some pieces out here? I would just say don't drop one or the other. Don't drop the idea of the moment of instantiation or that model slice and don't drop the two instantiator to modulate. Make sure that, like I've used the expression when in don't zoom in, zoom out. When you're zooming out, you're actually trying to incorporate longer timeframes versus a moment in time and that's how I would mitigate some. I don't know that you can wipe out the overfitting problem. There's always a potential for that, but you can mitigate it somewhat by making sure that you keep the two ideas up at the same time without one blending into the other or a total focus on one or the other. I threw up an image from paper 25 which was the mental action paper of Sanved Smith and this is kind of one way to address it. Let's just say that the overall goal were to kick the football. It's a good example. It works for whichever side of the pond you're on with kicking the football. The overall preference would be for, for example, the sensory observation of seeing the football far away or of observing the foot making contact with the football at high velocity and the foot is starting in a neutral position. So just sort of like standing and the greedy gradient descender would just go from the neutral position to just kick forward. But maybe it's possible there's like a three step movement. For example, in the first time step you pull the leg back and then you move it forward two times. So over three clicks of time in the model that is going to be the outcome that ultimately best realizes your preferences of making contact with your foot on the ball at the maximum possible velocity. And so the policy selection is happening at the motor level. So the policy here is a parameter that's describing like the angle of the hip. That is the state that's being inferred. So we won't go too deep into the skilled performance angle or the interactionism angle, but 23, check it out. And two policies can be compared or a whole family of policies can be compared with one another. One policy is just move the leg forward. The other policy is move leg backwards then forward twice. Another policy would be like move it backwards then forwards and backwards and forwards and never kick the ball. Then over a given time horizon like the time that the organism has to make this kick different expectations can be calculated for different policy sequences. And again, if it turns out that moving the leg back one step and then forward two steps makes contact with the ball at higher velocity, then at the level of deep policy selection that three step sequence will be selected over the just move leg forward sequence or the back and forth not kicking sequence. So that's a mechanical version. And it's also the kind of model structure that applies to not just motor actions, but mental action. So I hope that gets a little bit towards it and also starts us towards this discussion of modularity because where is the modularity in what we just described? So Stephen and then anyone else who raises their hand or throws out a string of emojis. Yeah, this is a very useful example. A, because you see how it's about phenomenology and it's interesting because traditionally philosophical psychological perspectival type approaches averaging out over social approaches aren't phenomenological. And phenomenological ties in more with our best guess approach of how an agent is experiencing things. So in what you just mentioned here, I think this is quite good for the kick in the football is I'm trying to work out how to kick the football. What am I trying to do as it's a policy selection? I'm trying to work out the best way to do it. And I may be unaware of what I'm paying attention to in trying to help do that. There might be some subconscious processes which are helping to calibrate it. However, once I've actually mastered that performance, what I'm paying attention to may not be how to do it. I might be in the flow and now what am I paying attention to is how do I look up and just kick the ball towards another player? So I think that I'm wondering what people's thoughts are there in terms of how that changes things to now being an agent based mode. Good question. I think it also relates to the event based cognition. For example, Martin Boots and what he talked about. The micro level is indeed the policy of each muscle and joint being selected to write something. However, there's a modularity that arises from cognitive chunking of complex motor actions, for example, into broader, more handleable units. So like writing a letter is like a unit of motor actions. It's actually a really complex sequence of motor actions. And then writing a word is a given event that subsumes or entails or requires or is supervening upon lower level policy selections, like which way to move a muscle of a thumb. However, it's the increasingly chunked and modular, perhaps even semantic levels that appear to arise more to our phenomenology, which is why we, for example, reach for something and a whole host of processes play out without us needing to micro manage the motor actions. Yes, Dean? Oh, I was just kind of... I think what, to Steven's point, I think what we are asking here is, does the mind or can the mind go into directions? Namely, do we start with a frame or an architectural scaffold or a model? Or does the process of what we're watching, i.e. kicking a ball, spectralize the relativity, i.e. does it confirm what the edges are? So those are different, completely opposite views of things over a time scale. So then, I guess what we have to ask ourselves is, does the representation act as the takeoff point, meaning do we start with the model? Do we start with that scientific view and or does the representation evolve as the results, sort of more the creative view? I think we're capable of doing both. In fact, there's lots of evidence that we do do both. So I would come back to... Do we modulate? Yes. Is that a process? Yes. Can we take slices of a modulation, exercise and see them as representational and modeling? Yes. So that's all I would point to. I don't know when the transition is, Stephen, when it's kicking a ball. Thanks, Stephen. And then, anyone else? Yeah, with this transition, this may be part of what changes. And I suppose this is one of the interesting things, it depends on the nature of the situation. But everything at some level which is interesting is hidden in active inference and we're trying to get some purchase on that which is interesting as opposed to it being out there as knowledge and we somehow know it. It's like, okay, what am I perceiving? Will there always be more hidden states than I can ever really know? What am I trying to do? There will be more hidden states. I'm trying to gain attention to. And I think what's quite interesting is what is at the realm of being available to us in our consciousness? Okay, so the sort of thing that Daniel was talking about, the sort of process, the sense making process which is present when it's not immediately obvious how to kick a ball because Messi doesn't do that. Messi knows he just, he feels it, right? There's something really interesting about the scale and I think my feeling is we have a meso-middle scale. There's things which come out of our awareness or are harder to purchase or get a grip on as they become faster and as they become slower. So what am I paying attention to? There may be something that my unconscious is sort of aware of, that phenomenological feeling about how the events is going and how my stomach is feeling waiting for the next meal, which is sort of too slow for me to really notice. And there's other things, like we're saying, which are just biologically at speeds faster than the rates at which conscious awareness operates. And we've sort of got both of these and that moves a little bit depending on how I take my attention in the world. Thanks, Stephen. Yes, so we have a few different axes or continuum here. There's the phenomenological or what is experienced and then there's sort of the third person impersonal where it's more just about what's happening in the model and we don't even associate it with a conscious experience. In this mental action paper, it threads the line because it is talking about computational phenomenology. However, this is the structure of deep active inference policy selection models which don't need to require any kind of phenomenological experience. So blue, and then we will continue to the next slide. So just in thinking about agent systems and information encapsulation, it seems like to be part of the same system, there must be partial information encapsulation. Like you can't play a basketball game with a bunch of other people without sharing information such as the position of the ball. So there's, you know, and the rules of the game, etc. So the partial information encapsulation, I mean, how to what degree, like what percentage is information encapsulated between individuals, between modules of an individual, tissues of an individual, and so forth. I really wonder about that. I mean, even like you think about the butterfly effects, like we're all sharing, I mean, globally information such as climate change or this type of thing. So there's information at a global level that affects every single one of us because we're all part of the same global system. And so there's got to be some kind of partial encapsulation, I think, just to be connected in a system. Great. Steven? I think that's a good point Blue makes there because there's only certain things available at certain temporal scales to make some of these, the temporal depth of being aware of the ball in a practical sense is a certain, at some point it just becomes noise, it's too fast to sequence it. There's no point getting outside of actually processing sensory information itself at these faster levels. And as happens with all the uncertainty and error prediction that goes up the chain, there's a certain Goldilocks zone, right, for knowing how the ball is going in the game. And if I'm standing and watching the game, there's a sort of a Goldilocks speed which is probably relatively slow, just like in a soccer game, if I'm sort of running down the wing and not getting the ball a lot. But when the ball's actually kicked to me, I've got a player coming in to tackle me, I've got another player running and we've got to do something with it. Now, bang, I've got a window that's got to happen in a shorter period of time. The regime of attention and the action policy selection is in the terms of like half a second, maybe milliseconds of which something and not even available in our conscious awareness, unlike maybe the tracking of the game, which might be a window of 5, 10, 15 seconds, maybe longer. And I think that in itself, would that not itself create some sort of modularity? Great question. So we explored several different senses of modularity in the dot zero. So there is the physical connectedness, like the anatomical connectivity, anatomical modularity. And then we talked about the effective and functional, which are two statistically different. They're not interchangeable, but they're two different ways to talk about modularity. So now we're exploring how these notions of modularity line up with potentially the nesting of multi-scale systems through space and time. So I'm pulling up one of the figures from Rammstead et al. 2018, answering Schrodinger's question, if I can just resize it properly. And so here on the x-axis is time and on the y-axis is spatial scale. And what this graph is laying out loosely and it's been described in other papers and discussions since is that things are kind of existing on the line where we're seeing that smaller things are happening faster and larger things are happening slower. Things that happen over continents are happening over longer time scales than things that can happen at a sub-cellular scale. So there is kind of this natural relationship between spatial and temporal scale of processes and how does that relate to the modularity that Blue was describing. If there's complete encapsulation of a given module, if such a thing could even exist, in reality is one question. Now, if something could exist in our model, of course it's true. We can develop a model that has that connectivity amongst variables. Doesn't mean it's the best model or it's the one that is the perfect territory description itself, but it's not. It's a map. And then if we had no integration, that's kind of like saying that it's just encapsulated at the next scale up. Or I'm sorry, no encapsulation within a level is kind of like saying it's just perfectly encapsulated at the next level. So in a way, the partial information encapsulation is the gray zone that everything is going to be existing within. Steven, and then anyone else? Yeah. And actually, because they've sort of gone back on this particular diagram to some extent after feedback from Casper Hess. I think this is really where there is a, because this continuum, this idea of a continuum going out is what I think active inference sort of changes how we see that. And it's hard because when we try and map things like what Maxwell was doing here, he was taking stuff from the social sciences and the different approaches that are mainstream. And this is how it pans out. But this is often because this is what we think of something from the outside. We see from the outside group behavioral patterns. We see an organism. We categorize it. We make models of it. And they sort of stack up in this way over time and in time and temporal and physical scales. However, from the perspective of the dynamics, are we having a different Goldilocks zone within smaller and faster temporal dynamics in terms of what's actually happening? And when that means something's out in the niche, we may see stuff dropping out from that. But there's no reason, for instance, where niche construction is not being under-girdled by rapid decision-making at real time, just like the soccer players are all making choices on the pitch around where they run. It changes the way the grass is going to get worn down. So it's interesting actually now looking at, so what is the replacement to this diagram? Has there been one that doesn't have this kind of micro-macro and actually is able to have an action-orientated meso that can meet the requirements of what we're talking about, pluralistic ontologies and ontological pluralism? Well, where is that? Where is that being described? And I think that's a missing link at the moment. Great. I think that this takes us to a good return on the Markov blanket, scale-free or scale-friendly debate. Because just like you said, Stephen, somebody talking about a cellular scale and the kinds of perception, cognition, action happening at a cellular or an organismal scale, they're not saying that niche construction doesn't exist. Ecological scale niche construction, because we could also think about tissue-level niche construction, rather that the regime of attention of the scientific modeler is being directed towards a given kind of modeling. And so that's sort of how we take something that is a unifying theory like free energy principle or a scale-free or a scale a priori model, whether it's Markov blankets or linear regression, but in any application to a given system, there are going to be timescales of analysis that just do and don't make sense. And that's where we operationalize the Markov blanket, or as Benny wrote, although Markov blankets are in principle scale-free, meaning there are no inherent constraints on their size and scope, their application or imputation to various target systems will impose a definite scale on Markov blankets. Dean? Yeah, and I think, again, that's why I want to talk a little bit about the instantiation moment. I'm sort of piggybacking on yours and Stephen's comment. When we apply a Markov blanket, we can ask the question from an encapsulation perspective, did you see what I saw? It's a question. We're both behind a blanket. We're both wondering if there's some agreement that we've gained some sort of a contextual alignment. I could also say to you, did you see that? Which may sound like almost the same question, but it's not the same question. It's not even... They're radically different questions, because one says, are we in agreement on what's being encapsulated? Meaning, are we forming a co-contextualization, or the other one says, I'm not sure, right? I'm not sure what you're seeing. So to this question of scale, free energy is moving us, I think, to that place of co-existing contexts. But I think the critical thing is, we've talked about this so many times. Don't confuse the something or other, the map with the field or whatever. I just get lost in it all. But the bottom line is, can we do two things at once? I know we can. Our brains can realize that applying a partition versus or in conjunction with, what is that slice of time as a relativity question? And then, when does that relativity question appear? At the beginning, is that our takeoff point, or is that our conclusion? That's all that this paper to me does. It says, at what moment do we employ categorization? Interesting. Dean, thank you. Dave, then anyone else? Yeah, you know, to Dean's comment, when I ask, did you see what I saw? The proximal pole of your attention is being invited to be projected into me. So you're seeing it through me. You might compare that to what some crazy Russians were doing 100 years ago. They were trying to get the notion across that very few people are conscious more than a few times a year. And they said, well, look, I can explain away your impression that you're conscious a whole lot. Here's what I can say. I can say, hey, are you conscious? The very fact that I asked you, are you conscious causes you to be conscious? It shocks you into consciousness. And you wake up and you say, well, yes, I am, of course you are because I force that on you. Right. Even? And this takes in the idea of what is the adjacent possible, what is available in that ecosystem for awareness that we're talking about. So just like you said there, if someone says, did you see that, that can take someone into a sense of like, I'm trying to work out often, particularly if there's a power difference between people, what that other person's now wanting me to see. And I've seen this a lot actually when we're doing work in South Africa and with communities. One of the first things is to let people, and you ask them a question to get them to a point where they're actually telling you what they think, not what they think you are thinking about what they should be saying. Right. So where is that attention? Just like I say, where is it in terms of being conscious as an awareness? And how much of this scale is, well, what's available and what's going to be an impact in terms of the adjacency. So a cell has some awareness potentially and maybe even some adaptive inference at the kind of, at the actual organ level potentially, and maybe further, it will have an impact maybe in terms of the way cells divide on the nature of speciation. Yes, but it hasn't necessarily got any trajectory to adaptively couple it, because it's not in its adjacent possible. And I think that is quite useful in terms of, it starts to open up in terms of the information, either temporally, how much is something separated by time, spatially, how much is it by space, I suppose also by statistical contingencies and the ability to adapt and interact. Those types of things, well, what is either within or adjacent and what becomes outside of that? I think that starts to give some of these, these scale in modularity questions, just maybe an applied direction to come out. Thank you, Stephen. Blue? There is a huge fundamental difference between asking someone, what do you see and do you see that? Even if you don't give them a that, you're still looking in a direction of that. So what do you see? Let someone look at their surroundings, take stock in what they see and give their independent observations versus just even do you see that and you look over in a direction, at least you're directing someone's attention with your question. So asking an open question versus a directed question can give very different answers. And I wonder what the impact of that would be on modularity or even just an information sharing. There's more shared information in do you see that or do you see what I see? That's like an intention to share your generative model at that point versus what do you see is actually trying to take stock in someone else's model. It's just an interesting thought. These all point using the example of communication and visual perception to where we have partial information encapsulation. If the two entities were totally encapsulated, they couldn't communicate. If there was no encapsulation, such a question wouldn't be required to be asked. But there's a specific type of edge in this network. We could almost think of these like two modules as like maybe the two generative models of the individuals and there's this connection when some of the modules are connected. And it is a nice how we're kind of exploring these subtly different questions. I think did you see that? Is that corresponding to an external state, like an observation? Did you see that red blur in the sky? I don't know what it was could have been a comment, but did you see that versus what I saw? Well, we know that you see with your mind generative model of vision and whatnot. And so especially when we're outside of the signal processing paradigm of vision, did you see what I saw is like quite literally asking is your internal hidden state related to mine? Are we seeing the same thing? That doesn't mean are the photons hitting both of our retina. It means do we have a shared generative model of visual objects? So Dean and then Dave. Just one last thing. That's a great summary. First of all, secondly, when we're talking about what is the nature of the partition where I think as a group, we're still trying to define what we hope is is not completely definable yet. Do you want to understand with that partial encapsulation? Not saying it's undefined. I'm not saying it's zero, although down the road I might say we're passing through zero because that's what I'm kind of moving towards. But I think when we partition, that partition is not static and the definition of it is to leave it somewhat undefined. I like your explanation better, Daniel, but that's what I think we're trying to move towards here eventually. Great. Dave, do you want to add something there? Okay. So blue or if not blue, then Stephen. I suppose just pushing a question back. We talk about this encapsulation modularity and where we put that by putting the blanket somewhere seems to be kind of implied by that. So that kind of is done through a way of speculating on where the observations are generating some sensory states and where we are in terms of the action dynamics at play. How much is this dropping out of the literal availability at a certain scale? And then we make the blanket around it and how much of it comes from structuring the blanket in terms of how we implement this. I think that changes what modularity may or may not mean because I think that how those separations and modularity are done is it a case of how parallel processes are happening and are being integrated and they're modally almost independent. Maybe certain senses are independent and how much is it that they're independent because they're separated at an integrated level by temporal or physical space? Great. Questions? Are these partitionings are a priori model or do we merely specify what we're going to measure and then the blankets if we're using the technical definition of a Markov blanket like the set of variables upon which two other sets of variables called internal and external states become conditionally independent. In that case, you don't need to specify which nodes are blanket states, you just make the measurements and then relative to a target set of states some other set become a blanket with respect to a third external type of states. So Dean and then Blue. Okay, I'll be super quick. This idea that our minds are able to categorize it's a world of segregations like in backslashes when we're looking at it symbolically or integrations i.e. hyphens. These are symbolic knots to the Markov blanketing so the instantiating or partitioning and applying I'm sorry. So my example here might be a little strange but be generous. When yellow becomes orange in an fMRI or a rainbow which begs the question is a rainbow a piece of encapsulated information? Is that representation transitive or intransitive? So i.e. the claim that the most important color in the rainbow is yellow on one end on the other is at the end of all rainbows must exist prisms made up of unicorn sea serpents and flying monkeys or maybe free and energy and dynamic or principle. See where I'm going with this? Our minds are able to do both ends of that continuum at once. And that's what I think where the architecture and the process both come into play. If we can look at this from more of a did you see what I saw perspective and most people won't say they're comparing flying monkeys to free energy dynamics but people can. And maybe I think maybe that's what affordance or minds are trying to set up here. It certainly is with how we encapsulate the information how we as the observer choose to do that and then try to gain some consensus around them. Thank you for your patience. Late nights in the Dean household blue and then Steven tickles me just tickles me. So I have to just comment on Dean's comment really quickly and then I want to go back to what Steven said. So when yellow becomes orange in an fMRI now I don't really know a lot about how fMRI works but I don't think that there's actually any color spectra applied like yellow is 570 to 590 nanometers in the electromagnetic spectrum but I don't think that fMRI actually turns out yellow. I think it actually turns out something between 0 and 256. So like in terms of light intensity and we just faithfully apply yellow to that. That's correct right? So it's an arbitrary distinction when yellow becomes orange or rainbow. Like this is just something we made up. Just to share our generative model like to share information about the picture that we're seeing on the screen. But going back to what Steven said and then this is going to hit on a point that Dean is going to love also. Because we apply the Markov blanket to the system that we're interested in studying like to the organism, the individual, the heart, the liver, the cell to define what makes the internal and external states conditionally independent we apply the blanket therefore we create the module. Is that the same or is that not the same? I vote same. Are we engaged in module projection or module discovery and not even or or what other ways do modules apply? So Steven with raised hand and then anyone else? So there are two interesting points that have just been made there by blue and in that moment and Dean. When we say doing both at once I think there's also times I would argue though that we're able to do both and some of these processes can be possibly be running parallel. However the regimes of attention at certain scales we can't do together in the same way we almost have to go from one to the other unless there's an adjacency between the two types of regimes I would say to enable that to happen. So if I'm thinking about something for instance that's quite somatic and then I'm thinking about something which is quite abstract over time I might be able to train myself for that abstract to have a marker in the somatic that I can then integrate with but I'm still integrating two somatics together in the somatic regime or I might have a model of the somatic in the kind of ocular visual modelling mode and I can imagine the two but in the same moment I can't hold both with the same type of embodiment. So that's one of the things that I think is quite important to wonder and then also once we then get into these is that ability to configure is that the Markov Blanket and this sort of takes us into sense finding, sense making type of work is the sense finding that's okay how am I even sensing what to do in this moment and we have that and then I start to sense and then I'm sense making how to do it and then I'm into skillful practice and maybe I can just do it and some things are more or less in and out of is that sense finding sort of aligning a blanket I basically align in what the sensory and action kind of states that are relevant are going to be done and then we maybe have that set and now we're in a kind of more generative sense making maybe even just skillful flow okay I've got my general blanket in place in a skillful flow now right and that sometimes takes time you know and it's also maybe you know we can only switch attention to something that's in a different regime with a certain speed you know so anyway both those points really struck me The speed of attention it's really interesting and Kristen and others have related that to the rapid oscillations like the gamma cycles in the brain which you know they don't happen a hundred thousand times per second but they do happen more than once a second there's a certain number of times and it's interesting how that lines up with our perceptual capacity like when something is like alternating and rotating and then it goes from sort of black and white to gray there's a moment where the perceptual speed is just not able to see that anymore as two contrasting states but it's one blurred state and it doesn't have to do with the photons hitting the retina being blurred it has to do with the generative model of vision Dean Steven I completely agree with you and I reference back to sort of Wittgenstein's illusion of the duck rabbit and how do we put a Markov blanket between those two representations you can't see the rabbit when you see the duck and vice versa and then that leads me to the parable of Bob and Alice so I just wanted to put that in there I think that's a great time let's move Bob and Alice up let's put Bob and Alice on their own page with the duck I don't want to even bias which one it is with this representation so would you like to maybe set up the Bob and Alice or we can look up the full text go with the text because it's really clear yeah awesome all right so here we go okay okay we're going to assume there are two neighbors Bob and Alice so A and B kind of a classic quantum parable every night when Bob is beginning to brush his teeth Alice begins to read a section of a tale of two cities at the adjunct flat state B contains Bob beginning to brush his teeth in his flat and state A contains Alice beginning to read the book at hers let us assume there's no causal relationship between the states A and B given the two states are segregated there may or may not be a common cause that pre-establishes a harmony between A and B say when what happens in A and B are both bedtime activities coordinated because of various cultural and geographic habits however the assumption of common cause does not need to affect the present discussion because what is important for us is that information processing in A is inaccessible to information processing in B and vice versa this is in harmony with satisfying the assumption of encapsulation and the assumption of modularity to some interesting extent so Dean maybe unpack or explain what you think is interesting and I'll start to get some nodes and edges on this slide so I don't know which the duck or the rabbit which is Bob or which is Alice but to the observer there's information encapsulated in that which can be broken down into A or B then the question becomes one of so how do we modulate how does the switch between those two different representations held in one one block of pixels work how do we partition when that potential for one or the other exists did you see what I saw I guess we're coming back to that question okay Steven and then I'll be developing these nodes a little bit yeah and what we see well I'm gonna add one thing here it's sort of related I think is look at that rabbit and that duck okay well the rabbit is forward looking and decisive and the back is backward looking at the past because the visual grammar that we use in the west is mostly that the left of a vertical flat image or flat piece of paper is the left is the past the right is the future so even that flipping the nature of that rabbit is caught within a cultural frame which would be different if the duck was facing the other way and the rabbit was looking backwards so there we go there's something interesting there yeah now the the rabbit's like looking back in the past right so there's something interesting there about some other kind of the contextuality I suppose of how things are so I just thought I'd add that in yes as the symposium with Friston discussed a lot illusions reveal our priors so if you're trained in ballerina dance and it's only a clockwise rotation when you see the rotating ballerina you're going to see that that's going to be overwhelming posterior likelihood given the ambiguous visual stimuli or if cubes are always facing down into the left you're going to perceive that when you see an ambiguous cubic stimulus and so it does matter these deep priors on how we perceive even visual stimuli blue so there's an important component I think that's missing from the Bob and Alice paradigm and the author leads to it in describing shared cultural norms bedtime etc but there's Bob and there's Alice and then there's the niche the environment essentially that they both reside in the environment and in the environment it's 9 o'clock and so something interesting that it reminds me of is that I was reading that in transitivity paper I was just looking for it Daniel maybe you have it in a slide and can reference it here in transitivity in various contexts and we gave the example of rock paper scissors but they also mentioned in transitivity in the Locta Volterra population model with the fox and the rabbit right so normally like you don't think about that as being an intransitive relationship it's like the foxes go up and the rabbits go down and the rabbits go up you know it's like this cyclical thing between the fox and the rabbit but they mentioned the importance of the environment and how like it's just normally considered that environmental resources are constant but that's not normally the case and so eventually like if the rabbits eat all the grass the population of rabbits will go down unrelated to the population of foxes and so it's I think important to consider in Bob and Alice in this relationship like what is the role of the niche environment and then does like this third hidden state like okay we see Bob and we see Alice and they don't know anything about each other so information is encapsulated like Bob doesn't know anything about Alice Alice doesn't know anything about Bob but they both live in the same place and that's what I was saying earlier about the butterfly effect and climate change like the environment plays a role in the distribution of information between Bob and Alice great so we can look at two different stories and I think this totally gets to the difference between realism and instrumentalism as well so what are the difference between these two well first off it's funny that the book being read is Tale of Two Cities because that's like the Tale of Two Den cities like we read and it's also invoked in the title of this paper so there's going to be two stories of red lines these are two narratives that can be told scientifically by modelers okay so no one's disagreeing about the niche no one's saying well actually they're one's on the moon and one's on earth so we're talking about the same niche now let's imagine that Bob and Alice were two brain regions B and A so we observe brushing and reading to be co-activated they're co-expressed behaviors and so what does this edge reflect we have an answer there it's the functional connectivity functional connectivity first in 94 functional connectivity is the temporal correlation between spatially remote events so that is functional connectivity the alternative is to refer to effective connectivity which is the influence of one neuronal system over another let's go back to Bob and Alice so this is a functional connectivity edge if there was an effective connectivity edge it would have none because although they happen at a temporarily correlated time let's just say they each did it 90% of the time so overall like 81% of the time they're both doing it and so there's a really high temporal correlation however the 10% of the time when one is not doing it it doesn't have an effect on the other so they're conditionally independent so actually the effective connectivity wouldn't have an edge if it were calculated appropriately so functional connectivity we see an edge effective connectivity we don't see an edge and then there's the third story then Dean which is a causal mechanistic story which is the niche broadly being the time zone and the human circadian rhythms and all these other features the niche is influencing Bob and Alice independently it's a confounding variable and then that is what causes Bob to brush and Alice to read and so yes we do get this temporal correlation arising from a latent causing variable and so how does that relate to neuroimaging studies when people find that there's functional or even effective connectivity amongst brain regions and then cast that as a mechanistic story about what is actually happening Dean and then Stephen isn't the effective I'm asking because I don't know is the effective connectivity here on a Z plane meaning on the observers standpoint the effect is this XY relationship um so because that's time right the changes through is through that from my eyeballs through that screen into that slide and then beyond on the other side of that is that not the effective connect connectivity can't show it here because we have to work on 2D rectangles but isn't that it um correct you could think about for any given edge like between brushing and reading there's going to be some fc score between zero and one and then there's some effective connectivity score between zero and one and so you could have cases where that edge is like a zero comma zero it doesn't have an active nor functional connectivity in a given experimental measurement right and then you could have high for both and it always has to be described in terms of the measurement that's being taken because the temporal the temporal correlation doesn't exist outside of the time series that the correlation is being done on right and so that is where Benny introduces the model based science it's like if you're talking at all about the connectivity activity statistically defined you're talking about a statistical model on a specific experiment so even if you're using a tool like a marco blanket or linear aggression that is not posing an a priori scale in that time series there is an a priori scale it's the specific time scale that you measured um so yes the edges can be of different kinds and in neuro imaging again as we talked about dot zero there's the anatomical the functional and the effective and so wouldn't that be simple if they were all the same but it's just literally not the case there's brain regions that don't have direct connections to each other that have temporal correlations and there's ones that are directly connected but they don't have temporal correlations and every other combination that's why we have to do empirical measurement so Stephen this the difference as well is now we see niche being brought in when we have functional connectivity in the brain it's kind of assuming that it's all brain processes so okay you've got the brain and what's going on and the connection between different areas now if we start coming out and we start looking at well there's active influence with marco marcovian approaches okay there's brushing but brushing is an action coupled to a niche and the body and the generative model and so you've now come and the niche it's not just that there's a niche out there that's also interacting with a kind of a brain based process you've got a niche but there's also then another recursion where okay we're now looking at the idea which is maybe being reformulated conceptually more of this whole thing happening and this whole thing this whole process, this model is in a niche itself okay and I'm creating in this model of this story a functional connectivity which maybe substantiated through data if science is used between brushing and reading because the story has put that system of information to me because I've never met Bob or Alice and they're most likely fictional so there's an interesting move from what started out as brain regions being looked at as functional connectivity trying to then think what that's doing because this something out there this hidden states to an active influence agent where you now have action to give you a grounding to jumping back out again to another sort of conceptual system of information which we may be working with to look at ourselves so I think that that changes where does that word niche fit great thanks Dean I just think that if I'm going to incorporate effective connectivity I'm actually looking at the observer observing so somebody throws me into a CAT scan machine starts taking slices on my brain I'm sure they're going to be able to point out some sort of a functional connection between what image pops out when my brain is focusing on the rabbit or my brain is focusing on the duck that functional connection will definitely be able to show up in terms of some sort of differentiation between those two ways that I'm perceiving that image I think it's harder and maybe there's because I don't know but maybe there is some way of being able to show how the brain is swapping building the Markov blanket in situ I don't know if there is a paper about that I'd like to read it because I know the observer in Betty's example here is swapping at some point between am I looking at Bob am I looking at Alice or am I looking at nine o'clock or am I looking at the swap and I'm not sure if it's easy to be able to show the swap part but it's definitely there great blue did you want to add something about the transitivity of the Volterra ecological modeling so I had already mentioned it okay just putting the image up okay great Steven sort of it's going to be a little bit of a question but it's rhetorical if you want it to be but if we've got here the relationship between the brain processing or the body processing of the organism there could be effective connectivity because there's some common relational processes if we're talking about how they're in their niche and maybe some loose way that feedback loops can reverberate through the buildings that they're in and the social malure you can see the niche coming in and then if we then go to well what system of information that we're engaging so if Alice and Bob are coupled in our system of information here that could be functional because we it's within the variables that I assigned to Alice and Bob in whatever way that I'm embodying this meaning making so I was wondering if those three sort of ties in a bit to what you've got here but I kind of feel it's useful to it starts to think about where was Friston coming from when it was kind of a trying to get at what the brain's doing in response to and really get at what's going on in there to when it's something which is actually happening to something again which becomes a kind of an abstracted level and I don't know which of those is closest to what is normally being taught does that change between the actual nature of the system do we have a way to think about that or is that speaking to this process that when you just have DCM models and you just have these kind of informational models that has to be kind of set you have to kind of tell it and believe that what you're talking about the beauty of active influence is I have a deeper anchor to help me tether all of those three I have an embodied way of knowing how it is to imagine I have an embodied way of knowing how to brush and read I have an embodied way of enacting motor skills in real time to make the brushing and reading even possible and that's something that the previous models haven't really got a plausible mechanism to explain until we got active influence DCM doesn't seem to have that ability it's all still kind of untethered information based approaches okay interesting I think these three tail of three cities now we have a functional so these are predicated upon observations of so first to Steven's point yes we could think about brushing and reading as affordances and embodiment and that is something that active influence brings to the table which is like the ability to zoom in on one of these observations and it's like we're not just studying the observation of brushing there's an entity that's engaged in this skillful action that's encultured and that just takes you a million other places so yes active inference is a development on what Friston and others have been doing for 30 years we really have to be careful whether we're using terms in their technical statistical sense just to show how easy it is to get mixed up you can say well I mean brushing and reading are correlated so it's like they're effectively connected but they're not that's called functional connectivity functional connectivity is the temporal correlation and then here's how easy it is to get confused on the other side here's a story about the world where Bob does call Alice but we are not wiretapping and Bob says hey I'm brushing you should read and so it turns out that in this world not just are brushing and reading functionally correlated like they have a time series correlation but actually brushing does induce reading through a mechanism that we didn't observe that sounds like a functional correlation or a functional connection it's not it's an effective connection how it's technically defined the influence of one system on another and so yes function is happening action is happening but it is very important that when thinking about these edges people are accurate because these stories are all happening at the same time and different experimental designs or statistical analyses might not be contradictory to each other they might be totally complementary to each other somebody might be describing the enculturation of how brushing and reading come to be somebody else might be talking about the functional or the effect of connectivity or all these other kinds of pieces and so I agree active helps us understand the non-rival risk dynamics of all these different descriptions however we're going to be on the ground floor forever if we can't be clear about what those different descriptions are and adjust our regime to these different important parts of the system Dave and then Dean ok Dean and then Dave if you unmute later so Dean and then Steve and then blow yeah so could you put a second thought bubble under ec and could it be hey I'm nine o'clocking are you like because that to me is the effective connectivity I keep the hey I'm brushing you should read but also there should be a second one hey I'm nine o'clocking are you nine o'clocking because that to me is speaks as much to the effect as opposed to the function yes good point and this there'd be depending on how literal you want to take this metaphor or how you're imagining the story Bob and Alice or how close you want to tie it to neural systems one could imagine various pieces this is just to highlight like red is this causal chain the difference that makes a difference the actual phone call that induces reading that's unobserved in this experiment so fmri is measuring blood oxygen level dynamics bold signal and so that's not even neural activity now read the spm textbook because that's how you infer neuronal activity from the bold signal but it's not what's being directly measured right but but I just want to add so even the observer is nine o'clocking so now Bob is nine o'clocking Alice is nine o'clocking and the observer of Bob and Alice is nine o'clocking that's the that's what I kind of want to bring into the conversation yes totally agree like here's our happy scientist don't know if there's a better icon to represent the scientist but right they are going to be observing they're the ones who are making all of these observations and it's it's um it's somewhere between misguided to dangerous when it's like well this is simply how it is it's like well no there's the smiley guy who is projecting that who's modeling model based science science is a social praxis humans doing science this is really important to keep in mind otherwise you might have you know a smiley and a frowny and then they're debating two different stories and one is actually talking about the fc and one is the ec and one's debating the niche and you could have all these different perspectives and they don't have a common rosetta stone to actually come together and some people are just talking about what happens in principle no reference to a data set someone else is talking about an empirical correlation the coefficient from a specific data set so yes steven then blue yeah this is very helpful to to map it out like this and um it sort of gives a sense of this journey of functional connectivity functional um to effective and I'm wondering whether we the the process has evolved even further to affective connectivity in the sense of like Alice could ring up Bob and say hey Bob hey I'm feeling good tonight it's been a great day okay Bob's like okay I'm feeling good hey I'm gonna brush my teeth um so um the there's a and I'm a bit more motivated to be into that big book that I really not wasn't so interested in before but I know Alice likes you know so there's a is that is that kind of that extension something that's almost could be charted here from functional to effective to affective so it's a good question we can imagine this little variable because again we're not um in the territory we're on the map if this variable is Bob's affect and Bob's affect has an edge that is influencing how likely Bob is to undertake the policy selection of calling Alice then yes there is an effective edge or a connectivity between the variable corresponding to Bob's affect and then the policy of what to say and then continue onwards but not every single piece is explored here but yes affect can enter the picture and that's what so fun and exciting about these incredibly composable and flexible models like active inference so Stephen and then blue and this also you know sort of harking back to that earlier question about social sciences and other sciences and this actually does provide a useful way to have a construct to look even at the arts the sort of relationship people be having so as you may be thinking about as people think about narratives and stories as well as okay what is explicitly happening and measurable through modeling what kinds of relational dynamics might be at play and that itself could be a useful boundary object in constructing co-creation work because you started to get some sort of framework again provided you get some of the minimal constructs clear so that it doesn't just become noise thanks blue so I'm also wondering about the relationship between functional connectivity and effective connectivity and just really I think about this always in in relation to like gene expression right so you know we observe a functional connectivity because two genes are expressed in the same tissue at the same time they're both up regulated right so this is something like that Daniel and I have studied and maybe four into the rest of you guys but effective connectivity also in terms of gene expression is like there's a downstream effector molecule right like so there's a molecule that has an effect downstream of the gene expression so like you know there's one transcription factor that diffuses the tissue and then activates the transcription of something else right and I wonder so it's not that I think functional connectivity and effective connectivity can occupy the same space at the same time but what is it then when there's effective connectivity like I'm the transcription factor so I'm effectively connected to downstream genes right and so when I activate two genes at the same time are those effectively connected to the transcription factor but functionally connected to each other like I wonder just if there's any degree of overlap between functional connectivity and effective connectivity thanks Dean can I ask you a question blue is is because I know nothing about genes other than I wear them occasionally do do genes automatically go to the place of affording us certain things as an effect or can genes be an affordance as a cause because it sounds to me knowing nothing about genes that you're describing that you're saying that genes can also be a cause which I think would be really interesting but I'm not sure if that's I don't know like is it is gene only an effector whereas gene also an affordance around cause so genes can be causes especially like genes like transcription factors what I was mentioning earlier a transcription factor just like activates the transcription of other genes that's like what it that's what it's job is so and it's something that happens like very normally during the process of development and the transcription factors diffuse along axes and then it you know causes like a cascade of of you know developmental gene expression timing that happens so if it could be both a cause and effect are you saying that it has whether we whether we measure it traditionally with a time stamp that it in effect has a built-in affordance for time is that because it sounds fascinating if it's true because I said I don't know anything about this but if that's the case maybe some of the traditional ways that we measure change have to be reviewed so right like like in let me see if I can understand your question fully so in the process of development there are specific genes that are expressed in specific issues at specific times that aren't expressed any other time so in that way it could I guess mark time is that kind of what you're asking yeah yes genes as virus is a very interesting idea like there are certain critical periods where developmental factors are expressed in sequence but let me also respond to the comment about what blue said about genetics so unsurprisingly genetics uses a different ontology they use a different own ft actually ontology narrative formal documents and tools then neuroscience so when people in genetics are describing functional connectivity or functional interactions call those connectivity usually for biologists function is equivalent to mechanism and so a functional connection would be like a mechanistic physical linkage so two proteins that bind that's called a protein-protein interaction or like DNA protein binding like the transcription factor example that blue mentioned so that is often what is meant by functional connectivity effective connectivity I don't think is used that much but if it were it would be used informally to describe the effect of one on another mechanistically probably not in this time series perturbation framework and this is one of the most fascinating areas like people talk about gene networks or gene regulatory networks but there are multiple kinds of networks that you can make you can make networks of proteins that bind to each other you can make networks of proteins that are co-expressed so they have correlated expression patterns which is functional connectivity but that doesn't mean that they're functionally related there could be one factor or cellular scenario that causes two genes to be up regulated so they're going to appear in the same co-expression group and therefore a neuroscientist would say that they're functionally connected a biologist might jump to say there's a functional relationship or oh there's a gene ontology enrichment of this co-expression module so there's some sort of functional relationship but it's being used in a slightly different way and wouldn't that be fun to have shared ontology for systems that ultimately are related like gene expression and neurophysiology so blue and then Steven okay let me just respond to that really quickly so take my gene expression comment aside let's go back to the brain so say there are two neurons that light up at the same time they're not connected so that is a functional connection between those two genes neurons or brain regions that use neurons for the sake of simplicity so two neurons that light up at the same time they're functionally connected now one neuron that lights up and then touches another neuron and tells it to light up that's an effective connection is that correct yes or if you had a hidden third neuron with a time delay and it influenced one in one time step and it took two time steps to get to the second one you would see an effective connection measured because the second one would always get activated after the first one and so that's the challenge with interpreting effective connectivity as a mechanistic story when actually a whole diversity of mechanisms can give rise to the observed empirical effective connectivity okay but wait so here's my question like I'm trying to see if there's any overlap between this functional and effective connectivity so say I have one neuron that touches two neurons those two neurons to light up at exactly the same time is that effective connectivity between the original neuron and the two neurons that light up subsequently and then functional connectivity only between those two neurons that light up at the same time but are not touching is that correct to think of it that way yes the two children would have functional connectivity because they have a temporal correlation between remote events however they would not have an effective connectivity because activity in one would be conditionally independent upon activity in the other and then from the point of view of that parent neuron it would not have functional connectivity because let's just assume that it spikes and then the next time step it's not spiking in the children are so it wouldn't have a temporal correlation in its activity at a lag of one it gets a little bit more complex when we're talking about time lag models but it would have effective connectivity through time because it is like a directed edge influencing them and functional and effective are independent of the anatomical connectivity which is what you'd find out with a microscope without taking any measurements so there's no overlap between these two categories there are situations where they can be high for both or they can be low for both and there can be anatomical connectivity and functional or effective or any combination it's like for two nodes there's like a tuple of how strong that edge is functionally, effectively and anatomically and it could be one one one it could be negative one one negative one it could be anything so I'm trying to think of a circumstance where something can be functionally and effectively connected how is that possible so there would be two cryptocurrencies who over the year timescale have both gone up so at the course graining they have a positive correlation that's the functional connectivity and they have a day to day correlation in their movements such that one moving leads the other that day so then they'd have an effective connectivity on the short timescale because through time changes in one caused the same direction of changes in the other and when you zoom out they have a broad scale correlation got it so depending on the timescale they can be functional or effective yes and this is like where it does matter exactly how there's not just one way to do a time series correlation because a time series correlation can be just like an Arima model can have different time horizons so one can look at the temporal correlation like let's just say you had two out of phase oscillators if you had a time horizon of one they would have very strong negative correlation but if you had very very long time windows they would have about the same average so you'd think that they were uncorrelated for example so time series correlations extremely depend on the specifics of how it's calculated which brings us back to model based science and you know a W for the instrumentalists over the realists and that's why Benny is saying like yeah don't get too concerned about whether we're finding cyclic or directed acyclic models because those are just modeling conveniences that are downstream of the experimental setup and the measurement and the quantification techniques we'll just focus on that and we'll see that we're engaged in different flavors of the scientific endeavor and ones that include humans and all the ways that they work and that just trying to look at that distilled kernel of like well the coefficient is 0.7 so it's a module that is a metaphysical conclusion or an ontological conclusion being drawn out of an ontological process that might be unwarranted so in our oh yeah Stephen go for it yeah I was so if we were to take between these two and I think this has really helped clarify because yeah I think you're right that these words are possibly open to some confusion at first if we were to take a brain scan in which is essentially you've got the you know you can have electrical which aren't very easy to identify where they're coming from that you can get a good sense of temporal fidelity and then you've got FMRI which tends to be a bit more over time Kristen was bringing has been bringing that to the table the idea that well what's the the more general function at play and what's the type of facially understood changes happening over that temporal depth whereas the effective connectivity might be more like what can we effectively say changes between this and this for a more specific definitive time scale would that be one way that you're talking about the methods influencing which types of models are relevant that would tie into that yes awesome point about sensor fusion as well as the differences between FMRI and M and EEG so EEG is on the surface of the scalp although you can also take electrical measurements within the scalp it's on the surface and it has extremely fast temporal resolution like many many times per second and it's localized and it's measuring electromagnetic activity FMRI has a very different spatial and temporal resolution and it's measuring something different which is the blood oxygenation dependent signal inside of the brain which is related to the hemodynamics and the metabolic activity of brain regions and the vasculature so they're measuring different things that's why it's so important that in active style models that we clarify what our observations are and what the hidden states are that we're doing inference on and so one of the ways that sensor fusion happens in SPM is so what's being described here is multimodal integration of the EEG and the FMRI is there's a latent unobserved variable which is the neuronal activity and then that neuronal activity at a given voxel is going to have an edge that outputs FMRI measurements and then another edge that measures EEG outcomes and then what's happening with that unobserved variable is you fit based upon the jointly measured FMRI and EEG so it's not just like you take a big time series and do descriptive statistics but rather you use this hidden state and that's what allows the generative modeling of brain activity because you're fitting the hidden state which gives rise to observations which can be of totally different types so it is an awesome thing to explore. I hope we can continue to delve into this type of sensor fusion and what these Bayesian graphical models actually enable Dean and then Stephen I'm trying to see if I can come up with ways of being able to sort of keep the effect of connectivity and functional connectivity segregated before they're reintegrated I don't know whether if we were to we were to find ourselves out in the middle of the wilderness and come upon an ant hill and suddenly out of the ant hill poured a bunch of elephants the size of ants whether we would need to look at the mechanics of that or we would just be drawn into the idea that now there's these really what was surprising and essentially ant sized elephants or if we saw elephants suddenly walking or ambling as though they're still the normal sized elephants but petting as though they were ants so taking the tiniest of footsteps relative to their overall scale I think that's where the effective connectivity and the influence based on something some of it based on priors but essentially I want to keep the functional and the effective apart not that they can't overlap or that our minds aren't able to make those connections but I think each serves a purpose so that when we come up against surprising we know kind of what went into that that moment or that event I'm kind of seeing the size of the animal as the year performance of the crypto and then the day to day is the effective connectivity with the footsteps so yeah the scale all I was trying to the scale and connectivity as opposed to the sequence interesting Steven and then there's a question in live chat I'm going to ask a little question with this again maybe rhetorical over the time scale of this effective connectivity it can effectively be brain bound to some extent when we have functional connectivity it's the the measure or the unit of analysis is implying something outside as being connected to that or something that's a slightly more macro IE the idea of the function that's being used to as a unit of analysis to then give a way to get purchase on temporal correlations would that seem to be fair enough and would that also possibly also then reflect the sort of journey that's happened in cognitive neuroscience and computational neuroscience because of Kristen's work it started to that even in itself brought in this process of well what does it mean to have function again we want to be precise about what function means here if the measurements are all brain related then that's the nodes corresponding to the variables that edges are going to be calculated within if there are measurements that are outside of the brain then you can have functional or effective connectivity amongst those variables but we're not actually talking about function like mechanistic impact nor teleological impact in the niche which is like the function of the hammer is to nail things that's not what functional is meaning here and just to give an example this is the awesome recent work by Guillaume Dumas and collaborators and so this is a toolkit that expands on this neuroimaging framework and just says well we're just going to concatenate those matrices kind of reminds us of the stochastic chaos paper 32 right like when we went from having one coupled system to having the two coupled systems and then remember in the off diagonals we had the effect of the two different systems on each other the causal impact of the systems on each other the effective connectivity of those systems and so this is hyper scanning where you have two people or could be a human in a computer whatever it is whatever the measurements you're making are the rows and columns on this matrix and then you have that matrix the measurements at each time slice going through time it's a tensor and then you do time series statistics on that tensor and so these connections between the two brains are not anatomical people's brains are not actually wired up that way however you could calculate the functional connectivity which is not the function of one on the other it's the temporal correlation between the events and you could also calculate the effective connectivity which is what is the statistical impact of one variable on another through time whether or not that's the mechanistic link is a separate question and that is what was revealed with this example was even if you hyper scan those people's brains and you say I mean come on this brain region always leads this other brain region and the other person but that doesn't mean that's the mechanistic connector it just means that it's an effective connective edge and so this toolkit and this hyper scanning approach like really demonstrate that you can have functional and effective connectivity amongst any variables that are being measured in your experiment but that's not the same thing as a mechanistic story about what's happening Steven okay actually just one thing to clarify so is that bottom diagram saying that there's both fMRI and EEC there's electrical and blood flow measurements happening and they've been integrated this one so in this paper that was the sensor fusion that was the integration of multiple neuroimaging modalities fMRI and EEG here you could do sensor fusion hyper scanning or you could just have EEG on each person or you could have EEG on one person and fMRI on the other you'd have to qualify what it is that you were actually measuring but this is a general framework for multi-agents neuroimaging statistics okay perfect so then the point I was thinking that sort of relates is I think it's a really useful point that you say about how it's not functional in some sort of teleological sense however sort of hidden in the linguistic couching that's going on here for it to be viable okay not necessarily for it to be experimentally how it's done is implicitly the effective connectivity seems to be couched on or there to be something meaningful happening or some sort of process happening effectively a one neuron has to exert something or potentially have some direct effect okay so there's it's like a contingency that's present functionally it's like within whatever functions going on but we don't necessarily know what that is but within some sort of process there is something happening that we can infer is useful is coming into play we're not saying because we know they're spatially separated we can't say for sure then or they're temporally hard fuzzy we can't necessarily say they have to be connecting but functionally we're getting a feel for that this dynamic is present rather than this almost more signal based approach if there was more effective and I think that that is philosophically couched in those two words yes I totally hear you the mathematical and the statistical definitions are the ones that are being calculated so trying to read a lot into the English word and then apply that reading in to the calculation is where we get a little lost because again the effective connectivity it doesn't mean that there's a mechanistic relationship you could have spatially remote neurons that have an effective connectivity edge like the example we discussed earlier where there's a causal neuron and then there's a it takes one time step to get to this one and two to get to the other so then those two recipient neurons are going to have an effective connectivity edge it's a hypothesis whether there's a mechanistic edge but that's something that you can explore with science is you can actually look at whether then you test well what happens if I silence this one you go wow the firing pattern didn't change so I falsified or reduced the posterior likelihood of a mechanistic story that involved a mechanistic edge it doesn't invalidate the measurement of an effective connectivity it just means that you have to look elsewhere for your mechanistic story you know you shut down Bob's phone and there still is this effective connection between the brushing the teeth and the reading well now you look back in the system iterated scientific modeling so I just want to ask this question from the chat this will be our last question and well final thoughts but very fun and interesting and I hope next week Majid will join and that will be great to hear his perspective as well so Joseph Clark in the chat asked how do you utilize functional and effective connectivity in active inference analysis to better inform an answer to a question such as why I scored the goal i.e. how does it enrich an analysis so Dean something I would need time to think about that question so maybe that's something I can think about in point 2 because right now I'm trying to figure out if a gene is both a cause and effect how is that then going to be encapsulated and partitioned because basically blew blew up my day I'm not going to need to go away and rest for a little while and think about this because if it's both is there information just encapsulated is that information partially encapsulated because now it's a whole bunch of genes that are dealing with their betweenness and all of that good stuff but again my final thought on this paper is that I think whether we keep it on the abstract level of something or partial or whether we pull it down as he does in the paper to the actual moment of we're looking at something now in particular now I've got to sort of go and take that off to the level of genes and see whether or not that changes exactly it flipped my duck rabbit on a I'm rolling downhill here I need to take a break I agree I think this is an excellent question because effective and functional connectivity you'll find like tens of thousands of citations in the neurosciences and you'll find people talking about the applications and translational value of neuroimaging towards real life settings I'm sure there's even neuroscience of football or of soccer so how do we actually connect all those dots and describe how specifically within an active inference framework how do we think about these different kinds of modularity functional the effective the anatomical the informational how is that related to Markov blankets what's unique about the framing in active inference compared to the neuroimaging that has existed for 40 years now much of it was also developed by Friston but still there's more to it so I think these are great so one thing that we know we'll discuss is that question how do we deploy functional and effective connectivity so that we can get better answers we also want to talk about modularity intransitivity and preferences and let's Steven and then any other last thoughts what are you excited to discuss next week yeah I think this is a good question to end and also start the next one I suppose one thing this idea of that this is useful is how ideas of how certain types of effective connectivity was important for scoring a goal like what was tethered to what in terms of the connectivity and where are we talking about something where what's the dynamical system the generative type of system where that involves making it happen I get a sense that the functional approach is more connected to dynamical systems thinking and in comparison to the effective approach which might be more related to how things are potentially wired up and what's first order, second order third order effects I don't know if the other people see the same but it's helping me this is really useful for me starting to think okay I see where Bristol was sort of coming in with this question of how to address these hidden states these hidden dynamics these hidden causes and how coming in it through this functional dynamic or influence approach it could say it changes everything in some way so it certainly means that you have to have more than one way to understand the hidden states which is also like how the footballers also got more than one way of trying to understand what it even means to score the appropriate actions being mixed with visual being mixed with auditory being mixed with tactile all these things are some things do need to effectively happen certain muscles need to happen in a certain order if certain coordinated action has to effectively happen for it to even be possible but there might be multiple dynamical functional ways that I could also bring purchased and have different ways to get to the same result so and again it's not it becomes relevant when you're trying to do the type of measurements that have been designed for a particular reason so that you can get that you're not going to try and get at the short temporal inferences through using fMRI and vice versa so I think that's something that's quite useful great my closing thought will be studying the footballer is our football as researchers of football yeah thanks Dave, Dean, Steven, Blue and fun to participate today and thanks a lot Majeed for writing this provocative paper getting our 22 off on a good foot in Act In Flap so thanks everybody for listening please get involved come join us next week or a future week if you'd like and till next time