 Hello and welcome everyone. This is the Third Applied Active Inference Symposium in Acting Ecosystems of Shared Intelligence. It is August 22nd, 2023. We have a lot of really awesome presentations and sessions coming up in the first interval and in the second interval. So let's just get right into it. With our first talk, this will be by Andre Bastos. So Andre, thank you for joining and looking forward to your talk. Thank you very well, Daniel. And thank you for the other organizers of the Active Inference Conference and Symposium here. I'm glad to be talking to you guys today about my work. It's the title of the presentation as Multilaminar Multi-Area Recordings and the Non-Human Primate. That's these guys right here, the cat monkeys, suggests that predictive coding is implemented by a predictive routing. And so what do we mean by these things? Well, just as a point of introduction, I'd like to start by just saying that we started out, I started out working with Carl Friston in about 2010 and we were really interested in trying to map the neurobiology of the predictive coding circuits that have been theorized and hypothesized onto cortical laminar architecture, which you can see here. And in that work with him, we proposed that different layers and different oscillations were involved in either capturing and calculating prediction error or predictions. And so what we do in my lab, and I recently started as an assistant professor in the Department of Psychology at Vanderbilt. And so what we do is we train resys macaque monkeys on tasks, which are more or less predictable and that have different sensory elements that are more or less predictable and we'll get into what that looks like. And then we use multilaminar probes. So high density probes with dozens or hundreds of contacts that can record across different layers. And so this is a beautiful drawing here of what the layers look like of cortex by Santiago Ramonica Hall over a hundred years ago. And you can see this really beautiful layered architecture, where one, two, three, four is this dense one in the middle, five, you get these large pyramidal neurons and then six on the bottom. And so we're interested in my lab and primarily at two levels of explanation. The first is what do these neurons spike to? What makes them fire action potentials and excites them? And do they get more or less excitable depending on how predictable stimuli are? And second, how do they oscillate? And which frequency bands do those rhythmic activity tend to cause them to fire more or less action potentials? And so we record across layers in one area over here, but we might put in our electrodes in several other areas so that we can also capture aspects of cortical communication. And these oscillations in this communication we think is really intimately linked to the interaction between inhibitory cells, these cells in red here and excitatory neurons, these cells in black that I've written. Okay, so without further ado, let's just dive right into it here. So here's the outline of my talk. In part one, we're gonna be talking about first an overview of what do we actually mean by this predictive coding model and how do we think it may be mapped onto cortex and paying special attention to the neurophysiology? So here's the bird's eye overview of predictive coding at the level that maybe a neurobiologist or a cognitive scientist or a cognitive neuroscientist might need to know about in order to be able to apply it to their own work. And so here's what I mean by predictive coding, that there's a hypothesized circuit that is especially rich in higher order areas. So here's at the front of the brain, which learns about these statistical regularities of the world and creates predictions about what is going to be seen and felt and touched and so on in the next moment in time and in this moment in time and sends those predictions to earlier parts of the brain. So it's a lower order cortex receive these predictions and sensory inputs, the inputs coming from the eyes and ears and all of our sensorium is then compared to this top down prediction. And if there's a mismatch that's a prediction error and the prediction error is fed forward, it's a mismatch response and it can be used to drive models, drive these internal representations in order to be better over time and to make better predictions. And in particular, these prediction errors are hypothesized to be housed in superficial layers of cortex. And so that's a hypothesis that we've been able to test now with these laminar electrodes. So let's see how our experimental design worked in order for us to be able to look at the difference between predictable and unpredictable sensory stimuli. So we train monkeys on what's called a delayed match to sample tasks. It's a classic task of working memory where monkeys are trained to fixate on the center of the screen when there's a visual stimulus, in this case a car stimulus that's shown. There's a brief delay in which that stimulus has to be held in working memory and the animals tested on which of the three items was in working memory. The animal makes a correct response when they saccada, in this case to the upper left to indicate that the car stimulus was last shown. So we then manipulated the predictability of this information that was to enter working memory in the following way. For 50 trials in a row, we would either keep the sample stimulus constant, so car, car, car, and that's being depicted by the A here. Or in another context, there would be an unpredictable sampling regime where either the car orange or green could be sampled at any moment in time. So the animals went back and forth in being in a predictable context or an unpredictable context. And then we ask the question, what does brain activity look like when the exact same stimulus is processed in a predictable regime versus an unpredictable regime? And in order to do this work, we implanted these monkeys with multilaminar probes. So these are probes, again, with these, at that time that I did this work about a dozen or so, or maybe up to 32 contacts, nowadays we can go into the several hundreds of contacts along a single probe. We can use MRI guidance in order to position those probes, perpendicular to cortex, such that they'll span each of the six cortical layers, layer one through six. Here you can see the white matter here below. And then we sampled from different parts of both the visual cortex, remember those are the areas that might issue prediction errors in the face of an unpredictable sample, as well as higher order cortex, which has been hypothesized to create these internal models that then issue predictions. So here's what we found. We first just analyzed the multi-unit activity, which is a measure of the spike rate. So how many action potentials per unit of time are these neurons firing? And what's being shown here is the multi-unit activity as a difference between when that sample was unpredictable minus predictable. So positive activity means more activity during an unpredictable sample, i.e. a violation of a prediction or a putative prediction error signal. And all throughout the brain from V4, the sensory area all the way to these higher order frontal parietal regions, we saw without any exception that neurons were much more excitable when they were processing an unpredictable stimulus compared to predictable stimulus. And also not surprisingly, this started already at the feed-forward sweep of activity such that these red bars here indicate the start of significance. So Chernoff area V4 was the first to show significance. So more spiking to the unpredictable over predictable and this activity fed forward up the cortical hierarchy. Next, we looked not only at the firing rate of these neurons, but also the local field potential, which is informative about the local oscillatory activity that's present. So you can think of these oscillations as massive waves of activity that are synchronous across the large population of neurons, but of course are produced through the interplay between these recurrent inhibitory bouts of activity. And so here's what we found. We performed this power spectral analysis, again comparing power in the unpredictable case versus power in the predictable case. And we did a percent change metric, very simple percent change metric. So again, what's shown here on the Y axis is in positive, more power to unpredictable samples. And in the negative values, more power to the predictable samples. Now what we saw in the gamma range, so that's everywhere above about 30 or 40 hertz, again in all areas, very strongly mirrored what we saw in the spiking activity, namely, that there was more of this high frequency and also the classic low range frequency in the gamma range. So in both the low gamma and high gamma ranges, an enhancement of power during the unpredictable cues. And that was also the case in the theta band. And this makes somewhat sense from the point of view of neurophysiology that this theta frequency and this gamma frequency, they very often track one another and do similar things in cortex. Why exactly that is the case? We actually don't know what the reason for that is, but that's an observation that we've seen in many other tasks. But what we saw in between the theta and the gamma bands was this other band at between 10 to about 20 or slightly higher in frequency, this classic alpha beta band. And this alpha beta band acted in the exact opposite way. Namely, there was more power to predictable samples than to unpredictable samples with one exception. Area 7A was an exception. It did have that canonical pattern in the alpha band, but in the low beta band, it was actually similar to gamma. And we think that this might be a working memory update mechanism. But for now, we are really stressing and emphasizing the fact that across all of these areas, you had this data signal that was stronger to the predictable case, to the predictable sample. Okay, so next we had these laminar electrodes. So let's look at the difference between what these neurons are doing when you record from the so-called superficial layers of cortex versus the so-called deep layers of cortex. So the superficial layers are layers 2, 3. The deep layers are these layers 5 and 6. These superficial layers, they are the layers that feed forward. So remember from that introductory slide that these are the layers that were hypothesized based on predictive coding theory to house the prediction error neurons. Because according to theory, those prediction error signals are sent in the feed forward direction. So we then tested that in a second way, which was to test whether or not those enhanced signals during unpredictable samples are more present in layer 2, 3. And that's exactly what we found in V4. So when we again take the spiking signal and subtract unpredictable minus predictable spiking, we see that the signal is more enhanced in superficial layers of V4. There was a trend towards that in PFC as well. But in V4 it was a significant observation. And furthermore, when we now look at the LFP gamma power, this power in the 40 to 90 hertz span, again we do the power change. So these positive signals mean there's an enhancement of gamma power to the unpredictable signal. And again now in both V4 and PFC, it's the superficial layers that are having more of this unpredictable signal. Again, and that's consistent with the theorized type of circuitry for these canonical circuits. Next, we were interested in whether or not these power changes were blankets across the board that just overall increase new activity and gamma activity independent of the exact sample that was being shown or whether these signals were actually specific to the actual item that was being predicted. Because remember in our task design, animals were for a large portion of time predicting that there should be a car stimulus while predicting an orange or predicting the green stimulus. So in that particular task requirement, it wouldn't make sense if task representations for all three stimuli all went up to an unpredictable stimuli. That would be more consistent with just an overall arousal signal. But what we were next interested in is whether or not neurons that actually were selected for the car, if it was those neurons in particular that had the unpredictable signals. And what we found is that that was indeed the case. So we performed the same analysis as before, LFP, gamma power difference for unpredictable minus predictable, but now as a function of whether or not those neurons preferred the stimulus that was being predicted. And what we found is that the preference actually modulated this gamma band activity very strongly. And so it was this unpredictable signal change was more specific to neurons that actually preferred that predicted stimulus, indicating that this prediction was not acting as a blanket, similar to an arousal mechanism, but was actually reaching down at the level of the representations that were specific to the thing that needed to be predicted. And that was even more the case in the beta frequency range, actually, where the neurons that didn't encode the predicted stimulus, they couldn't care less whether or not that stimulus was predictable. It was really only neurons that carried information about the stimulus that was being predicted. Those were the neurons that were not enhancing but diminishing their beta during an unpredictable signal or set another way, enhancing their beta to a predicted signal change. So here's what we think is going on. We call this neurophysiological implementation that we found of predictive coding. We call it predictive routing to make a semantic point, but also a slightly more neurophysiological and computational argument that we don't actually find strong evidence for this mismatch in prediction signal at the level of neurons. But instead, we think that this prediction circuit exists but uses these oscillations in order to implement what is predicted. So here's how we think this works. So when a prediction matches a stimulus, as is being shown here, when you're predicting A and you get a stimulus A, then the A column, in other words, it doesn't have to be a column, but the area of the brain or the neurons that encode A are going to become prepared to receive the stimulus A by enhancing the power of this alphabeta signal. So this top-down beta from higher order areas, we think prepares a cortical circuit to receive a predicted stimulus. And it prepares that stimulus by enhancing, that's this up arrow here, the alphabeta, especially in the deep layers, which can inhibit both the gamma and the spiking signal in the superficial layers. As a result, so this top-down signal inhibits the superficial layers. And as a result, there is decreased gamma and spiking and decreased feedforward output in the case of a predictable stimulus. The case of an unpredictable stimulus. Well, stimulus A, just like before, is still going to be coming in, but because it's not predictable, that column, that set of neurons that normally represents A was not previously prepared by an alphabeta signal. And so the alphabeta signal is relatively low in an unpredictable context, releasing the inhibition that the deep layers exert on superficial layers. And as a result, causing more gamma, more spiking and more feedforward output. So in other words, what we are hypothesizing here is that predictive coding and this error computation comes about as a failure in predictive inhibition. And as a result of the lack of these mechanisms being in place during a unpredictable stimulus, there is an enhanced gamma output and that's what we call a prediction error or that's what we interpret as a prediction error. So here's the predictive routing model one more time. In a predictable stimulus, top-down inputs prepare the column to receive a stimulus that preparation leads to inhibition, less feedforward output. In a unpredictable case, those mechanisms aren't in place as a result more feedforward gamma output. So that's part one. That's our cortical neurophysiological hypothesis then that we call predictive routing of how we believe predictive coding is implemented in the brain. And next, I wanna turn to a little bit more about how these rhythms are actually produced in the brain. Because one of the things that we noticed from just placing our laminar electrodes into cortex is that independent of where we put our probe. So here's an example of prefrontal cortex activity. We would see these high-frequency gamma waves and superficial layers and slower frequency alpha-beta waves and these deep layers. And you can do this analysis very easily and quickly to reveal this what we call spectral laminar pattern. And what this is is simply dividing each contact on the probe, dividing the power that you observe at that contact by the maximum power along the probe. And what this does is it normalizes power such that a value of one is the layer that had the most power at that particular frequency band. And so if you look here on the x-axis, this is frequency. So these red colors are tracking which particular layer has more power at that frequency. So you see more of this low and high gamma power in the superficial layer. So that's above 0.0 on the y-axis and more of this alpha-beta activity in the deeper layers of the cortex. And so in a large-scale survey of cortex together with Bob Desimone's lab and Diego. And so I did this partially while I was at MIT when I was a postdoc with Earl Miller and I continued this now at Vanderbilt. So this was a really cross-institutional study and involved multiple labs. And the reason that that was important is that most labs specialize in being a prefrontal cortex lab or a V4 lab or a LIP lab, et cetera. And relatively few investigators had actually done the necessary work to be able to record these laminar probes in multiple areas simultaneously. And so therefore by coordinating and pooling data for multiple experiments, we were able to record from a total of 14 areas in the study that we have now in review. And what we found was that independent of the area we found a spectral laminar pattern that was highly conserved across areas. And so here it is from zero to 150 Hertz for area of V4 in the sensory cortex, 7A in parietal cortex, prefrontal cortex in the front of the brain. Everywhere we looked, we found that there was this enhanced gamma power in these red lines here and superficial layers and enhanced alpha beta power in the deeper layers. And what we then turned to next is, first of all, well, let's really confirm at a histological anatomical level that it's really these layers. Is it layer one? Is it layer two? Is it four that expresses gamma? And the same question for beta, let's really confirm that that is, in fact, the case. Because it seemed like a very prominent hypothesis and a very prominent observation. We wanted to really confirm it. So that's the first part of this part two. And then we're going to get into which specific mechanism might generate these oscillations and how does that inform computational modeling? And so then this was worked out in my lab at Vanderbilt, led by a very talented undergraduate student, actually, Max Liffenfeld. And so we performed these histological reconstructions. Here's the probe, an example probe, where we performed these tiny electrolytic lesions in particular areas while we were recording. Here is the overlay of the laminar boundaries. So in red here, this is layer four. And then, of course, we had the electrophysiology. So we could look at the gamma power, gamma relative power, the beta relative power. And the point at which they cross over, actually, we determined was pretty consistent in layer four. And of course, then we also mapped on the other features of these things to the layers. And so here's what we found is that the crossover, the point at which the gamma profile in red and the beta profile in blue, the point at which they cross over, at which their powers are equal, that tended to land in layer four. The peak gamma power tended to land in layer two or three and the peak beta power tended to land in layer five or six. And so we mapped these distances between these different electrophysiological signatures or distinct profiles to the different layers. And we did this for several different probes and for several different animals until we were able to get a very nice distribution here, which confirmed that indeed the gamma beta crossover is consistent with being at laminar position zero, we call the middle of layer four. Whereas this gamma is at the boundary between layer two, three and beta at the boundary between layer five and six. And so here is our current toy model of what we think is going on. So we think that there's different inhibitory neurons that are interacting with their excitatory cell partners to generate fast frequency gamma activity in the superficial layers. And in turn, that there is enhanced, that there's another type of interneuron that is generating with slower time constants, this alpha beta rhythmic activity. So in order to examine this, we again turn to anatomy, we again turn to this question, well then which exact inhibitory neurons are most prominent therefore in the layers that express these different frequencies? And in order to perform this work, what we did is we quantified the laminar expression of three distinct interneuron subtypes that are known to account for over 90% of the inhibitory interneurons in the primate brain. And those inhibitory neuron subtypes are parvalbumin positive, calretin and calbindin positive. And importantly, previous computational modeling work had really singled out the parvalbumin cell class as the critical class for generating these gamma oscillations because of their fast time constants and strong synapses on the cell bodies of excitatory neurons. And so again, we performed this painstaking histological analysis where you perform different stains, mounting, cover slipping, microscopy, semi-automated thresholding and cell counting in order to be able to then really have a quantitative view on which layers have which cell types that are most associated with these frequencies. And so here's what this data then looks like. We have the gamma and alpha, beta electrophysiology here. And then at the same time, we have the presence of different neuron types with the three inhibitory cell classes here. And then we can perform an analysis to determine which of these cell classes is most strongly associated with the gem. And here's what we found. But in both the low gamma and the high gamma across over 14 regions, what we found quite significantly was that it was really the parvalbumin cell counts shown here as a function of laminar depth that were giving us the best explanation for the laminar profile of the gamma rhythmic activity, both when you take a very wide band as well as when you perform a low versus high gamma split. It comes out in both ways. So how can this inform computational models of oscillations? Well, here's what we think is going on then. We collaborated with several people here on this work, including Sanchez Toto at all and computational modelers in Spain. And what we did was we performed an analysis where we took different interneurons with different types of connectivity. So these PV, these parvalbumin interneurons were modeled as having strong inhibitory connections to the pyramidal cell class and strong self inhibition, which has been important for gamma oscillations previously. And when you hook them up in these particular ways, sure enough, and you measure the spectral output of these neurons, these P2 neurons, superficial pyramidal neurons, sure enough, they express gamma. And when you put in different types of interneurons of slower time constants and different types of connections to these deep layer neurons, sure enough, they come out with more alpha-beta oscillations. So indeed, this anatomical work could produce even more, in the future, even more accurate models of how these interneurons work together and with pyramidal cells across different layers in order to express these oscillations. So this is really work in progress. And one of the first steps towards this, and our ultimate aim is to really pull together this rich anatomical and neurophysiological data together with computational models to be able to approximate the laminar structure of cortex to a better and better degree. So this next part here, I'll talk a little bit about a causal test now of the predictive routing model with propofol anesthesia. So this will be the last part of the talk here. And so we were interested in anesthesia. I got interested in anesthesia actually through my work with and contact with Jake Donahue over here who was a very talented MD PhD student at MIT at the time and was being co-mentored by Emery Brown and Earl Miller, who was also my mentor. And so we became interested in this topic for the following reason. Well, first of all, as surprising as this may seem if you go into a surgical clinic and you're administered an anesthetic such as propofol, well, it turns out that the anesthesiologist may not know exactly the cortical mechanisms and the systems level mechanisms for how that anesthetic is actually working in the brain and it's actually, to this day, remains more often than not the case that there's actually no intraoperative monitoring of brain state. It's far and few between the anesthesiologists that are actually doing that. And so Emery Brown, who's an anesthesiologist himself, is really pushing for more and more adoption of simple tools like EEG and spectral estimation methods by the clinic. And so our work was clinically motivated, but it was also motivated by the ability to test this predictive routing model when activity, especially in prefrontal cortex areas, is relatively silenced. And so in these experiments, we administered propofol to, again, these Rhesus macaques and we recorded brain activity. Here's these local field potentials and spiking activity in the awake state as well as in the unconscious state. And what we first noticed is that in the unconscious state there were these slow rhythmic bouts of activity at about one hertz, one cycle per second or slightly little faster, a little slower than that. Those are the slow frequencies and these slow frequencies organize these massive bouts of spiking, which could sometimes be very synchronous across cortex. But that there were long periods of silence in between these bouts of spiking in the unconscious state. And so propofol anesthesia we found is accompanied by, again, these strong increases in slow frequency activity, which create alternating on and off periods for spiking. There's reduced spiking activity throughout the network, but the strongest reductions are in prefrontal cortex and sensory areas are relatively spared. So they're still quite reduced in their excitability in the unconscious state, but prefrontal cortex is more inhibited. There was also we found reductions in cross area synchrony and very pronounced reductions in the alpha beta band. And so as a result, we thought, we could actually use this model, this propofol anesthetic model in order to test certain aspects of predictive routing, because what's going on here? We have lower order cortex, that's hypothesized to feed forward prediction errors, the higher order cortex. And in turn, these higher order areas might feed back their predictions. But as a result of propofol, these higher order areas are profoundly impaired and very reduced in their spiking activity. And the alpha beta rhythm is also the synchrony between areas is highly reduced. So what's going to happen in this case? Well, if you think from the point of view of simply everything is going down, then sensory areas in a response to an unpredictable stimulus should also just go down. But if you think from a predictive coding or a predictive routing point of view, the function of these higher order areas is to create feedback predictions that inhibit activity. So in the absence of those feedback inhibitory activities, you might actually hypothesize that there would be an enhancement of the feed forward response when that feed forward response is characterized by an unpredictable stimulus or prediction error. And so we sought to test this out using the simple sensory prediction paradigm where, oops, we have a tone stimuli. These tone stimuli could either be repeated or occasionally there could be an oddball stimulus or bbbbbb. And what we would expect is that there would be enhanced feedback predictions when stimuli were repeatable and predicted and that the occasion of an oddball would signal an enhanced gamma feed forward signal. So we performed the auditory stimulation task both in the awake state as well as after there was an administration of purple fall leading to loss of consciousness. And here's what we found in the local state just as we found in the visual cortex for an unpredictable visual stimulus as in the visual cortex, so in the sensory cortex, my PhD student, Sophie Shong who performed this analysis found an enhanced gamma. So right at the time of the local oddball. So we have time here on the x-axis, frequency on the y-axis in the spectrogram, hot colors mean more activity to an unpredictable stimulus. So there was an enhanced gamma response to this unpredictable stimulus, decreased alpha beta to the unpredictable stimulus or set another way when the stimulus was predicted, there was more alpha beta. And what here's what happened in the anesthetized state when we performed this exact same contrast, we lost the predictable alpha beta signal and instead everything was enhanced. So there was a decrease in frequency towards the low gamma range. So the cortex just essentially went into a constant ringing in the gamma frequency range, which to our surprise was much stronger in power compared to the gamma that we saw being enhanced during an unpredictable stimulus in the awake state. And the way we interpret this is that in the absence of this inhibitory predictive alpha beta signal, this gamma signal becomes uncontrolled and enhances itself in the local anesthetized state here in area TPT is a sensory area, should have mentioned that. So now to conclude, we propose predictive routing. So predictive routing is a neurophysiologically inspired implementation of predictive coding which stresses the importance of these alpha beta signals for implementing predictions and inhibiting sensory areas and areas of the brain that are experiencing a predictable stimulus. In the absence of that, you get an enhanced feed forward gamma output. And this distinction between gamma and beta across layers we found was ubiquitous across cortex, really suggesting that something like this may in fact be a truly a canonical circuit motif that's repeated across the brain. And these alpha beta rhythmic predictive preparation or inhibition we found we observed was lost in the unconscious state with unconsciousness being mediated by propofol. By the way, it's one of the most often used anesthetics and that as a result of this loss of predictive alpha beta, there was an enhanced gamma modulation to these unpredictable inputs. In other words, the sensory cortex was in a disinhibited state. And so with that, I'd like to thank my many collaborators and the Vostos lab at Vanderbilt as well as collaborators on the anatomy and collaborators at MIT as well as my financial support. And lastly, we just started the lab about two years ago where we're still in a recruiting stage. So you guys can check out my website, my ex slash Twitter feed and an email me if interested in contributing to this line of research. And so with that, I'll turn it over in case there are any questions. Thank you for your attention. Awesome. Thank you, Andre. I'll just ask one quick question from the chat. Dave Douglas asks, does propofol change the activity of neuronal microtubules in any interesting way? Thanks, Andre. I'll just ask one question from the chat. Dave Douglas asks, does propofol change the activity of neuronal microtubes in any interesting way? I haven't looked into the microtube kind of hypothesis. That's one of the hypotheses advocated for many years by people like Stuart Hammerhoff. And there has been some recent work done by George Massure, I believe. And so you should go check out some of his papers on that topic, but I haven't looked into that. And I remain somewhat skeptical to the quantal explanation for consciousness, but mostly because the mechanism doesn't appear at first glance to be very biologically reasonable. And also that there's not a lot of, there's essentially not very much data on it, but that may change in the coming years. So something to look out for. Great, one other question. More generally, how do you see experimental work in the active inference ecosystem and how do you see the synthesis of theoretical and in principle work with this kind of real wet lab and real recordings on the neurons type analysis that you're doing here? Yeah, so it's a great question. I think that what we need to be doing as a community is having more interactions between theorists and experimentalists. And so we wouldn't have performed, for example, this experiment here if we hadn't already been inspired by biological and theoretical models of predictive coding and of active inference as well. And so that's really what we need and it is to make not only hypotheses that are turned into great theoretical and computational infrastructures and models, but we also need to be thinking about, if we care about the biology and we care about trying to figure out the brain, then we should also be trying to map those things onto neurobiology and psychology. And so that involves them both considering the neurobiological elements that may implement and mediate active inference in the brain as well as the psychological and cognitive constructs that we think are going to be particularly sensitive to capture certain elements of the theory. And so I remain very welcoming to that type of suggestion. And in fact, there's going to be a postdoc joining the lab soon, Eli Sinesh, who has been working very deeply in the active inference community. And so we'll have an opportunity there to both work on theory guided data analysis but also perform more theoretically guided experimental designs that can go further into the neurobiological implementation. And so this was here what I've presented really simply the first and small somewhat step towards linking the neurobiological implementation level with the algorithmic and computational level of predictive coding and active inference. And so the more interactions that we can have between the communities, the better that that will be. I think what would be somewhat of a shame is if the active inference community went way too far into just pure theory and that and delved into increasingly areas that were divorced from the biology. So that's of course one of the great critiques of the deep learning and more machine learning, purely machine learning driven approaches is that they have been, they took a little bit of biological inspiration but then started using algorithms like backprop that were very biologically unrealistic. And so of course active inference is a response to that in certain ways but in order to continue being that, I think that there needs to be indeed a tight dialogue between experimentalists and theorists. And so if you have any particular ideas I'd be happy to hear them. Please reach out to me by email or through X which you can find here on the slide. That's really cool. That's awesome. You're gonna be working with Eli and one final question. This is from Alexandra. Is it possible to detect the contribution of glia to the neural firing rate of the field potentials? That is a question that I have not considered yet. So we normally do not consider glia to be active in the network but of course that is likely to be an oversimplification. We know that they can have interactions with neurons. We know that they can also have different levels of excitability and of course that's how they can regulate neurons metabolically but as far as the contributions of glia directly to the local field potential I am not aware of how to determine whether or not they make a significant contribution. I actually haven't seen any work on that but thank you for the question. It's always good to reconsider our core assumptions about the cell types that are most important to capture. So it's certainly something to think about. Thank you. Thank you. All right. Thank you, Andre. Super cool work and good luck with everything. Till next time. Thank you. All right. Bye. Bye-bye. All right. Next up is going to be Keith Duggar who has submitted a prerecorded video. The prerecorded video is called Active Inference and the Actor Model. So I will bring that up right now. Okay. Here comes Keith's prerecorded video on Active Inference and the Actor Model. Active Inference and the Actor Model. Hello, hello. I'm Dr. Keith Duggar, platform CTO at X-Ray and extended reality AI company and co-host of machine learning street talk podcast. Now, as we all work to build an ecosystem based on Active Inference, software will obviously play a foundational role. To make the most of Active Inference, we'll need to use software engineering paradigms that align with the principles of Active Inference. And I think there's one Taylor made for our needs. It's called the Actor Model. Active Inference and the Actor Model are two deeply connected understandings of the world. They provide foundational frameworks for dealing with the dynamics of complex systems with a focus on autonomous agents that interact in an ecology of nested systems. I'd like to force some of their key connections, including the role of agents, concurrency, autonomy, uncertainty and behavior adaptation. We will see that Active Inference and the Actor Model are both paradigm shifts away from a deterministic centralized step-by-step thinking to a decentralized network, concurrent perspective of both complication and cognition. Just a little bit of history about the Actor Model. Back in 1973, Carl Hewitt, Peter Bishop and Richard Steyer were all working at the Massachusetts Institute of Technology AI Lab to fundamentalize a concept of concurrent computation that included both structure as well as adaptable algorithm execution. Conventional methods at the time lacked robustness and secure mathematical foundation. Their collaborative effort ultimately led to the creation of the Actor Model. At the time, it was viewed as revolutionary due to its characteristics of both increased error tolerance and distributed computation abilities. Throughout the 1980s and the 1990s, the Actor Model became the basis for numerous research projects, as well as practical projects gaining popularity for its flexibility and intuitive approach to concurrent computation. It was finally used from artificial intelligence and multi-agent systems. Sound familiar? New actor-based languages like Actor, Salson, and Erlang contributed to the refinement of the Model shaping it into a more robust and flexible approach to concurrent computing. And it remains alive and well in computer science today. More recently, the Actor Model has gained renewed interests mainly due to the growing need for distributed systems, cloud computing, and edge computing, fueling the Internet of Sains and Web 3.0. These computer tasks are ideally suited to take advantage of the Actor Model's architecture, which is designed exactly for modeling concurrent handling of both large volumes of data on the one hand and fine-grained disparate autonomous systems on the other hand. This application of the Actor Model has had profound effects on major companies that have utilized its principles to handle big data problems, such as Twitter, Facebook, LinkedIn. So, what does this have to do with Active Inference? I'm guessing you've already heard some of the parallels in the intro. Well, let's start by looking at a few core principles of the Actor Model and how they relate to the principles of Active Inference. Let's start with the concept of isolation. Isolation means that an actor in the Actor Model does not share its state with any other actor. It can only be affected by receiving a message, and it can only affect change in the state of other actors by sending a finite number of messages in response. From a software engineering point of view, this isolation principle limits potential side effects of an operation to a single actor, thus improving the system's overall predictability, the liability, and most importantly, if embraced fully, can actually simplify the design. Looking at the diagram, we see an ecosystem of actors sending messages to a particular actor, which in turn sends back messages to other actors. Where is Active Inference? Well, let's recast receiving and sending messages to a perception action cycle, and denote external, internal, sensory, and active states. And we now clearly have the necessary foundation of Active Inference, a Markov blanket. The actors of the Actor Model mount directly to the agents of Active Inference. In addition, the finiteness, the fact that an actor can only send a finite number of messages in response is also an important shared property. Because Active Inference models reality, it necessarily respects the resource constraints of real systems, and this is nicely bathed into the foundation of the Actor Model. Let's look at another core principle, asynchronous message passing. Communication between actors is asynchronous. This means an actor doesn't wait for a response after sending a message. It continues working. It continues living as it were. This is critical as it decouples the actors leading to a system that can continue functioning, living, and making progress, even when parts of the system are slow, or even temporarily unavailable. Professor Friston has said that the three energy principle is the ultimate existential question. If things exist, what must they do? Well, the Actor Model claims they must not wait on others. Of course, an actor can choose to wait on others, but it must not be forced to do so. In the model, it must be free to choose. This leads us to another critical principle that both models share, autonomy. The Free Energy Principle is a model of physical reality, and our reality is, after all, comparing. All throughout infinite space, systems are evolving simultaneously according to their local dynamics, and therefore, this is reflected at the heart of the Free Energy Principle. Of course, a model of computation may not constrain itself to physics, but he would at all, we're seeking to develop a model that did model the reality of distributed concurrent systems. And, luckily for us, the Actor Model embraces both concurrency, as seen from the principle of isolation, and after autonomy, making it compatible with active inference. Next, we come to nesting. The Actor Model allows for an actor to not only receive and send a finite number of messages to perceive and act. It also allows as an action the creation of a finite number of new actors. These actors can either be nested within the parent, say the parts of an animal cell, or it can be released into the environment as independent actors from then on. This principle model fits nicely with the beautiful concepts of multi-scale nesting and active inference. This allows for actors to contain ecosystems of actors, both all the way down and all the way up. Last, I want to cover two more Actor Model design principles, behavior change and persistence. Actors have the ability to change their behavior in response to a message. This adaptability allows for the construction of complex, stateful entities that can evolve over time. And indeed, it allows for entire ecosystems to evolve new, emergent behaviors. When used for software engineering, this adds a powerful tool for managing complex dynamic systems. Active inference, of course, embraces this to the extreme. The very essence of Thingus is the ongoing attempt to predict and adapt to an environment and thereby continue to exist to maintain one's more complicated and a broiling sea of activity. Along with this also comes the concept of persistence. Persistence allows actors to save their state and to restore or modify it later. A feature that embodies the principle of memory. Memory is a prerequisite for learning and adaptation. An agent's ability to predict depends on its ability to remember past experiences and thus minimize the surprise associated with unexpectedness. The vital role of memory is also emphasized when we assume that agents have inductive priors either from experience or inheritance contributing to their world model. This world model both guides their current behavior and it is continuously updated based on new experiences contributing to their ongoing adaptation and existence. Okay, great, you say. There are clear and deep connections between the actor model and active inference, but how does this help us in the active inference community? Well, firstly, in my opinion, it is a software engineering paradigm we should embrace. And if we do, there are, of course, actor model libraries and frameworks that we can use, such as ACA, Orleans, Thespian, Actix, Protoactor, and many more, which we can immediately use when building active inference software modules and applications. There are also libraries, languages, and even language features that align very well with the actor model principles, such as Zero and Q, Tokyo and Rust, Erlang, Async Await and C-Sharp, et cetera. But more important than the tools available to us today is the software design mindset that will guide our creation of the active inference software of tomorrow. The actor model provides a paradigm of software design and engineering that is the most perfect match that we have for active inference. This is evident not only from the alignment of the core principles we previously discussed, but also from the insights that active inference in the actor model bring to each other. For example, consider what is now called Hewitt's Law, informally stated as everything is everywhere. This law signifies the idea that in a truly asynchronous distributed system, it can take an arbitrary amount of time for a message to go from one place to another. And any actor must be prepared for that event. There are simply no such thing as instantaneous in such a system. And no component can make an assumption about the timing of another component's actions. In fact, one must act as if a message may never arrive. This has important implications. It's just that it is impossible to accurately and consistently determine the state of the entire system at any given time because the information just may not have even propagated across the system. And also attempts to implement global synchronization will inevitably introduce bottlenecks and reduce efficiency. Hewitt's Law emphasizes the need for systems to be designed in a way that they can effectively handle these unavoidable delays and uncertainties, highlighting the importance of robust, non-blocking communication mechanisms and local decision-making abilities. In short, after model systems are by nature, non-deterministic. Does this sound familiar? What other paradigm highlights operation under uncertainty and the autonomy to continue despite the environment? Active inference and the pre-energy principle. Active inference reflects the reality of an unpredictable world in which our software systems operate. Different outcomes may result from the same initial conditions due to the occurrence of events in a random, unpredictable order. This is the concept of surprise that we all know well, where an agent updates its beliefs about the world when the sensory input it receives does not match its predictions. Both the actor model and active inference acknowledge that the world is unpredictable. Even more than acknowledge it, the models accept this uncertainty as a given and not something to be managed away. Indeed, as we know, in the free energy principle, the uncertainty that we maintain in our models is what gives us the flexibility to adapt. Perhaps this is just my personal flight of fancy, but I imagine a future where software modules guided by active inference do away with hard-coded error handling and swap in probabilistic learning algorithms that optimize themselves as the error landscape evolves. Modules that are robust and self-healing, distributed systems with no single points of failure that focus on predicted disaster avoidance rather than reactive disaster recovery. Looking towards the future, we as a community have the potential to push the boundary of both active inference theory and practical implementations of the actor model. By leveraging the strengths of these two paradigms together, we can create software systems that are robust, adaptive, and more aligned to the physical world in which they actually operate. Imagine the future where software components using active inference in the actor model can anticipate potential issues, learn from past mistakes, and adapt in real-time to environmental changes. With this approach, we can build systems that are fundamentally more resilient and more efficient. In my opinion, this can bring a step change in software reliability and performance and scalability, and heralds a new era of computing, weaving principles of biology and cognition into the fabric of our software systems, bringing them closer to life in the process. In conclusion, the coupling of active inference to the actor model provides a powerful new lens through which we can look at software design and engineering. Whether we leverage existing languages and libraries aligned with active inference, or invent new ones, we are standing on the brink of an exciting frontier. So, let's seize the day, have a look at the actor model and its relationship to active inference, and let's shape the future of intelligent distributed computing. Thank you for listening. Awesome. Great talk by Keith. Thank you, Keith, for sending that in. There were some comments in the chat. So, Keith, potentially if you want to join for a Q&A, some future time, but really cool presentation. All right, the next presentation is by Sanjeev Namjoshi. And this presentation is going to be called Developing NextGen Active Inference Tools Broadening Accessibility, Educational Resources, and the Software Ecosystem. I'm going to start this talk right now. Hello, everyone, and thank you for being here. My name is Sanjeev Namjoshi. I'm going to restart it. I'm a machine learning engineer working at the AI services firm. Just because it's a little quiet. All right, restarting the talk. Hello, everyone, and thank you for being here. My name is Sanjeev Namjoshi, and I'm a machine learning engineer working at the AI services firm Kung Fu AI, where I primarily focus on computer vision projects. Today, I'm going to be talking to you about my progress toward providing greater accessibility, visibility, and knowledge of active inference and the free energy principle. I'm excited to be presenting this material at the Active Inference Symposium this year because the idea of an enacted ecosystem of shared intelligence perfectly captures the philosophy that underlies the projects I'm currently engaged with. I've spent the last seven months on sabbatical for my job to work exclusively on an active inference textbook and related tools, presenting chapter presentations, code reviews, and receiving feedback weekly at the Active Inference Institute. The Institute has provided a space where interdisciplinary research can flourish as the connections and influences of active inference spread to other fields. It has consistently fostered the spirit of collaboration and shared intelligence that I wish to embody in my own work as part of this ecosystem. I intend to continue closely working with the Institute to provide materials that will bring active inference to a much wider readership. I originally chose this project when I saw the great potential in the active inference field and couldn't help but make a comparison to the state of deep learning in 2006 when neural networks are one of just many possible models rather than the dominant choice in academia and in industry. This, of course, all changed in 2006 when Hinton and his colleagues released the Deep Belief Network paper, which is generally understood as a start of deep learning as we understand it today. After some hardware innovations and the release of the well-known ImageNet library, we started to see coverage and success of AI in the news as well as in academic research. But in 2012, deep learning truly provided its value with AlexNet, and for the first time, deep learning achieved better-than-human performance on image detection tasks. But this was just a start. What followed was a proliferation of deep learning all across industry and research. I've added in some well-known milestones just to highlight the explosion of progress in deep learning in the last decade, though there is so much more here that we could discuss. So what about the state of active inference as a field? From my perspective, active inference lies in the same position as deep learning in 2006, influential and on the brink of exploding in popularity. This paper, which is from contributors at the Active Inferenced Institute, shows the current growth of publications in the institute and its community in active inference. In the last three years, the active inference field has seen a number of important milestones. Here, I show just a few that broaden the scope and attention to the field. We had the first international workshop on active inference in 2020. We had the first active inference symposium and the founding of the Active Inference Institute, then called the Active Inference Lab. We had the release of the PAR et al. 2022 textbook and the PiMDP Python package. And I see myself now as perfectly poised to bring active inference to greater visibility and attention. This is in part because of the current academic interest and deep reinforcement learning and generative modeling. Working alongside the institution and other organizations, my aim here is to provide some of the fundamental materials to capture the attention and notice of machine learning researchers and students to bridge this gap to bring active inference into its renaissance. To this end, I've been working for the past seven months on sabbatical to finish work on a comprehensive textbook. The aim of the textbook is to provide the tools to bring active inference to a wider audience, primarily those in machine learning research in applied fields such as robotics, and to decrease the challenge in learning the material, largely by separating it from much of the neuroscience background that is usually a prerequisite. This decrease in prerequisites means labs will have to spend less time helping students becoming acquainted with the field, and researchers outside of neuroscience will find this book an accessible entry point that uses terminology familiar to machine learning rather than neuroscience and fMRI image analysis. All derivations are in one place. Currently in the field, many derivations are spread across different papers, even behavioral papers instead of just technical ones. And it's hard to know where to look if you want to understand a particular equation or concept. Part of the success of deep learning in the last decade has come directly from focusing on narrow improvements to specific aspects of the modeling. There are many open questions in areas of research, such as how to prune policy trees, exploring second-order optimization rules for state and parameter updates, and scaling active inference. The increased accessibility for researchers would also lead to many new industry applications, such as autonomous vehicles, robotics, video game design, and AI. The textbook would also put a spotlight on Bayesian mechanics and invite contributors and contributions from researchers as this exciting nascent field grows and develops. Part four is largely a literature review and can be very helpful to those writing about active inference from fields such as philosophy, sociology, and many others. And the historical context sections that are part of this book provide a lot of that context as active inference is built upon decades of research in neuroscience, psychology, and many other fields and also draws upon current work in many fields that have emerged in the last 25 to 30 years. Finally, LaTeX reproducibility may offer interesting ways to rearrange the book and integrate it with the code for an online-only experience. Now I'd like to share with you some of the progress of my textbook and the general structure. The textbook is divided into four parts. The first part introduces fundamental concepts to set the stage. In particular, I have focused here on presenting well-known statistical ideas from the perspective of an agent modeling its environment, who states it must infer from an observed noisy signal. The second part focuses on continuous and discrete state-space formulations of active inference, where the algorithm of focus for the continuous state-space formulation is active generalized filtering. Part three, which I'll begin writing in a couple of months, focuses on a sketch of Bayesian mechanics and the required background designed with the knowledge this field is still dynamically changing and evolving. Here, I will focus on some of the fundamental concepts and ideas as well as code simulations to allow readers to get a deeper and more intuitive understanding of some of these challenging ideas. Finally, part four is a systematic literature review that covers all the various applications and extensions to active inference that have been innovated in the last six to eight years. These applications include things like robotics, all the behavioral modeling and neuropsychiatry and human and animal behavior, theory of mind, and so on. The extensions talk about how applications of active inference can be used to talk about dynamic systems more generally and apply to things like ecosystems and to economies and other types of things like governance and so on. As of this week, the rough drafts of part one and two are complete and ten chapters have been presented to the Active Inference Institute. In support of this textbook are four separate tools that I will discuss over the next few slides. But before I do that, I would like to first highlight some special aspects and features of my approach to this textbook. The major focus is on writing this book for a machine learning audience or students learning in this and adjacent areas. Neuroscience is out of scope for this book. Many of the recommended and even optional prerequisites that are shown here are typically known by undergraduate students in science and engineering and certainly by graduate students in these fields. Nonetheless, the book is written to be readable by those that desire to focus on everything that is the math, the code, the concepts, those that just want to focus on the math but are not interested in implementation and even those that may skim over the math and just try to understand the ideas intuitively. One thing that's very important to me and trying to express these ideas clearly is by spending a lot of time working on typesetting and style, which is very important to successful learning. So I've spent a lot of time attempting to make my work clear and readable. To this end, I have margins which collect specific key terms for references later, which you can see in some of these figures that are shown here. And these terms will eventually correspond to the ontology project ongoing at the Active Inference Institute. Margins also provide further explanation to accompany the text and this will be useful to readers who want more detail and explanation. There's a large focus on building an intuitive understanding of the concepts. For example, importantly, all algorithms are explained from scratch. And this book, we typically start with a description of the agent environment modeling problem. We then start the book with a extended the multivariate case. Then we introduce variational inference. We add dynamics, generalized coordinates where applicable, hierarchical models, action, and also learning and other modifications we might add to our models. We have a very big focus on figures clearly walking the reader through the text and giving detailed visualizations of important concepts. And in terms of how the textbook is set up, the early part of the book focuses on basic concepts such as hidden state estimation. That is estimating the conditional distribution of a latent variable given some observed data. The aim here is to explain the modeling paradigm in the context of an agent attempting to infer the states of the environment. That is the interaction between a generative model and a generative process. A perspective that differs from the Bayesian inference style that is normally taught in universities and many introductory textbooks. Typically, introductory textbooks on Bayesian inference focus on parameter estimation or learning. And part one introduces the expectation maximization algorithm as a way to explain the connection and separation between hidden state inference on the one hand and parameter learning on the other. Additionally, a large focus has been placed on variational inference which is explained in detail. It introduces a catalog of all the different forms of variational free energy and expected free energy in the literature and how they can all be derived from one another. The book also covers raw and ballard style predictive coding terms and ideas such as key ideas such as prediction error minimization as well as clear and intuitive explanations of fundamental concepts such as surprising. The textbook focuses heavily on building intuition through derivation and the general flow of most of the chapters is to set up the problem that needs to be solved which is defining an interaction between the agent and the environment showing the elements needed to solve it which is usually random variables and parameters that form a joint distribution or generative model replacing probability distributions with their algebraic formulations and then moving through the algebra to a final analytic or gradient based equation. The reader should be able to recognize most equations in the literature upon reading this book. I'm also making extensive usage of Bayesian networks and other types and styles of graphical models such as factor graphs will appear in part 4. There are hundreds of custom figures that have been created so far and the figures are detailed to give the readers a deep understanding of the different types of content that is covered throughout the books and also summarizes much of the information and equations that are pervasive throughout the active inference literature. Another important focus of the textbook is that many of the models that are presented are also shown in pseudocode which should aid the reader in implementation. And finally each chapter is filled with numerous what I call experiments and these experiments correspond to the Jupiter notebook and try to show the application of a concept in a simulated environment. So a lot of these experiments start out by generating data so we have some kind of generative process and then we have that data passed to a generative model or the agent which then attempts to either perceive and learn from that data and even act on it. And the example that's shown on the right margin here this is just a perception problem on a continuous grid which has been divided into pieces for the purpose of the simulation represents a continuous state space and the agent in the bottom left corner shown as a mouse has a prior belief about where some reward food is in its environment but it then needs to use perceive from sensory data that it observes the true location of that reward or food which is obscured or occluded in some way by the mist that's shown in that figure. So these types of experiments give the reader a better sense of how to apply these statistical ideas to a real world situation so we can understand how it might apply to some kind of autonomous agent. Next I'd like to cover and shift my attention toward Jupyter Notebooks and videos and I'd like to note that upon publication of the textbook these Jupyter Notebooks will be released on GitHub and should be fully reproducible using Docker and other version handling tools. One of the big emphasis of the Jupyter Notebooks is it has to be direct correspondence between the equations and explanations in the code and in the text. This will build a direct understanding and show applications of the concepts explained. Notebooks are filled with simulations and visualizations many that appear in the main text and in addition to that I have also over the last seven months been presenting chapter presentations to the active inference institute. So far a draft version of the first 10 chapters of the book have been recorded at the active inference institute and these are just some sample slides that I prepared that try to explain these concepts in great detail. In the final stages of writing this textbook video lectures will be re-recorded and released alongside the book. I also plan to create detailed code walkthroughs videos that walk through the different examples in the Jupyter Notebooks. Now I'd like to talk about a few planned future resources I'd like to work on after the book is complete or toward the final stages of the book to have further support and educational tools. Some of these planned future resources include a software suite in Python to enable an alternative learning approach for those who do not wish to learn about the algorithms from scratch and this will expand the possible landscape of engagement. As PyMDP already exists I will not be working on a discrete state space Python package but I'd like to fill the space for things like active generalized filtering and also just an availability of different types of simulations of Bayesian mechanics as it's currently defined today or at least the different versions and varieties of some of those key concepts. I'm also very much interested in interactive learning and my aim would be to have these preset simulations concepts that are explained in text with various simulations interspersed in the form of plots and demos and other visualizations and the idea would be that the user could manipulate sliders and knobs to tweak various parameters that would help aid in learning as they get a feel for how these systems behave especially ones most of the systems we talk about are dynamic so seeing how they change over time. The Active Inference Institute has made tremendous progress in the past couple of years to provide a collaborative environment for researchers and for students of Active Inference. I hope to be part of this ecosystem as I continue to support the spirit of accessibility and collaboration and I'm excited to continue to contribute to this ecosystem of shared intelligence and look forward to what we can build together. I would like to thank the Active Inference Institute for hosting my presentations, code reviews and feedback sessions and inviting me to present this symposium and also thank my employer, Kung Fu AI for letting me take time off to write for the past seven months. Please feel free to contact me at any time. Email is the easiest way but I'm also available on the Active Inference Institute Discord. If you would like access to the textbook and related materials, please send an email requesting access and I can get you set up. And that's all I have for today. Thank you very much. Awesome. All right. Thank you, Sanjeev. Greetings, Ines. Oh! Welcome. Hi, Daniel. Thank you. Yes. Well, we're really looking forward to your presentation. You can share your slides and go for it. Great. Right. Amazing. Can you see it? Yep. Looks great. Thank you. All right. Good. So, thank you so much for having me. I'm really happy to present this paper today and particularly because it's the very first time that I present this work. So, I'm really looking forward to any feedback and any reactions to this work. Okay. So, the aim of this paper was to apply the free energy principle to analyze the interdependent mix between living systems on the one hand and non-living planetary processes. And this hopefully would, we hope, allow us to gain some insight into the future states of Gaia. And by Gaia, we're going to mean the Earth and formulate some predictions and devise potential interventions to alleviate future dysfunction on the planet. Okay. So, then the lineup for the presentation is, I'm first going to go over in brief the theory, the concept of Gaia, so known as the Gaia hypothesis. Then I'm going to go into just giving some essentials on the free energy principle which I'm assuming that it's not really necessary but just for the sake of the flow of my argument, then I'm just going to present some things on the free energy principle and specifically on self-organizing interacting systems. And then I'm going to apply the free energy principle to the Gaia hypothesis and the comparison analysis is in how do they if at all complement each other for epistemic gain. And then I'm going to, after having done that, this should be to allow us to inform or to gain some insights as per the human behavior in relation to Gaia or the planet Earth. Okay. So, let's see how it goes. So, first I want to introduce the notion of biophilia which plays a fundamental role in this work. So, biophilia came first or came more prominently in this particular book here and it's quite interesting. The idea is that the relation of humanity to the rest of the natural world is essentially one of dependence and this is very much already in the free energy principle but more than that there is this view of the necessity of a continued survival and flourishing that is utterly contingent upon the state and functioning of the environment. So, we're talking about things like very utilitarian things like survival, flourishing, mental health, right? So, individuals interact with the environment so to achieve that. But then on top of that we wanted to add another conceptual layer which is of concerns how humans possess an innate cognitive disposition towards nature which is known and developed in this particular other field of work as biophilia and this is the idea of the natural psychological disposition of humans to seek out and connect with other living organisms and the natural environment. So, that's kind of like the idea that is encapsulated into the whole framework and theory under biophilia. So, then this seems to, if that's the case of as things are described by biophilia then that seems to bring us to a dilemma. On the one hand we have this natural tendency so to speak of humans to engage with the natural world in a very positive and constructive way and on the other hand we see the lack of action to protect the earth and here all you have in mind particularly the case of the climate crisis. So, in the presentation I'm going to make my way slowly towards getting to that and to better understand the dilemma. Okay, so let's go in brief over the Gaia hypothesis. So, this is a concept that comes more prominently in James Lovelock's book, Gaia A New Look at Life on Earth and the idea there is that 3.8 billion years that life has persisted on earth and conditions have remained remarkably constant and favorable across a huge range of parameters. The Gaia hypothesis is that this consistency or constancy to be is a natural product of evolution. In a modification or weather of the canonical Darwinism the Gaia hypothesis somewhat modifies canonical Darwinism how? Well, by thinking that rather than evolution of the individual organisms on an inert environment what evolves is the whole earth system with its living and non-living parts existing as a tight coupled entity. So, this is the kind of so to speak refreshing or new way of thinking that somewhat reformulates a little bit of Darwinism. The emphasis is not how one individual organism evolves or species evolves but the whole earth as a system. So, that's quite interesting because for example the view that combined genetic wealth of information of the whole system becomes much more prominent here than the Darwinistic kind of view and these acquire characteristics that are conductive to its continued existence and in turn affect changes on the environment. So, then seen as a whole organism or as a whole system one can look at Gaia or the earth as a complex system which appears to have the goal of regulating the climate and the chemistry at a comfortable state of life. And these views and these ideas have come to gain a more official expression in 2001 with the Amsterdam declaration on earth system science precisely endorsing this view on the planet earth as a whole system. So, then the Gaia hypothesis is a complex system in various ways because it changes in an adaptive and non-linear way it also displays dynamical, emergent and sudden tipping behavior and when bifurcation points are reached the new tractor states form new tractor states are formed and this means that it also self-organizes and there is interdependence between the processors of the system. And finally, the Gaia hypothesis seems to be quite interesting and extraordinary in the sense that it brings together so many different fields such as ecology, climate science, cybernetics complex systems theory chaos mathematics and flaws of your mind and that's just a few. Alright, so then the idea is of the Gaia hypothesis is that we could look at the planet from orbit or as it is known the overview effect and the Gaia hypothesis offers this glimpse of a fundamentally different way of thinking of the planet of thinking that in fact dominated mostly by human history. So it's a little bit of like that very inspiring clip by Carl Sagan the pale blue dot whereby the world is considered so throughout the human history there's typically a tendency for the world or the planet to be considered from a single living organism though rather than an it. So that's the idea of the overview effect and taking this look at the planet as this whole living system rather than an it. Okay, so then we can think about the Gaia hypothesis as like Newtonian physics or even as the pre-energy principle. The Gaia hypothesis can be seen much more as a framework or a method of investigation rather than a falsifiable empirical claim. So in this sense these particular frameworks they are and should be judged by their usefulness and not by their purported correctness. With usefulness, importantly usefulness is to be determined by factors such as parsimony coherence inconsistency plus its empirical and epistemic power to explain phenomena to produce new facts and so on in a given context and at a given time frame. So that's the stance that we think that should be taken on the Gaia hypothesis. Alright, so now having said a few words on the Gaia hypothesis, now I'm going to just prepare the stage by saying a few words on the free energy principle which given the topic of this conference needs an introduction but for these purposes I must mention just a few things. So the free energy principle we take it to be the case as well that it is a methodology to understand observed patterns of behavior observed patterns of behavior in the natural world. So then we apply the free energy principle as the tool and methodology to analyze these interdependent dynamics between living systems and planetary processes. Kind of like these goggles these methodological, technical scientific goggles that you put on to gain insights into the future states of Gaia or the Earth such as to make future formula predictions and devise potential interventions in this particular case with the care of alleviating future potential or present actually dysfunction. Okay, so the free energy principle very much aligns with this thinking by Eulen Schrodinger and the view that the second law of thermodynamics or under the second law of thermodynamics that open systems like us should tend to dissipation. However interestingly there are displays of this super cool thing the negentropy which are pockets of the universe where order and bodily integrity are maintained in the face of the search towards chaos or towards entropy or towards dissipation. And this view of the pockets of the universe is incredibly interesting because this is where we find in the self-organizing systems like us and what is so peculiar that distinguishes these self-organizing living systems from non-living systems is that they have the capacity of acting upon and exchanging matter energy information with the environment in order to seemingly defy the second law and therefore or thereby avoid dissipation and chaos and entropy. So that's extraordinary. So then active inference becomes this very useful tool to tell us or to explain and lucidate how these systems seemingly defy the second law how they resist entropy how the self-organizing systems combat entropy through the interaction maintaining their integrity via homeostasis. So then they maintain this energetic integrity they utilize energy to stay in a specific range of states that is vital for their survival and preserving their existence and they also engage with these feedback cycles which are the feedback actions like drinking when thirsty or moving away from heat signify working accomplished by bound energy contrasting with the free energy. They also defy the thermodynamics as I just mentioned by seemingly challenging the law to minimize entropy and safeguard their free energy. And finally this behavior that we observe in these self-organizing living systems in striving to minimize free energy is or agrees with a certain principle that we know which is the free energy principle. Okay, so then as I said earlier and as we know but just to keep in mind then active inference becomes a very useful tool to understand and make predictions about these coupling dynamics taking place between interacting systems. The living systems internal states in both is both operationally closed and thermodynamically open to the external states creating this distinct boundary which we usually mathematically capture with Markov blankets and then this interaction unfolds through the imagery of active and sensory states. Then the key point here to keep in mind is that what we want to have for adaptation in well-being is that there is a balanced interaction between sensory and active states and this is because it is within the interaction between sensory and active states that so to speak both important things happen in order for the system to be well-adjusted tuned, adapted to the environment. So living systems avoid this dissipation they want to remain alive they do everything in their power they navigate their environments in order to avoid this dissipation which is also or can be also said to minimize free energy avoid their dissipation by doing either one of the two things either by changing the model or changing the world by acting upon the world and this is extremely crucial for the point that I want to make later on. So now I want to apply the free energy principle to the guy hypothesis and see how that comes together in identifying and developing and why do I want to do that? I want to do that because I've also already mentioned it but I'm just going to refresh it because I want to identify and develop interventions to rectify any potential maladaptive responses namely actions which are antithetical to maintaining the integrity of the system. What we want is to have healthy well-adjusted maladaptive systems and human behavior particularly that leads to the imbalance within the market states is or could be considered sometimes maladaptive or pathological and this will be the last point I'm going to make in the presentation but for now I want to focus on how does the free energy principle deal with the guy hypothesis as a self-organizing system like we were just saying the guy admits the requirements to be formalized as part of an active infant system. It exists in a non-equilibrium steady state which means that its processors and states must persist at a time within a range of states that are far from equilibrium that means resisting entropy and the guy is also recognizable from its larger scale cosmic environments such as other planets and the rest of the universe it is also defined by the conditional independence that we were just talking about between the internal and external stage which influence each other only vicariously so then we can taking that into consideration we can even see that some of the people have already done this kind of like line of reasoning they've already run it through and they did that by employing advocates to analyze the earth's climate system dynamics which is quite interesting what they did was they looked at metabolic rates of the biosphere and they represented them as internal states and space, weather as external states and then they looked at active states as mirrored greenhouse effect shifts and sensory states as echoed ocean driven temperature changes and then internal and external states the idea here is that they indirectly influenced each other through oceans delayed response then earth's climate is interpreted as anti-anticipatory minimizing variational free energy so we take this work and we want to take it a step further agreeing with this work and building upon this work which shows the climate change we wanted to look at the more broad understanding of Gaia through the lenses of the free energy principle so instead of focusing on climate change which is very brilliantly done by these researchers we wanted to then scale it up and apply the free energy principle to understand the Gaia so then one can think of the internal states as the biosphere geographical processes and the external states as everything that there is beyond the system such as the outer atmosphere, silicea bodies and molten core then we can think of sensory states as the external environment impact such as earth rotation orbit, tides affected by the sun moon, planets plate tectonics, climates weather influenced by solar radiation, space weather etc and then we can think of the active states as earth's adaptability and adjusting properties from external influence such as gravitational force altars, orbits magnetic field interacts with solar winds or emission of detectable radiation influences distant objects that's a few examples so then I want to focus on the active states active states then through these lens active states then embody its primary mechanism for self-regulation representing its efforts to remain within the bounds required for ongoing existence based on predictions about the present and future states of the system for instance the current levels of greenhouse and albedo effects soil pH and the number of living organisms reflect the predictive attempts of the geosystem to adapt to anticipated effects of solar events volcanic eruptions ocean salinity changes and other environmental factors in the future so it is in this context that the role of life itself assumes a paramount importance in the self-regulation of the biosphere because the role of life itself in particular as we are going to focus on next the human life has a particular influence on all of these sort of already natural dynamics going on in the Gaia this allows us before we move into the human effects this allows us to make one final point before we move there which is to address the criticism that has been made in this paper by Vicente Jaja on the Markov blanket trick and what they have identified is that a purported weakness of the free energy principle what we say is that a weakness of the free energy principle that is identified in this paper actually speaks in favor of its application to the Gaia hypothesis and let me just try to explain that so Raha and colleagues they rightly point out that Markov linkage do not automatically capture every relevant property of biological systems relational properties such as affordances and by affordances they mean what the environment offers the animal what it provides or furnishes either good or evil fall into this bracket so what they're saying is that Markov linkage do not automatically capture every relevant aspect of biological system such as affordances and we want to point out that however affordances correspond to Gaia as one wrong in the leather of the nested Markov linkage within the system in question at every level ranging up from say cells organs, animals ecosystems, soul system the whole cosmos and how is it that we think that this particular framework that we bring here does have or address the notion of affordances our solar system is one that affords a living planet so the affordance available is an environment in an environment are not determined solely by the physical properties of the environment and now we think that this is the point that we would like to address in Haja's paper is that the affordances are not determined solely by the physical properties of the environment but also by the organism goals, abilities and previous experiences and in this case of the sun so what is meant here is that the sun is what has the affordances for or what affords anything else to happen on the planet at whatever each scale right the physical properties of the sun as well as the goals and abilities of the organisms on earth shape the affordances that the sun provides so it's a quite interesting way of thinking that any affordances whatsoever that we might think of that we have as living systems they are afforded by the sun so that's the idea okay so now having gone through how the furniture principle somewhat speaks to and contributes to understanding the Gaia hypothesis then we want to focus on how human behavior has a profound effect on the dynamics that already existing in the natural world so according to the furniture principle living beings have a natural tendency to act upon their environment to prevent the dissipation of their integrity so that's what we do in order to remain alive there are therefore two ways for a thing like us to minimize its free energy and thereby maintain our own integrity upon evidence as we navigate the world and gather and observe evidence we have two options one is to change the model the other one is to change the world by acting upon it like I just said before so then why is this relevant well this is relevant because there is the possibility that in ways in which living systems and in this particular case human beings interact and adapt to the world there is the possibility that we see maladaptive behavior and how would that be well if it is the case that there is a rejection of the information available so as to remove the imperative for action or if it is the case that possessing the information available but we still remain or refrain from taking action then there seems to be if the free energy principle is correct if the free energy principle is to be correct that systems living beings tend to act upon the world in order to remain alive and if it is the case that one rejects that kind of information and does not take action to remain alive such as the information of the climate crisis then we might be looking at maladaptive behavior so if the free energy principle holds then the observed contradictory behavior is a maladaptive state or should be seen as a maladaptive state that arises from an imbalance within the Markov blanket and how do we see this imbalance within the Markov blanket we see it to put it in very very simple terms we see it as an insulation of internal states that are not being affected by the information available outside of the system so the influences or the interactions are occurring between internal, active and sensory states and this is an imbalance that results in an insulation or an echo chamber effect in internal states that are impermeable to the information from or potentially existing in the external states that demands action so then some foot for thought can we make of this the free energy principle if we are to use the free energy principle to model the dynamics of the relationships between humans and the environment then we think that it is possible to see the current lack of pro-environmental behavior as a pathology of the human species that must receive treatment we label this condition as biophilia deficiency syndrome biophilia deficiency syndrome and we use it is important for us to focus on that to construct some validity is useful to consider in this case other applications of the free energy principle that may be useful as an analogous case of pathology mainly conducted in psychiatry and neurology and here in very very very short because we don't have time it has a failure to assign the right way or prediction to prediction errors so then the free energy principle from this point of view this is a proscenious pathology because doing nothing is a base optimal response in the face of pathological attention to various sources of evidence so evidence is ignored so concluding and recapping according to the free energy principle living systems possess a biological encoded inclination to interact with the environment in order to survive and adapt to the environment and despite the climate change being a pressing issue and the information being available the evidence being available productive measures to reverse its effects are not giving the priority they deserve these state of affairs seems to very clearly a divergence from a natural inclination that we call biophilia this discrepancy between the expected and the observed actions can be understood in the framework of the free energy principle or so we think as a newness it implies that systems including human society may form erroneous inferences that prevent them from effectively addressing the problem at hand now thinking about future directions and this is my last slide current ecological imbalances could be thought of as a stuck state that require an active disturbance to bring about some change a stuck state is a state the system is at a local minimum where the system is at and it won't get out of that state unless it is disturbed so that's the principle in complex systems theory and we think that that's how this particular state should be looked at but that's for future research and what we would be calling is we would be calling for amounts to the kind of homeostatic awakening so the sort of like getting out of this stuck state which requires some kind of like homeostatic awakening a deliberately induced disruptive shift in the trajectory reflecting a prioritization of the planetary homeostasis over infinite growth or staying in the same place and yeah this is my last slide, thank you so much I would like also to acknowledge my co-author Casper Montgomery thank you so much for your attention yes, thank you well a lot of ways to go, I'll just ask one question just in the years that you've been in the game how do you see the active inference ecosystem and the ways also that it's being applied to ecosystems evolving for what time scales or in what ways does that really present itself to you sadly slow I think there's a lot of work that needs to be done we try to do this particular one here to apply active inference to the planet or the earth and the living system precisely to bring out this sort of overcome so one is to look at this problem of the climate crisis as what we think it is which is more of a pathology if the free energy principle holds then it seems that there is maladaptive behavior of the collective of the human species so I think that there's a lot of work that can be done and that would very much be useful in the climate crisis in general there is some other work that came out recently by Maxwell and colleagues as well on the ecosystems which is really really cool and I would also mention that one and I really hope that we can we can pursue this line of research and employee active inference in the free energy principle to this particular to this particular case what really one of the things that really attracts me to the free energy principle is the capacity that it has to bring to the forefront certain laws of physics but also the living systems navigating and interacting with their environments which I find much more appealing than any computational framework per se so that's what I find that it's a very useful conceptual kit that I think that I would really like to see more applied in the future for the climate crisis the earth great well thank you so much for building on that line of work and for sharing with us tonight so thank you so much for having me till next time see you thank you bye alright great talk by an s and now welcome aswin how are you doing hi daniel i'm good hanging out looking for yeah yeah yeah sounds good and yes looking forward to your workshop on sophisticated inference in pyMDP so it can be up to 90 minutes or it's totally okay if it's less and we can take a short break but please take it away and just let me know how ever I can help great thank you so much maybe I'll share my screen and start so regarding the structure of what we are doing here I'm assuming that people who see this later might want to try it hands on along with the tutorial or something so that's the structure I get in mind and so I'll go slow please bear with me so welcome all of you to this session on sophisticated inference in pyMDP so here we are going to attempt to model some sophisticated inference simulations especially the one in the original paper using the pyMDP module and how it is not part of pyMDP right now but we are on the process of adding sophisticated inference to the pyMDP module and I'm going to mainly talk about the code that I have developed to kind of add that functionality so I'm Ashwin Paul I'm finally a PhD candidate at Monash University and mostly I work with active inference models and try to understand how to use them as an explainable model to basically understand emergence of intelligent behavior right so let's dive into the material right away so to give an intro of the energy principle I'm sure all of you right now have an understanding of what it is but the central idea is that an agent is always trying to minimize the entropy of its observations right so if an observation is having really low probability in your mind and that happens then you are probably surprised and vice versa right so and entropy here is defined as the information theoretic entropy where if you have a low probability then automatically this is a high surprise or high entropy observation and as we all know active inference also gives us a methodology to define what we call an agent environment loop and this lets us define what is the agent that we are looking at and what is the behavior of the agent given the environment around it and so on right so you are also familiar with the idea of Markov and this is important because we always have to remember I mean have to remember the difference between the generative process and the generative model which is quite a famous point of confusion in active inference literature and for the people who try to understand it in the beginning right so the central question is that how does an agent recognize entropy because how does an agent know which observation is low or high probabilistic right so that is by maintaining a generative model and the generative model will tell you which is a high probabilistic observation and which is a low probabilistic observation so the idea is that all the agent has access to is an observation that is coming from a generative process which the agent cannot directly observe and an intelligent agent will try to build up a generative model in its mind which is a model of the hidden states and the observation it has access to and it can hope to kind of compute probabilities using this generative model right but there is a problem that in general it is an intractable problem to kind of marginalize the probability of observation from the generative model and that is why we have to define an upper bound on the surprise that the agent is trying to minimize and there comes the idea of free energy right so this upper bound the agent is supposedly minimizing is the free energy and that is why it is the free energy principle and here we have a new term called capital QS which can be interpreted as the belief that the agent is maintaining about the hidden states in its generative model and this quantity is the free energy and traditionally we see this variation of free energy being interpreted in mostly the machine learning way where it is a balance between complexity and accuracy of the model so when minimizing the free energy the agent is trying to come up with a single model but at the same time an accurate model because here it's a minus sign with accuracy right it can also be interpreted in the physics way where the agent is always trying to minimize the energy of the model but at the same time maximizing the entropy of the model which goes in conjunction with the maximum entropy principle and so on from the classic literature so given that this idea of a generative model is so important in a software point of view that's the first thing that you might want to do right to define a generative model for the agent which is informed or not informed depending upon the experiment that you are trying to model so in classical active inference usually decision making is defined in terms of policies so for example if you are an agent in this environment so in the Mario game Mario is the agent and everything else is the environment and Mario has three available actions run, jump or stay in this environment and the classic definition of policy is that it is a sequence of actions in time so if you have a time horizon of capital T then a policy is nothing but a series of actions you might take so run, run, run, jump and so on in time so this is the superscript is the action so here it is jump, run and so on and the subscript is the time and then what you can have is a policy space which is a collection of many such policies smaller pie and what you essentially do to take decisions in active inference is compute not optimize, compute the expected free energy of every policy in your policy space and basically that can be interpreted as a balance between risk and ambiguity and yeah so when you compute this expected free energy what you are trying to do is minimizing the risk that is how different is your belief about the observation from your prior preferences capital C so this is also part of the generative model when you are trying to model control and at the same time you are trying to minimize the ambiguity when you are choosing a policy that has the minimum expected free energy right but this formulation has a problem that a policy space quickly becomes intractable that there can be an enormous number of small pies or policies in your policy space and sitting and computing the expected free energy for all such policies even for a small time horizon is not possible but this is the classical structure that has been implemented in PyMDP nonetheless where we have different modules in PyMDP that is meant to be implementing different aspects of behavior so for example for inference or perception we have belief propagation fixed point iteration marginal message passing and all that implemented in the inference module in the control module we have different methods to evaluate expected free energy for policies one depending upon the expected utility the other one depending upon the classic method that I just explained then we have a module for learning so we learn the parameters in the form dp like capital A, capital B so likelihood transition dynamics and so on and then we have algorithms for implementing all this in the algorithms module and then the most powerful thing in PyMDP right now is the agent class where it is easy for you to kind of define the agent environment loop and we are trying to build up so today I am going to talk about an agent class that implements substicated inference rather than the classical active inference that we just saw so as I mentioned how many valid policies can be defined say for a time horizon of 15 in classical active inference right so the first policy of course is a series of first action that is jump then you can have the last action changed and you already can see that there can be n number of combinations and for this simple case the policy space is as big as 10 to the power 13 and in a stochastic problem setting there is no way to kind of come up with a small subset of this policy space so that you can tackle this problem of computational complexity and yeah as I mentioned in a stochastic problem setting it is an intractable size policy space there comes the idea of sophisticated inference where we are thinking about taking this instance in a different way right so rather than thinking about sequence of actions in time we can directly think of what to do when we see something depending upon our beliefs about the current state and beliefs about the future right so if I see myself in a current situation what should I do and that's more like a straightforward thinking of how to take actions and here the expected free energy the structure of the expected free energy is the same but we are not evaluating expected free energy of policies but expected free energy of observation action combinations so if I see something and if I do this what is the expected free energy and that's what I am trying to minimize that's what I am trying to optimize in this setting right so here again we have the risk term where we are trying to minimize the deviation between the belief and the prior instances we also have the ambiguity term and this together makes up the expected free energy of this time point at time t but we also have an expectation about what is the expected free energy in the next time step and to evaluate the expected free energy of next time step you will have to again compute this equation with o t plus 2 and for that you will have to again compute this equation with o t plus 3 and so on and this automatically becomes a research because of the recursive way this equation is defined and it comes with its own problems but there are clever clever ways to get around them and that's we are going to discuss that in the code today so given the structure of sophisticated inference here as I mentioned the research replace the policy space that we saw for the traditional active inference we are used to so in this workshop what I am focusing on is how to kind of define a generative model and and given an environment so for example this is the grid that is stimulated simulated in the original paper and we are going to talk about how to build up a generative model for this grid that can be used in the sophisticated inference in the PMDP module and so basically what I am trying to talk about is that the environment will have a step function that takes an action from the agent in PMDP and the agent will get an observation out of that action and we are we will talk about this particular function agent dot step and agent dot steps will step will take up an observation and try to come up with an action for the next time step right and this creates a loop and cleverly designing this loop that will let you see emergence of purposeful behavior in the sophisticated inference setting so for example in this particular grid with sufficient planning horizon you will be able to see that the agent is capable of navigating in this grid and so on so this is the example that I am going to focus on in this talk today so excuse me I want to right away jump in to the code so we have the PMDP which I am hoping you are familiar with so we have this github repository where we have the PMDP module and inside PMDP module we have several parts for it so here we have the agent in the original PMDP module that implements the so called classical active inference we have several environments and we have helper functions like learning, inference, maths and so on so this is the module of PMDP but in the parent folder we also have examples where there is tutorials about how to use the agent class how to kind of deal with the environments and so on so if you look at the full requests so we are right now trying to merge sophisticated inference into the original PMDP module and today I am going to talk about the code in this full request so if you are if you want to try this hands on you might want to go to this page where this full request is there and it has the same structure of PMDP it is basically designed using PMDP and here what we have additionally is an agent si which is a sophisticated inference agent which does everything and also planning and decision making in the sophisticated inference way and in the parent folder there is also an example folder for sophisticated inference demo and what I am going to do today is to walk you through this tutorial of sophisticated inference and on the way I am going to discuss and at points where I reference the helper codes I am going to go to that code and explain what is actually happening and how we complete that agent environment loop where we can see that purposeful behavior so that is the PMDP home then I also talked about the full request so let us right away go into the Jupyter notebook so this is my local copy of this repository so it is easier for me to run it and show it in my personal computer so this is the parent folder with PMDP and examples and inside examples I have a demo folder for sophisticated inference and this is the notebook I am talking about right so here in this example what we are trying to do is deal with this particular grid world task from the original sophisticated inference paper and make this agent or enable this agent to navigate to this red dot which is supposedly the goal state in this particular task given up prior preference like this right so this prior preference is quite informative in the sense that we right away can see that this is the most preferred state the white color and the surrounding states are kind of less preferred but more preferred than the far away ones right so this is the grid world task that we are trying to use so the first cell is importing all the necessary libraries and some useful libraries like numpy and map plotlib and the most important one is PMDP so I am actually now calling the local copy of my PMDP with the sophisticated inference implementation not the original one which is not merged yet and the first thing I want to talk about is the environment itself right so the environment dot step part where if I get some action how does the environment work and for that inside this folder I have an I have a file which is grid environment si.py and this is basically an environment class so do not worry about how this environment is actually implemented the only thing to worry about is this function that we are going to use which is environment dot step so this function will take an action into it and depending upon the current state of the environment it will calculate what is the most probable next state given this action from the agent so that's the idea and then it will also calculate a reward of some negligible negative value if it is not the goal state and if it is the goal state it will give a reward of 10 and that's how the environment is designed and it will update the current state to the new state and basically what it will return is a new state depending upon your action the reward for that action and whether it is an end of the episode and so on so this implementation is the standard open ai environment implementation and this is the environment dot step function so in this grid for example if I am right now in this state and if I take an action up so I have 4 available actions north, south, east and west if I go north then the environment dot step will make sure that I am in the state that is above the state and if I go east or west then I'll stay here or south I'll stay here so that's the idea and yeah and there is an episode length limit that is 8 here that means that I am restricting the length of every episode to be 8 which is the ideal length of reaching this goal state just to avoid confusion so in this environment after 8 actions the environment will terminate and if you have to kind of reach this goal state in the optimal time point so that's the idea how the environment is implemented ok I hope that is clear and there are many helpful functions in this environment like rendering the environment rendering a prior preference matrix in this environment if you design a prior preference this environment can show that in a pictorial way how your prior preferences that you will see in the notebook below so now it's time that we define a generative model for the sophisticated inference agent before that let's define the structure of the generative model that we want the agent to have in its mind which is tailor made for this particular environment so here in this particular grid world task we have 25 valid states starting from this state all this black states in this path are valid states so there are 25 valid states then there are 4 available actions for the agent north, south, east and west so this is part of our generative model this is also in alignment with the reality of the grid this is about what the agent has in its mind right and then the observation is the state space the problem is fully observable so there is no ambiguity there then we define basically the number of states then we define basically the number of states which is a list of your state space number of factors which is now one because you only have one heat and state factor here then number of controls which is going to be 4 that's 4 available actions and your observations space like that so this is the structure of your generative model and let's look at the structure of the parameters now inside a POMDP so the first one is the likelihood function which is often denoted by capital A and here it is a function of application modalities that I have and how many state modalities I have if I run this cell I have to run the parent cells to make sure everything works so I have rendered the environment the structure of the generative model and here I have the capital A matrix which has a structure 2525 that means that I have 25 states and 25 observations and here because it's fully observable I am initializing it as an identity matrix of size 25 so that's my likelihood matrix that I am initializing for this particular great dual task then the second element is the transition matrix so please note that I am using all the existing POMDP functionalities to define a random A matrix and then using an identity matrix on top of that so yeah I am not doing anything new here it is the existing POMDP functionality then what I can do now is define the B matrix which is also called transition matrix so the transition matrix encodes transitions like where I am going to end up in the future if I start from a particular state and take an action so that's the idea where it depends upon the number of states and number of controls so it has this structure of state action state where if I take an action from a particular state where I am going to end up so that is also a future state right so I am going to initialize it as the true environment state so now this is part of the environment that I have built it will give out the B matrix it might be worth it to look at the structure of this B matrix so here we have 25 25 4 so that means that if I take an action from a particular state where I am going to end up and we have the true transition dynamics for this particular grid by design so there is a function called get true B and that will give us the true B of the system which can which the agent can use okay so ideally we would want the agent to learn this but for the purpose of this demo we are assuming that the agent already knows the structure of the grid and then comes the prior preference which is interesting here in the sense that it is defined as how closer you are to the gold state so if you are at the gold state then clearly that is the most sought out state you prefer that the most and how do you prefer the neighboring states right so that is dependent upon the square root of the distance or basically the distance from that particular gold state so you define a grid which is 8 x 8 the same size as this particular grid well task and then we have a method to kind of add values which is the preference you have for every state and if you render the particular C matrix you can see the structure which is the same where this gold state is more preferred and the surrounding states less preferred and so on so now we have the C matrix also defined in the classic prime DP way and then I initialize that C matrix as the C matrix we evaluated in the previous cell which is small C this particular C matrix and then lastly for the generative model we have capital D which is your prior overhead in states and for that I am using a uniform object array so that means that I don't have a prior of where I am starting so let me run the bending cells so here the D matrix is a uniform distribution over hidden states I don't know where I am going to start the simulations and so on so this is the basic structure of the generative model and then we have the agent class which I want to discuss separately in the environment so given these environment parameters how would you expect the agent class to work so where is the agent class inside this folder structure inside the prime DP module folder we have an agent si.py which is basically a class again and similar to the environment class here also we have a step function where this will take an observation through the function and also a flag whether or not to learn the environment which is optional so if you disable it it won't learn the generative model if you enable it it will update the parameters of the generative model and what basically it does is it will return an action which is to be taken at this point of time and the environment can basically do that action right so in this file we have the agent class which I will explain in detail so I am basically importing that agent class in this cell and then we are going to try and reproduce this behavioral result from the originals of state inference paper ok and for that so what we expect is that given this prior preference structure there are local maximas in this prior preference structure so if you start from this particular point if you do not plan deep enough what you will end up is in one of these local maximas where you don't see that there is a highly preferred observation say 4 steps down the line so if you are in this particular state what you will see is this local maxima and you will go and sit there because the neighboring states are less preferred and this state which is more preferred is not accessible because of the wall or the wall structure so you have to take a turn and pass through less preferred states and you made deep planning in order to enable the agent to do that the agent should be able to kind of simulate 4 time steps ahead in time to see that there is this highly rewarding observation coming to kind of do that actions so that's the point that we are trying to see in this particular demo so for low planning depth it will basically get stuck in one of the local maximas but with sufficient planning depth it will navigate to the gold state so that's what we are trying to see right so we have different planning horizons and what we are basically doing is give the agent a generative model which we right now defined the A matrix B matrix C matrix V matrix then we have the planning horizon of capital N so here I am iterating over planning depth so N will be 1, 3 and 4 for the loop then we have action precision which is often denoted by alpha in active impress literature so that determines which action is to be taken so a highly precise action precision means that it will stick to the action with the lowest expected free energy but a low action precision is kind of probabilistic where it will also consider other actions then we have a planning precision which is part of the planning function we will discuss which is often denoted in the literature as gamma then we also have a search threshold which is extremely important for substigated inference because as we saw substigated inference is a tree search and tree search is bad in the sense that it can have a lot of computations but you have to define a threshold to kind of ignore many possibilities to make it work and that's the idea that we will also discuss ok so just a preview before we go to the agent class what we are trying to do is in a loop we are going to call the agent dot step and environment dot step in series so the agent will see an observation it will take an action that action will go into the environment the environment will give it back new observations and this loop continues and we want to see over time how this loop evolves into a purposeful behavior and if the agent at all is capable for that so before I reveal the results let's discuss the agent class so in order to give an action when an observation is given the agent should have the planning and so on right so here is the agent the associated inference agent where we are actually using the existing PyMDP agent for some functionalities so in PyMDP we already have really well written functions for perception and learning so the only thing we want to kind of replace is how the agent is doing planning and how the agent is taking decisions over policies right so here here we are using that parent agent class so from PyMDP dot agent we are importing that agent class which is sitting next to the SI agent that we are discussing now and basically we are taking in the generative model structure from the main program for this class to work which is A, B, C and D and all the precision and threshold parameter I mentioned then it is kind of normalizing the prior preference that we mentioned in the main program so here if I look at how C is the structure of C is defined in terms of numbers and the prior preference is often interpreted or it should be a probabilistic distribution for the computations to work right so we are going to normalize it as a probabilistic distribution rather than having numbers that do not add up to 1 so that is what is happening here we are using softmax to do that then what we are doing is we are initializing the existing PyMDP agent with these generative model parameters and what we are intending to do is write a planning function for a given planning horizon and a given threshold for research okay so there are three functions in this agent class one is a helper function for planning which we will discuss now then there is a planning function itself which is going to do planning using research and then because it is a recursive research we are going to need an additional function that implements that recursive evaluation where we are going to call this function called forward search inside the function itself so we are calling this function inside this function so that is to calculate expected free energy for the next step and it will call it again for the next step till our planning horizon so that is the idea of recursive looping and finally it will return the expected free energy for all actions given an observations and then we just implement the step function where it is written sequentially what to do given an observation okay so going back to the demo here we have this first idea where you get an observation and it gives out an action so let us go to the function dot step function and imagine what happens so if it is time t equal to 0 or in the beginning of the experiment what it is ideally supposed to do in the first place is infer that state using its observation so what we are giving it is an observation and using the modules for inference it is going to come up with a belief qs which is a belief about states okay so self dot qs is the belief inside the agent and once it has a belief about where it is right now it can implement plan dot research which is do planning for this particular belief of hidden state right now and once it has done planning it can take decision using the sample action function in time dp and basically return it return that action and for every other time steps the sequence remains the same but it is also learning about the structure if you enable learning in your agent class so that is the step function but in order to do planning what it does is it kind of reorganizes the generative model structure for any number of hidden state modalities and any number of observation modalities so to discuss how melting works I would like to talk about the new A matrices and B matrices it evaluates for implementing that planning and let's understand that so let's go back to the original hidden state factors so here we only have one hidden state factor so that's why it is b0 right and b1 does not exist because we only have one hidden state factor but imagine that if I have two hidden state factors with the same size maybe so here it could be a location and maybe something else inside the agent's mind and we should also have controls for these two hidden state factors just like so there should be control for every hidden state factor if you are familiar with active inference idea and then maybe the observation space is also directly observing these two hidden state factors okay so this is a new generative model structure with multiple hidden states and multiple observation modalities and right away you can see that the dimensionalities of your parameters change you have 25 observations coming from 25 times 25 hidden states and if you look at the structure of the first observation modality it's the same but what we want is a new matrix where it's going to be 25 with not two hidden states but just one hidden state it's only a reorganization of the generative model but computations essentially remains the same so that's what this helper function is trying to do which will make things easier for us when we have multiple hidden state modalities so if you have multiple hidden state modalities we are going to compute how many total states you have which is the multiplication of number of hidden states in each modality okay so if you have 25 hidden states in one modality and 25 hidden states in the other modality you are going to have 625 total number of states and if you have 4 actions each in each modalities then you have total of 16 actions which is nothing but the combination of these 4 actions each in the modalities so you are going to have 4 times 4 16 actions if you have 2 modalities and it's basically going to build a generative model that has the same model parameters but just with a different dimension structure so that it's easier for us to calculate the expected create so now we have a new A and new B and a new belief which is nothing but tensor products of existing parameters and beliefs it's nothing but a new big matrix and nothing else okay it's not a it's not a change it's just a transformation of structure and given this A, B and Q we are going to predict what's going to happen in the future and evaluate the expected free energies for them so in order to do planning so that's the second function what we are going to do is first call the first function which will do the melting for us and set up the generative model in good dimensions we have the expected free energy itself for all the actions and then we have the probability that depends upon this expected free energy for these actions so why is it just the actions and not the observations because here we are going to evaluate expected free energy of actions for the given observations right so let me go back to the slides and discuss this pictorially to make things easier so here we have the grid and we have the prior footprints and what we are trying to implement is that if you observe some observation at time then you are going to consider the consequences of your actions given that observation because you can predict what's going to happen because in your generative model you have the transition dynamics that will tell you given this state if I take this action where I am going to end up so that's basically predicting what's going to happen in the future and you are right now considering the consequence of available actions in your arsenal and then if you take an action then you can predict what's going to happen in the next time step as a new observation right so you have a probability distribution that tells you that say this observation is the most likely to happen and the other observations are not really likely then what you will do is you do this again you consider the consequence of doing your actions from that particular observation and this goes on to your planning depth right so this can be thought of as maybe you want to go to the gym then you are going to consider all the consequences what will happen if I wear my shoes if I don't wear my shoes if I go in my car then you will realize that ok I have to wear my shoes then you consider the consequence that I am now ready to go to the gym and me going to the gym will end up me being in the gym so that's the idea like you consider the consequence of where you are right now and you can go as much as you want right you can predict so in a game of chess you might be in a particular state you consider your consequences you see the future you consider consequences from that future and you can go as deep as you want depending upon your computational abilities right so that's what you are trying to implement in this agent class where we are considering consequences yeah so for every modality we are going to consider the expected free energy for actions and this will basically call the next function which is forward search so forward search is implementing the thing that I just mentioned considering consequences and in forward search what you are basically doing is for every action so in line 149 I have a loop that goes over every action then I am going to consider the posterior or the consequences of all those actions I use my transition probabilities to evaluate that consequences then I am going to predict the observations because my prior preferences are defined in terms of observations I am going to predict my observations and then evaluate the expected free energy which is the sum of risk and ambiguity okay I hope that makes sense like here you have considered the consequence which is consequence of future which is post or posterior and you are basically evaluating how good that posterior is depending upon your expected free energy and that becomes the expected free energy for that particular action and you do this for all the action okay and why this is powerful it is because you can go as deep as you want so here in the next step you go to this loop where you will check if I am crossing my deep planning or the depth of planning and then you are doing basically the same given that posterior what is the consequence of the actions of that particular posterior so here for considering that we are again calling the parent function so the same function forward search to consider consequences of those combinations and it will basically come back and add up your expected free energy so what happens over this sequence is that consider some or all future consequences and then all that values will trickle up your tree and that sum of the expected free energy will tell you which action is good or which action is bad which you can take into kind of see your preferred observations so that is the idea of implementing research and I will also talk about the importance of this threshold here which is which makes this algorithm possible so without this threshold this algorithm will not work I will explicitly talk about why that is the case and then once you evaluate the expected free energy for all available actions given the present state you can basically compute what you call the action distribution that is how probable is my action how I should take my action so that is and we also have this action precision parameter alpha so if alpha is very high then it basically is a highly skewed distribution where you will always choose the action that minimizes expected free energy if alpha is really low then it is going to be a more sparse or spread out distribution and then you can use this action distribution to sample actions and so in the agent environment loop we just finished doing planning and computing that action distribution then using that action distribution you can sample an action from your policy space so let us look at the policy space in this generative model so I am switching back to the original generative model with one hidden state factor and let us do planning and maybe initialize this agent I just want to initialize this agent to see the policy space not run the loop so I initialize this agent say for planning depth of one and if I look at agent dot policies I can say that I have basically four available actions which is north, south, east and west and if I have an action distribution it will tell me how probable is to take that action so if I look at agent p i so this is not defined because I have not done planning but I can do planning and then it will be defined so I implemented planning with research and then now I have an action distribution so for this particular scenario I will take my third action the most which is 0.99 basically that is the probability which is north, south and east in this particular case I just wanted to familiarize you with the matrices but we are now going to see the agent environment loop in action and so now you can sample an action from the sample action function and then implement learning which is using the standard by MDP way where I will update my transition dynamics and likelihood dynamics depending upon what I see and what is my belief and so on so my emphasis is on the decision making part so once you sample an action then that action basically goes back to the environment so now let us implement this for a planning depth of one and see how the agent behaves so here if it is a planning depth of one then that means that the agent is only considering the consequence of one time step ahead just seeing the immediate future for doing planning so I am giving the planning depth of one I am resetting the environment where the agent is going to start from that initial start state and in the loop it is going to get that observation take that action give back an action and we will look at the action probabilities and also we will give that action back to the environment get back the observation and this loop continues till the episode is terminated and I have set the episode length to be 8 just to see the outcome of 8 action so when we run this loop these matrices are nothing but the action distributions so how likely each action is to be taken and this is where the agent end up in the last time step so let us maybe also enable this environment or render that will show us where the agent is at every time step so initially the agent was at this location and we have an action distribution of this here so north south east and west so the agent knows that it should go north why? because if I look at the prior preference this state is more preferred than this state so the agent successfully calculated the expected free energy and inferred that I should go to this state and not stay in this state and because it has the generative model of transitions available it can infer that I should take an action north to go to this state so that is good and the agent goes to north and at this particular state the agent inferred that it should go to north south east so it will take an action east and it will go here and at this point of time I want your attention where the action distribution is equally probable for north and east why is that? because the agent is only looking at the immediate future so let us go back to the prior preference where the agent is right now here or is it here? yeah it is right now here in this particular state and if the agent is considering immediate consequences of just one action then these two states are equally good for it to be in the next state so there is no distinction between these two states if it is only looking at the immediate future so that means that the expected free energies will conclude that I should go to north or east it will not matter if I am looking at just one time step at a time that is the idea and out of probability it is going here it took the action east and from this state when it is doing inference it is inferring that this state is better and basically it is ending up in this local maxima state which is this particular state where the neighboring states are less prepared and this is wall and you cannot go there because it is forbidden for the agent by structure so it is basically going to sit there forever where it only sees that local maxima of prior preference and let us look at what might happen if you have higher planning depth so if I go to the planning depth of three then that means that the agent is actually reaching the gold state at the last time step but still in the third time point it had two probabilistic actions north and east so here from this particular state out of probability it took the action north but it can all equally take the action east and end up in this local maxima so let us run again and probably it will end up in this local maxima okay and only for the planning depth of four which is sufficient enough which is necessary for this particular grid the agent is fully sure of what to do so at every time point it is fully sure of what to do that it has to go north then east north north north east east and south and reach this particular gold state so only for time step or planning depth n equal to four it can successfully navigate this grid with 100 percent certainty so that is the idea that is the implementation that I hope you got so there is also this idea of action precision so here it is a high action precision that is why it is taking the actions that is from the probability it is a low action precision like one then good actions are more probable but that does not mean that it will be taken here by luck it is taking the right actions and reaching the state but here probabilities are most fast but you will also see like exploration behavior in more number of trials if you control this action precision take it to a high value to make sure that the agent reaches the goal for this particular problem but it is worth playing and it is important so for different planning depths like one, three and four in this problem this is the behavior that you expect for lower planning depths which is not sufficient the agent ends up in local maximas or local minimas of expected free energy or local maximas of prior preference but with sufficient planning depth it is able to navigate and reach the goal so that brings us to the last point in this tutorial where why is it important to have a threshold in evaluating sophisticated inference so by threshold what we mean is that we can ignore future possibilities in two levels you can ignore not likely actions or not likely observations in this research but if you consider the consequences of all actions and observations that means that you will have to consider four consequences in the first place then you will have to consider four times the action states in the next time step then all of that multiplied with the number of actions and this research becomes intractable until explored and it is even worse than the classical active inference policy space problem but by defining a threshold of even a small value will ignore possibilities so where is that implemented in the forward search algorithm we are considering actions with only action probabilities greater than the particular threshold here I am defining it as 1 by 16 also in the parent paper it is 1 by 16 the action probability so if it is 0 then that means that it will consider all the consequences of future and that is intractable so you can ignore actions which is not probable also ignore states which is less likely or only consider states that has probability greater than this particular threshold value and that significantly reduces the computational complexity where you will only consider combinations that are probable in the future tree and that lets you go deeper in your planning horizon that is an important point here and if you compare the time that takes for deeper planning for a search threshold of 0 so a search threshold of 0 means that you will consider all consequences and the more deep you plan the more time it takes and if you see to consider only the first future or the immediate future it takes 0.01 seconds for considering 3 possibilities into the future it takes 3 seconds and for 4 it takes 300 seconds and you can see that the computational time is exponentially growing but if you have a very small search threshold you have that computational time that makes sense for implementation in real world right so here for n equal to 4 that is 4 time steps into the future it is only taking 0.1 seconds and that is ok I can still do simulations with this complexity but there is no way I can talk about how this is how less the computational complexities it truly depends on the nature of your prior preferences and environment in action but this search threshold actually works in real life and we just saw that in our simulations right for n equal to 4 it took only like 0.3 seconds in our simulations but if you said that search threshold as 0 it already is 300 seconds for doing pool depth planning and if I set planning depth of 5 then it will basically run forever maybe and I will not be able to do simulations so that's the idea of search threshold so actually that's it I wanted to explain the agent class the environment class and the particular demo and yeah maybe it's a good time for questions if anybody was listening and I hope people get to play with this code and look at the tutorial and implement this build generative models like this so this particular example is how do you build a generative model for this gradual task and see how the agent is able to take meaningful actions but here we gave it the true structure of the environment in the B matrix and A matrix and so on but you can also play around with learning in the sense that while defining the agent dot step you can add a flag that says learning equal to true and if you start with an uninformed A, B and so on you can experiment on how the agent is learning that environment here you can look at the B matrix in the beginning you can look at the B matrix after say 10 trials and see how the learning is taking place here it doesn't matter because the agent knows the structure and it won't learn much but if it starts from unknown structure then there is scope of learning also to be implemented and it's already implemented because we are using existing PyMDP functionalities for learning A and B and it's already part of the step function so I hope step function is clear which is the only thing you need to know if you are trying to implement sophisticated inference and just the names of these matrices if you want to probe them and look at them and yeah I hope the session was useful so I thank my collaborators and Connor who maintains PyMDP and also Brandon who runs the PyMDP fellowship which I was part of and that's where I worked on implementing sophisticated inference in PyMDP and it will be part of the original PyMDP module soon and I hope people can start using this module to simulate sophisticated inference experiments and this basically becomes useful so maybe it's a good time to discuss questions or clarifications on the code or maybe it's a good time to take a break as Daniel was mentioning great thank you that was awesome well I have a few different questions and I'll read a few from the live chat I'll first just go to the live chat then ask those and then ask some other questions so Dave asks how do you think about the neural implementation of recursion brains don't seem to implement computer hardware style recursion deeper than a stack depth of one aside from heavily overlearned tasks we can confine ourselves to asking about recursion for the purposes of exploring temperately deep state spaces searching forward in time so how do we reconcile this really beautiful and elegant and computationally efficient full depth tree search with the biological basis of multi-scale planning so I'm not an expert in neural computation but the answer to that would be basically you're doing only one computation at a time right and you have all you need is some memory to store your beliefs and use those beliefs to kind of do the same computation again so we are not talking about neural I mean this hardcore recursive implementation we are only doing local computations at a time and just because of the structure of the generative model and we and because we have memory this can be done and I don't see why brain can't do it even though individual neurons might not be able to do it the brain has memory the brain has the ability to store memory and the ability to dream the ability to simulate it knows the consequences of actions and you do this on a daily basis where you plan your future and decide what to do right so on a single neuron level I'm not really sure how to answer that question but I don't really see why the brain can't do it as an organism cool okay to the code I guess I have a few questions let's can we go back to the maze and of course if anyone else has questions in the live chat just go for it so in the maze how does the the moves that are possible how is that reflected what what stage is it updated that what for example that it initially knows that it can only go up and then it can only go right or down where's that reflected with the updating of what what the kind of that relational aspect of affordances what is even possible to do and then how does it evaluate in a deep research when it does it need to know what things could or couldn't happen in the future so if I understand the questions correctly there is an important distinction between the generative process and the generative model right so in the great world which is the generative process we have implemented all of that manually where we have this transition that already knows what will happen if you take an action from a state so it's like an environment that knows how to work so it's like a reality where there are consequences of actions and it's already there but this information is available to the agent to be part of its generative model and in the generative model what it basically does is use that transition dynamics given or learned to simulate what might happen in the future ok so once I have that transition dynamics if I go to the where is the agent if I go to the research what it is essentially doing is evaluating the posterior given a particular state and an action J so from my generative model I know that if I start from this state and if I take this action I go to this posterior and I consider all the consequences and if in the generative model it is unlikely that if I go east I don't go there and so on it's basically it's automatically reflected in the expected free energy so I hope that answers the question so here if I set the action precision to be high and also enable the environment so initially I am in this particular state and with respect to the expected free energy what I am doing is I am using my generative model to predict what will happen if I take four actions and my predictions will say that if I take north I will go here and if I take all the other actions I will stay here and because in my prior preference north is more preferred I can infer that north is the action I should go for so that's the idea so here the grid structure is given to the agent and it might be a little confusing but the agent can also learn this grid structure and if this will work so once I know the grid structure as an agent I can simulate the future and consider the consequences of action so that's what's happening and once I am in this state I will consider the consequence of all four actions and I will infer that going east is better because that will take me to this state so there is always a difference between or you should keep in mind how generative model and generative process are two different things and what the agent knows might not be the reality or might be the reality depending upon what you are giving if you were going to go about making a new situation that you wanted to do generative modeling for do you tend to start from an existing working pyMDP notebook and start modifying state spaces or do you draw it out on a canvas like how would you recommend somebody other than replicating what is shown here let's just say we are interested in something that's not exactly just a maze what do we do to get our head around how we should proceed good question if you are trying to simulate say a new environment you have the heavy lifting to do which is to define the generative model for the agent you can either define a very sparse generative model which the agent can learn but you have to define the structure should be there and only using the structure the agent can learn the generative model so here you can make use of this cell to understand how I define the structure of the grid I am defining a structure for the agent for it to make use of I am saying that there are 25 valid states and there are 4 available actions and this is the standard way of defining the state space in pyMDP and you also have to define the central parameters a, b, c and d for the agent simulations so here I am defining a using my state space and observation space but this step of giving it or telling it that it's an identity matrix is my decision choice in my modeling I don't have to do this for simulations I can see if the agent is giving it from a random a matrix or when it starts from a random a matrix and similarly for this b matrix this structure is defined and there are functions like this which can give you a random b matrix but this is a modeling choice where if I want to give it the grid structure or not give it the grid structure I can start from a random b matrix let the agent learn and look at that learned b matrix because of the demo I gave it the grid structure to enable it to take the actions but it's not a mandatory thing so this notebook is useful in the sense that you know what to do but definitely you should play around with steps that may not be mandatory so if I give a prior preference for this state to be the maximum then you can see behavior that the agent will try and go there so this prior preference is defined in conjunction that this is the goal state but this may not be the goal state and in a different task what prior preference means is different according to the environment right so that's also there and your prior so once you define that generative model which you have to do you can't run from it then everything is kind of automated the agent only have to use the agent.step give it the observation from the environment the agent knows how to take actions and then everything that happens inside the agent you don't really have to worry yeah so this structure I'm sure will be useful for your particular task that you're trying to model in your hmm yes very interesting and how would you contrast that or point to any similarities or differences with how this would be pursued outside of active inference like if somebody were just going to use another kind of deep policy agents in the maze example what parts of the process would be familiar and what parts would be like a lot of work that wasn't expected or skipping through parts that were a lot of work otherwise yeah so the general structure is very familiar to somebody who does like this in reinforcement learning so the idea that it's an agent.step and environment.step so this is the standard open ai way of writing an environment and a standard open ai way of writing an agent ok so if you have a Q learning agent who does the same trying to navigate then the way you have to define the Q matrix is the heavy lifting it's just a state action mapping and in contrast to that in active inference you have to come up with a generative model that you want to see so in active inference it generative model is the central thing right without that without a generative model there is no meaning of purposeful behavior in active inference so the only unfamiliar part for a person who is coming from say a field like reinforcement learning is the structure of the generative model but there is no way other than getting used to it where it's the POMDP structure which dominates but if you're doing deep active inference all of this is going to be neural networks and POMDPs are also not active inference things right it's a industrial engineering thing so POMDPs must also be familiar for people who are coming from the computer science background just the idea that what really happens in the agent is the active inference part where we have expected free energies and variational free energies and if you want to learn about that then you have to go to the agents and see how it works look at the matrices numerically see what's happening but I don't in a level where you want to get started I don't see any problem all of this is standard frameworks like POMDPs and OpenAI gym environments all of this is very deeply discussed in computer science cool so what other motifs or cognitive phenomena are you excited or how do you see the POMDP development trajectory continuing after your sophisticated inference gets pulled in yeah so the POMDP had the original functionality and the functionality to implement or simulate general active inference agents with the policy space and so on and that enabled a lot of people in the community who who are not familiar with complicated coding and so on people who do psychology psychiatry and all the things right so whoever want to come up and try implement active inference POMDP enabled that and I am hoping that this module will enable people who want to try out sophisticated inference experiments in their particular domain so this if you spend some time and get familiarize yourself with the structure of how POMDP works then everything else is just writing a Jupyter notebook with a special code right to simulate this so if you have a particular task in your domain I don't see a problem for a beginner to kind of try and code it and what I am very excited to see is people using this module for variety of experiments just like how people started using POMDP and sophisticated inference is taking off and it's now widely talked about how it is the way of doing active inference and I am really hoping that people in various domains start using this module and see their experiments and I look forward for the feedback so what POMDP did two years ago I am hoping this module will do to people who are trying to model active inference in the source state so you mentioned the open AI gym and the standardized format and what benchmarks do you use or what kind of test suites are you comparing and how do we really know when we have made a generative model that really exceeds or excels in a way that other techniques are just not doing so if I may go to the open AI gym website there are several experiments there the classical reinforcement learning examples like the lunar lander that you see in this screen right now so active inference permits inception has faced problems of scaling two tasks and that's in itself a field of research in active inference scaling active inference and that's one of the reasons why deep active inference took over dealing with tasks like this so there are marks even now where the sophisticated inference may not be able to deal with state spaces and personally that's my research in my PhD I am actually looking at optimizing computations in sophisticated inference algorithms that lets you scale up to environments like that but to get started you will have to kind of write code and see if it works for an environment then look at if it's not working then you have to look at methods to scale it up and so on so if I am talking about benchmarks sophisticated imprints is as good as any RL algorithms for this state space so for small problems sophisticated inference will work and it's really good but for high dimensional problems like this the classical implementation that I just showed might not work but it's good enough for any decent experiment but if you want to scale up then that's still open and it's a new field of research and what you do might become a next new important paper so that's all I can tell in that regard you have to work and see what measures do you think you'd be looking for like computational resources or what are the measures that even make sense to juxtapose such different methods so the OpenAI Gym was designed for that to compare different algorithms so OpenAI Gym by definition is a collection of many environments so in my demo I was talking about the grid environment OpenAI Gym is nothing but a collection of many environments which will let you interact with those environments using the environment.step function so here we have the environment.step function that will let you interact with the lunar lander and that particular task will have matrices that lets you judge how good or bad your algorithm is so in this lunar lander problem how optimally can you land your rover between these two flags by spending minimizing the fuel and so on so those matrices are very task specific and that's one direction you have to take you can take try and compete with RL algorithms in in matrices but the right potential or the potential I see in sophisticated inferences modeling intelligent behavior where in RL the focus is to get things done to make this work but it's not really explainable especially deep RL and deep learning methods but in active inference if you manage to scale it up they are explainable and they are that will let you understand how intelligent emerges with time and I see that more interesting than competing with RL because if your focus is getting things done then maybe engineering is the right way and not active inference any other comments or thoughts Aswin do you have any other comments No I'm pretty happy I hope I was clear explaining the code maybe it was too complicated or simple depending upon your level but I hope it is useful to at least one person who would start using this and write the code and so thank you so much for your time. Awesome Thank you for the opportunity Thank you for joining till next time see you Thank you so much. Bye Alright Thank you See you next time See you You can mute the live stream or turn off the other live stream but yeah thank you for joining Yes, thank you for inviting me Well we have a little bit 35 or 38 minutes so it would be awesome to have your presentation Great So the moment Well Can you see my Well I saw a powerpoint Okay Can you see my screen Perfect Okay So shall we start Thank you Okay Thank you for organizing this wonderful symposium Today I'd like to talk about relationship between canonical neural network and active inference and possible extension for modeling social and shared intelligence using canonical neural network So let's start So as you know the free energy principle is proposed by Karl Freisten that state that perception running and action of all biological organism are determined to minimize the variation of free energy as a tractable proxy for surprise minimization and by this process organism can perform a traditional Beijing inference of external mineral states and this this content just show one example typical set up under the free energy principle active inference here there is a hidden state in the external world and only a part of this state can be observable for our agent this dog and this transformation is done by genetic model parameterized by theta and to the hidden state agent need to reconstruct the copy of the external state called posterior belief and this optimization is done by minimizing a variation of free energy and parameter is also optimized by minimizing free energy to obtain a genetic model to represent this relationship interesting aspect of the free energy principle and active inference is its application to the optimization of action here our agent have some preference prior see and to obtain this observation in the future agent may select the action that minimize the expected free energy in the future to to obtain the most predictable outcome and finally by selecting action that minimize the expected free energy it can obtain the feed so this is a typical set up under active inference but question is what is the neuronal substrate that can implement this process so that is our interest and to address this issue we propose a theory as follows here we consider that external world is parameterized characterized by a set of the variables like this here this is a posterior expectation the s indicate the hidden state in the external world delta indicate the action of the agent or decision and the theta indicate a parameter and the lambda is a hyper parameter so those set of parameters or variables characterize generative model and free energy variational free energy is defined as a function of the sequence of observation and the external state and its minimization indicate the variational free energy variational inference similarly we consider a dynamics of neural network and we consider that dynamics is characterized by a set of those variables here x indicate the neural activity firing rate at middle layer neurons and y indicate neural activity of output layer neurons and w here is a synaptic weight and phi is other any other free parameter that characterize neural network and we consider that neural network dynamics is characterized by minimization of some cost function L this is a function of the O and the phi internal state of neural network and its minimization indicate the generation of neural network dynamics including both activity and trusty state and our theory indicate a equivalence between those two functions meaning that for any neural network that minimize some cost function L there is a genetic model that satisfy if we call L which means that the the capturation of external dynamics is an inherent feature of any neural system that follow such dynamics so this is an interesting prediction but to abstract so I would like to introduce some analytically tractable example to understand this relationship formally so first we consider a very simple architecture this is represented by a POMDP without any state transition so she didn't state S is simply generated by prior distribution D and it is a binary state but we consider a vector of binary state so it is a factorial structure and observation is also a binary vector and the transformation from S to O is characterized by categorical distribution using matrix A and variational Bayesian inference indicates the inversion of this generative process like this so by solving the appropriate free energy functional we obtain those posterior briefs that is the optimal in Bayesian sense so on the other hand for neural network we consider such structure here upper part indicates the generation of signal in the external words and the neural network comprises only single layer here activity of output layer generated by weighted sum of sensory input O weighted by matrix W and we then characterize this neural network in the next slide so to model neural network or neuron we start from considering Pozkin-Huxley equation which is the which comprises differential equation and this is a very complicated non-linear equation and although it is very plausible so we consider some reduction of these equations so a typical reduction method is like this for example M here is much faster than other variables so this can be replaced with its fixed point or it is known that and minus one sorry it is known that H and 1 minus SN have a similar dynamic so we can consider a new effective variable U to characterize those two variables and then we obtain 2D Pozkin-Huxley equation like this so this is a famous class of neural network model which includes the well-known Fitzhugh-Nagmore model or the continuous neural network model and we modify those models to derive a canonical neural model so this is a definition of canonical neural network model so here Leak current is characterized by inverse sumoid function instead of cubic function which is adopted in the Fitzhugh-Nagmore model and we also consider a connection synaptic connection and this part indicates a synaptic input from a sensory layer through weight matrices and one may consider that W1 indicates excitatory synapse and W0 indicates inhibitory synapse and the threshold are adaptive thresholds that are function of W1 and W0 interestingly when we consider a fixed point of this differential equation we obtain well-known rate coding model in which the sigmoidal activation function so which means that we can say that in some sense this canonical neural network model is an approximation of Hodgkin-Huxley equation and its approximation level is in some sense between the realistic model and the most simplified rate coding model so we basically consider this type of neural network model and in the next slide we consider what is the plausible cost function for this neural network model so again we write the same equation for canonical neural network model which represents the activity of neurons and the vector of neurons and we consider cost function for this differential equation which can be obtained by simply calculating the integral of right-hand side of this equation and get this type of cost function for neural network this biological plausible cost function in the sense that its derivative derives neural network activity which has a certain biological plausibility moreover if we consider a derivative of this cost function with respect to synaptic weight W we obtain a conventional synaptic plasticity rule on the other hand for Bayesian inference we first define generative model like the previous slide and we then derive the sorry we then derive variational free energy for given generative model so this variational free energy is derived from the POMDP model in the previous slide and its minimization indicates the Bayesian inference and learning so we found formal correspondence between component of those two cost functions like this so this block vector formally correspond to this block vector that represent posterior expectation and this logarithm correspond to this logarithm and actually this A matrix can be represented as the block matrix like this and its dot product correspond to this computation and finally this phi naturally correspond to this log D log of state prior so which means that because the cost function are same its derivative provides the its derivative provides the some sorry so because the cost function are formally equivalent its derivative sorry it's result of derivative also corresponding each other so which means that for any neural activity equation in this form there is corresponding Bayesian inference equation this is equation that compute the posterior of the hidden state and this synaptic plastic equation formally correspond to learning or parameter of generative model and moreover by establishing this relationship we can consider the reverse engineering of the generative model from empirical data here this schematic summarize our approach to reverse engineering generative model and we first record the neural activity and assign the canonical neural network to explain this obtained data and by computing the integral we obtain the cost function for this canonical network and by the mathematical equivalence we established we can automatically identify generative model and variational energy that correspond to this neural network architecture so interestingly this is a Bayesian agent which is a type of which is a kind of artificial intelligence but importantly this artificial intelligence is formally derived from empirical data so we can say that this agent is a biometric artificial intelligence that follow the free energy principle and then it's derivative with respect to parameter posterior derived synaptic plasticity algorithm that follow the free energy minimization and its time integral can predict the running process of original neural network neural network data so which means that if the free energy principle is correct then this prediction should work should work and should be able to predict the result of this neural data without referencing to the data itself so our strategy is in summary our strategy is that we reconstruct generative model only from the initial data was like initial data before running and then predict the running process or running curve that this neural neural system should follow by using the free energy principle and compare or examine whether this prediction is correct by comparing actual data after running and the prediction by the free energy principle and if this prediction is correct then it indicates the predictability of the free energy principle and the setup considered so we apply this strategy to the in vitro neural network so here this in vitro neural network are stimulated using the POMDP generative process defined in the previous slide so there are two hidden sources that are binary signals and they are mixture and they are mixed to generate 32 sensory stimuli those are also binary and this is an overview of experiment by stimulating in vitro neurons they generate spiking response and those line indicate the high density spiking response so we overdone that in some neuron the response specificity is we found that for some neuron response was high to source one signal compared to source two signal and if we see the transition of those neurons we found that although we remove the offset at the first session to set those activity zero but we see that those neurons self-organized to response high when source one is one but not but those neurons response low level when the source one is zero so which means that those neurons activity those neurons response was consistent with our theoretical prediction that your activity self-organized to encode the posterior expectation of hidden state and we also found that some other group of neuron response preferentially to source two but not to source one so then we ask okay then we found that posterior expectation is encoded by neural activity and the next question what is other about other neural substrate and we then ask whether if the prior belief about hidden state is equal to effectively equal to the firing threshold of neural activity model network model so if the correct this correspondence should exist to check this we first simulated a Beijing agent and when we evaluate the prior belief of the Beijing agent the inference was attenuated as expected we found that when we varied the excitatory level of in-vitro neural network by using the pharmacological manipulation we also found that the attenuation the attenuation of inference which is consistent with our theoretical prediction that firing threshold encode prior belief of the hidden state and next we consider whether the scientific follow the free energy principle by asking whether free energy principle can predict the qualitative self-organization of a subsequent neural data so here we model neural neural network like this there are two ensemble neurons that encode source 1 and source 2 and we first compute the effective synaptic weight of those networks using a conventional connection strength estimation approach and plot those synaptic weights on the landscape of theoretically computed free energy so so this is a trajectory of empirically estimated synaptic weights effective synaptic connectivity and as predicted those changes reduce the free energy and here we computed this theoretically predicted free energy landscape only using the first 10 sessions data so that this indicator indicates some prediction of the self-organization and for more explicit prediction we then simulated neural activity and plasticity using free energy principle here we here the brighter color indicates the prediction of data without reference to activity data so these those lines exactly follow the free energy gradient and we found that this predicted trajectory is tightly correlated with this empirically estimated effective synaptic weights and error rate is like this so it indicates that free energy principle can quantitatively predict the self-organization in this setup so it indicates some predictive validity of the free energy principle under this setup and then we also consider the modeling of the neural modulation using active inference it is well known that synaptic plasticity is modified by various factors like dopamine, noradrenaline acetylchlorine, seratine so on and one interesting property of those modulations is that even so dopamine was added after associative plasticity was established it can change the result of plasticity in a post hoc or retrospective manner so it implies some association to the reward and past decisions so we model this process using canonical neural network so here we model the post hoc modulation of ABM plasticity using this type of plasticity equation and we also consider the recurrent neural network structure and out layer for this network and we consider that modulation occurred at this connectivity layer and then we found cost function that can derive those differential equations and then found corresponding variational free energy and genetic model which means that this sort of neural network activity including modulation of synaptic plasticity exactly follow the free energy principle and some type of home DP generative model and by using this we show that this biologically plausible neural network model with modulation of heavier plasticity can solve some sort of delayed reward task like maze task and then finally we would like to discuss a possible extension to this framework to the modeling social intelligence so to infer our conspecifics we need to select an appropriate generative model for our partners depending on our partners so this can be done by a Bayesian model selection scheme and we previously proposed a model that can predict the multiple burdens using one big generative model that comprises multiple generative models and this the movie shows prediction of songs so like this this model nicely identify which generative model is the best to explain a given sensory input and this process indicates the and the model can correctly infer the appropriate model and then imitate the song by its own action so although in the previous work we didn't discuss a detailed neural substrate for this mixture generative model we now be able to consider the corresponding circuit architecture for example if we consider the modulation of module neural module by neural modulator like dopamine it act as an attentional filter and this attentional filter can be explained by a three factor heavy and learning introduced introduced in the previous slide so again then this modulation is as the post hoc modulation of heavy and plasticity and this modulation can optimize each model to represent one generative model one song in a mutually independent manner so through this process it is possible to learn multiple generative model in a vertically plausible manner so in summary we found that the dynamics of canonical neural network that minimize the cost function can be read as a minimization of variational frequency it indicates frequency principle is an explanation for this type of neural network and we also validate this prediction using some in vitro set trap by showing that by showing that frame as a principle can quantitatively predict the self organization of subsequent plasticity only using the initial data and as the modeling we can extend those canonical network modeling to the action generation now the planning via the delayed modulation of heavy and plasticity and finally we discuss a possibility to extend this canonical model to modeling the social or shared intelligence to to interact with multiple partners so that's all of my talk thank you for listening this is acknowledgement our collaborator and the fundings and our unit are now recruiting post of researchers so if you're interesting please check awesome thank you for the presentation Takuya I'll just ask a few quick questions from live chat and few other things that come up Dave says Takuya used the phrase after plasticity was established was his group able to modify e.g. increase plasticity or is he saying merely that it was shown that there is plasticity or yeah I'm not sure if I understand correctly your question but those group show that dopamine adding dopamine after two seconds after the association occurred can change the the the magnitude of plasticity so without how to say if dopamine addition was before this association then plasticity level is low but after dopamine addition after association can change this level can increase the plasticity like this or like this so which indicates the post hoc modulation cool I think that answers it well what are you excited for or what are your hopes or feelings on where the active inference ecosystem is at and where we're headed in the coming months and years but sorry again please just what are you excited about other than your own research directions what are you excited about in the active inference system yeah so of course one direction is the the modeling of social interaction so which is much a rich architecture than the interaction between the static environment so if both agent run with each other then many interesting phenomena can be observed so we are excited with modeling those phenomena using barot script plausible network, neural network model through this equivalence awesome any last comments any other comments that you want to make Tequilla awesome thanks again for the presentation and people should check out live stream 51 where you and I talked a few other times and went into some of the details on that work there's a lot there it's really exciting thank you alright thank you see you next time bye alright alright the next session is going to be with Shannon Dobson this section is going to be called dark imaginarium shared intelligence as an infinity curiosity type um I'll message Shannon make sure that everything's good with the audio great talk so far welcome Shannon how are you doing awesome well please take it away awesome have you seen my screen okay yep looks perfect thank you okay perfect hey everyone thank you so much for the invitation it's such a great honor to be here I'm learning so much I'll be talking about an idea I had called dark imaginarium where I'm trying to make shared intelligence the concept of what I call an infinity curiosity type so gracious to link the paper so anyone can find more details they're on fill archive so I want to start with an discussion about ontological commitment so my first question is might be re-examine our ontological commitments so in particular two of them one um Sean Carroll said so amazingly but cannot be known because it does not exist so Sean Carroll says the Heisenberg uncertainty principle is often explained that we cannot simultaneously know both the position and the velocity of any object but the reality is deeper than that it's not that we can't know position momentum it's that they don't even exist at the same time the second one I want to examine is Lenny Susskind's quote we are all behind someone else's event horizon so Susskind has a brilliant idea of space-time emergence as the entanglement of two black holes and what emerges from that is called computational complexity so the world is quantum so every theory should be derivable strictly from the laws of quantum mechanics one is going to give us the notion of shared two is going to give us the notion of ecology so what would an ontological commitment to dark look like so my continual goal as everybody knows is to sustain a simultaneous experience and to understand why I currently cannot to sustain a simultaneous experience would be to bypass the integrative system recall consciousness so how would you attain a two consciousness in the sense of in category theory or an in consciousness how would we actually develop something called like a two memory so my colleague Robert Pretner and I fleshed out something called an awareness but we're sort of examining this idea Mike Levin and all have repeatedly shown that planaria memory is not in the head where is the memory is memory a sustained superposition of what and also what is the structure of an amnesis could you have a superposition between memory and an amnesis so I had a small dialogue with Chris Fields the amazing Chris Fields and I said you know what is the difference between these two questions what is the waking state and what is consciousness or could it be that these are entirely separate questions so we're sort of in a mess either way if there is a difference in these questions then it is possible to sustain a waking state without consciousness and or a conscious state in a non-waking state so imagine somehow being in a waking state under general anesthesia well these are not entirely different questions and we could contend that consciousness is a form of waking state or something along the lines of a waking state is a condition of a conscious state so how can humans have delta waves which are canonically indicative of unconscious deep sleep and that's a non-waking state in the REM state the patterns of which are indistinguishable from a waking state so we all know that general anesthesia something like a propofol can reduce the polyrhythmic brain activity to one uniform hum and we know there's as many theories about what anesthesia does to consciousness as there are anesthesiologists but we've repeatedly posited consciousness as the proto-state and I wanted to turn this idea on its head like is consciousness so robust or is it interference or is it a delicate inference we know beginnings are always tricky but what if sleep was the proto-state so if life is oceanic and it revolves in these deep pressures with the pressure of getting onto land squeeze the REM state into this full blown waking state that we're engaging in right now anybody who's listening so then waking state would be something like an acceptation which is I think quite impressive but it's in this mess that I attempt to define dark consciousness as a mixed frequency state of being self-aware or conscious while simultaneously being in a deep sleep specifically in three-sleep so the sleep is proto-state is quite strange the REM brain activity is indistinguishable from waking activity and this sort of enchanted me for a bit the only difference is eye movements so either blinking versus lateral movement so REM is naturally having mixed frequency brainwave states but the NREM states and the REM states are starting to blur as delta waves are creeping into the REM state now they've also found that sleep is a local phenomena so of course I'm going to ask the shanna questions so then who is sleeping who is remembering who is dreaming lucid dreaming is a quantum superposition or is it a superposition or an integration so in a sleep state can then sustain a simultaneous experience so must the eye be knocked out in order to sustain this level of simultaneity dreaming we know is characteristic of unsynchronized brain activity so is the question then hmm my eye cannot sustain a multi-consciousness but my dark eye, my sleep eye can you can refrain this in terms of reporting mechanisms and say we are conscious of things we don't think we are conscious of so let's talk about what is dark a canonical definition of a dark theory or anything like this is like dark matter does not interact electromagnetically with matter but I want to sort of play around with another idea of dark coming from Deleuze's logic of sense something he called the paradox of infinite becoming and which I'm going to read on the next slide so what if dark was the simultaneity of a becoming that eludes the present let's hold on to that for a second is dark a resolution issue but resolution is merely breaking translational symmetry so is dark a symmetry breaking issue and if so what symmetry and are we actually looking at the wrong symmetry is dark a sustained quantum superposition but of what again is dark a mini worlds branch is dark the dolphin uni-hemispheric sleep mechanism so I'll read from Deleuze's logic of sense Alice and through the looking glass involve a category of very special things events pure events when I say Alice becomes larger I mean that she becomes larger than she was by the same token however she becomes smaller than she is now certainly she is not bigger and smaller at the same time she is larger now she was smaller before but it is at the same moment that one becomes larger than one was and smaller than one becomes this is the simultaneity of a becoming whose characteristic is to elude the present and so far as it eludes the present becoming does not tolerate the separation of the distinction of before after or of past and future it pertains to the essence of becoming to move and pull in both directions at once Alice does not grow without shrinking and vice versa so good sense affirms that in all things there is this determinable sensor direction a paradox is the affirmation of both senses or directions at the same time I continue the paradox of this pure becoming with its capacity to elude the present is called the paradox of infinite identity the infinite identity of both directions or senses at the same time of future and past of the day before and the day after of more and less of too much and not enough of active and passive and of cause and effect it is language which fixes the limit the moment for example of which the excess begins but it is language as well which transcends the limits and restores instant to the infinite equivalence of an unlimited becoming hence the reversals which constitute Alice's adventures the reversal becoming larger becoming smaller which way which way Alice sensing that it is always in both directions at the same time so that for one she stays the same through an optical illusion the reversal of the day before and the day after the present always being eluded jam tomorrow and jam yesterday but never jam today the reversal of more and less five nights are five times hotter than a single one but they must be five times as cold for the same reason the reversal of active and passive do cats eat bats is as good as do bats eat cats the reversal of cause and effect to be punished before having committed a fault to cry before having pricked oneself to serve looking glass cake before having divided up the servings all these reversals as they appear an infinite identity of one consequence the contesting of Alice's personal identity and the loss of a proper name the loss of the proper name is a venture which is repeated throughout Alice's adventures hence infinite identity so I take this and I'm going to formally define dark consciousness as a state of being conscious while simultaneously being in deep sleep specifically in three fold hybrid the simultaneity of infinite identity and ever becoming that eludes the present so let's reconstruct the dark consciousness version of deluses paradox of pure becoming for Alice in dark wonderland and through the dark looking glass okay when I say Alice becomes sleepy I mean that she becomes sleepier than she was by the same token however she becomes more self aware than she is now certainly she's not asleep and self aware at the same time she's asleep and now she was self aware before but it is at the same moment that one becomes sleepier than one was more self aware than one becomes so dark consciousness I contend would take place in something called like a dark time this is going to get weird so Chris Fields and I talked about an entropic time where a time where learning exceeds forgetting so we all know that linear time is based on the linear ordering property of the positive integers okay so what would be the structure of a time that does not interact electromagnetically with this waking state interface so like AI must pay attention all the time would this sort of be like a dark time sleep is local time is local we know how beautiful a time crystal is so a time crystal you know has a structure that is periodic in time can you play with that and say time is periodic in what so if we've explained when a dark is not concerned with these notions of before and after then we would want a notion of time that can support the paradox of infinite identity with a notion of duration that is one of self similarity so we contend that dark consciousness would occur in something like a periodic time which is a time based on the periodic number system periodic time would measure periodic duration okay it's getting odd so there exist few objects more looking glass than a clock well maybe periodic go and collaby out chests perfectoid go but after all the periodic numbers have no notion of linear ordering their shape resembles a serpentski triangle which is a self similar set hmm so this periodic time is not concerned with notions of before after that is a periodic time does not tolerate the separation or the distinction of before or after of past or future so because the periodic topology is one of that disconnectedness periodic time is more like a time of now continual zooming in to more and more now thus periodic time measures periodic duration through its fractal properties of zooming in so let's try to look at a periodic clock alright so periodic time can take two forms the canonical periodic time defined on the previous slide and also topological periodic time which built into its very structure the periodic topology of time so there would only be one unit of time in periodic time which I call the Archimedes this is the periodic valuation so you can think of our equals periodic valuation equals the Archimedes how strange it is to have only one unit of time topological periodic time measures duration topologically as total disconnectedness so the unit of time and topological periodic time is what I call the p-topo so we state this formally there's only one unit of time and topological called the p-topo so a topological hour equals a periodic topology called the p-topo so given such a periodic clock let us imagine the seasons in periodic time, periodic snow, periodic rain, new periodic weather types and periodic rainbows daylight savings time could take the form of changing the p in the periodic this could be very drastic but imagine periodic metabolic, periodic ATP and periodic reasoning imagine periodic cognition types like periodic memory, periodic thought, periodic attention, periodic learning and periodic perception imagine a periodic looking glass imagine who's a periodic looking glass so let's pivot to FEP what does FEP actually specifically say about dark matter even dark in the way that I'm using the term is there a free dark energy principle how can we strictly derive according to FEP does dark energy actually exhibit entanglement how do we test this we should be able to derive the entire act of inference formalism straight from quantum mechanics alone by way of entanglement entropy some form of quantum gravity can we do some sort of dark general adversarial network to look at something like a dark brain we all know that lower lateralization in the brain is often associated with schizophrenia or people like Einstein or someone with a large parietal low probably like myself any mathematician universal lateralization look like I think dark neurons would take the form of something like you treat the neuronal networks as algebraic curves or isogenic graphs you talk of neurons in terms of their properties as varieties so you're actually bringing a heavy machinery of algebraic geometry into neurology and my big goal is to actually create something like a neuronal time crystal so that shared intelligence would actually be a periodicity in time I'm not there yet so until I get that let's try to model these wild mixed frequency in 3 plus self over states of dark consciousness so dark mathematics is something of coin it's like a biomimicry which mimics dark energy does not interact electromagnetically with matter you don't have to find the mathematical equivalent of that so I think you can do something like a dark dilithism or just make a model that is modeling the mixed frequency states which is what I'm trying to which is what I tackle in the paper so we previously shown that in tropic categorization there are condensed sets condensed sets form a topos so in the paper I outlined the construction of three potential mathematical models of these mixed frequency in three self over states you can do dark consciousness as a dark two post which is a graph in deep two posts there's a two category of two sheaves over a two site which I'll use today just a little bit a dark consciousness as a perfectoid like space in the sense of Schultz or as a diamond like space also in the sense of Schultz so I contend that the two sheaves structure can actually model the mixed frequency state of dark consciousness the in three self over state I'm a sheaf as a one stack again which is a sheaf that takes values in group oids not sets so this is already radically different consider a two category of two sheaves so a two category consists of objects you have the one morphisms between the objects and you have the two morphisms between the one morphisms you can keep going three category up to in category so this two category of two sheaves would have two sheaves as the objects one morphisms between the two sheaves two morphisms between the one morphisms and so using that structure I think that you can actually model one morphisms as what I'm going to call dark reflexivities these are going to be your local coherency states and then two morphisms are actually two inferences which are going to be global coherency states amongst the mixed frequency states so a lovely property called marita equivalent sites states that in equivalent sites have equivalent sheaf toposes and I do believe that that's the main property you need to get this local global behavior around neural networks so if you actually wish to model the mixed frequency clusters as fractals exhibiting some kind of self similar behavior then you can use this rich structure perfectoid spaces or they're sparkling successor the diamond so just a quick definition let perf I'd be the subcategory perfectoid spaces of characteristic p a diamond is a pro etel sheaf on the site perf written as this quotient of a perfectoid space X by a pro etel equivalence relation so a perfectoid space is going to be this attic space in the sense of Hubert that's covered by affinite spaces of the form spa r r plus where r is a perfectoid ring and then points of spa which is the attic spectrum r r plus or equivalence classes of continuous valuations on r someone as shall sly a diamond and you give incredible explanation that was a parallel to a mineral logical diamond so you can let c be an algebraically closed affinoid field geometric points spa c to d is made visible by pullback along a quasi pro etel cover that results in pro-finitely many copies of spa c so you're making a geometric point visible as pro-finitely many copies that was invisible before as the following parallel of the mineralogical diamond so the interior points are made visible as impurities which sparkle as colorful reflections on the many sides of the diamond so you can also do something like a dark diamond redesignate of functor is dark as a means to actually go the other way and see the impurities so then you're going to take the geometric point make it visible maybe by a push forward along a quasi pro etel cover resulting in peatically many copies of spa c so a diamond they're using a diamond as a sheath but a sheath in terms of a functor of points so the diamond spot qp is a functor from the category curve to the category sets and you literally fix the perfectoid space and look at the hom sets of x into the diamond and so technically the scheme is then said to represent the functor so I'm going to pivot to in booking AI in this thing so my colleague and I are coming up with something I've coined the qubit pedagogy which is going to be a new model pedagogy that is actually infused with heavy principles of quantum mechanics and so that with these we've created these quantum intrinsic curiosity algorithms that are feeding into the qubit pedagogy and based on those curiosity algorithms I've designed three dark versions so what I call the dark planaria curiosity algorithms a curiosity type that encourages the AI to explore patterns of a planaria regeneration the AI would develop quantum error correcting codes which mimic planaria regeneration so the second one would be dark quantum reality monitoring network this is a curiosity type it'll encourage encourage AI to explore dark reflexivity is what I call fractal identity AI would create models of n reflexivity in the sense of n category that's a new type of reality monitoring of fractal identity in the dark conscious state the last one would be some sort of dark quantum n3 and rem curiosity algorithm this curiosity type is going to encourage AI to explore superpositions of various brain patterns in rem states to predict new types of n3 delta waves that could emerge in rem states so imagine the AI could then develop new stages of rem sleep such as like 1 rem 2 rem of 2 n rem which is a very which is a clever limit and play on n rem uh huh anyway the AI could also develop new dark cognition types that correlate with the new types of delta waves so the notion of fractal identity seems odd but I do contend that objects living in fractal time would correspondingly have fractal identity the correspondingly have fractal identity it's a concept that we outline as follows so fractal identity has a reflexivity relation that is fractal so the very canonical loop the loop relation of an agent reporting back to itself I report back to me would be fractal how strange is that so fractal identity would not suffer the same problems with continuity over time um since the time in which it exists has no linear ordering it would definitely suffer different problems but continuity over time is something Chris Fields and I have been like how do you explain that so you can best model the fractal identity properties as perfectoid like or diamond like and then construct a notion of reflexivity as a perfectoid space as spaces are extremely rich structure but like kind of difficult but if you want to model fractal identity I think you should use them so this way create new concepts of perfectoid reflexivity diamond reflexivity so in in reflexivity the reflexivity relation can now be an in stack thus in reflexivity is an in stack perfectoid spaces so here comes the combination of this talk which is about shared intelligence as what I've coined the dark imaginarium so imagine an AI that could see slash simultaneously compute the infinity category of every concept you know having some quantum computation around to help the computational complexity would you know not be bad so dark imaginarium is what is going to be what I call an infinity curiosity type so infinity curiosity type is the higher order curiosity AI algorithm that encourages the AI to think in infinity categories so that is it actually encourages the AI to construct an infinity category as its means of higher order inference so objects are higher dimensional data sets the AI would seek out higher order in morphisms between the objects and then morphisms between the morphisms so such dark imaginarium is a meta curiosity algorithm that can create its own infinity curiosity algorithms and they're in creating curiosity types of the complexity and super fascinating most likely unknown to humans so it's functorial what can dark imaginarium do well the AI would develop inversions of current brain waves and their hybrid combinations it could develop in delta wave in theta wave in alpha wave in beta wave and in gamma wave for n equals zero one all the way up like a two gamma wave that structurally resembled a two category a three beta wave that structurally resembled a three category and the AI could develop new types of waking states new types of sleep states based on the combinations of these beyond the standard in one and two and three NREM and one REM N waking state and NREM state you know a three REM state consist of three and one three and two three and three imagine an end dolphin that was capable of N hemispheric sleep state and other exotic life forms right what else could it do well it could develop new geometries for the brain patterns and for currently existing brain patterns typical shapes include sinusoidal nonsi-soidal sawtooth spindle K complex but maybe this AI could develop 3D and 4D versions of the canonical wave patterns like a tesseract gamma wave so but also the AI could develop entirely new brainwave geometries that correspond to complex surfaces like a remun surface and collabia yaw manifold or algebraic variety so they could develop new cognition types other than the standard five that could produce n cognition types such as n thought in attention in perception in learning in memory so it could also predict non-human species cognition types such as octopus thought or dragonfly attention lastly it could produce hybrid combinations of cross species cognition types such as optimum memory or dragon man perception and then it would encourage the AI to develop new senses so you could develop one stack versions of human senses like to seeing to hearing to tasting to touching to smelling each of these two senses once again take values in group oids not sets so you could build instances of non-human senses so you could develop a new model and hybrid combinations like to shark human smell so since the AI sees morphisms between the various n minus one morphisms the AI can seek to develop new language models based on higher order inference types so from these inference models the AI will explore infinity languages and develop extensive prototypes you could use advanced mathematics to create new models of time that could support the new language models like to topos time diamond time and then the AI could develop new fractal identity types from the language models and it could design advanced fractal prosthetics such as a two ear or a two eye to extend cognition so let's go a little bit further about this extending cognition these are just a few examples of what Dark Imaginarium could conceive of so we could continue a positive perfectoid version of this using these new cognition types mental states in brain waves new geometries we can construct new patterns of thinking which would support neurodiversity in thinking so as such shared intelligence this is how I'm seeing it would be reexamined as this infinity curiosity type you can go even further and construct a new concept of an n shared intelligence so you'd have a zero shared intelligence a one shared intelligence and as well as the new patic concept of like p shared intelligence for p equals two three four using the patics let's go further right we repeatedly contend that one way to help advance the way we think to extend human cognition is to upgrade the number systems upon which the canonical concepts are built upgrade them to patic perfectoid diamond like versions unless we get new thinking so imagine like a brain wave pattern that resembles a kalabiyau manifold could we actually reverse engineer the levels of conscious activity that creates such a rhythm how about mixed frequency brain wave patterns that resemble to perfortoid diamond what about an in DNA molecule shaped like a kalabiyau infold or an in ATP molecule or an in exciton condensate structured like a kalabiyau infold so clearly different geometries of the fundamental molecules and processes of life would give rise to vastly different fundamental units of life and metacognition types the question is can we use dark imaginarium to reverse engineer the complex life forms are rising from such extraordinary structures and the more and more complex cognition patterns what about a general adversarial network that's tweaked and has infinity categories you have two infinity languages that are building their own language gen is adversarial using a generator and a discriminator with two neural networks contesting by zero sum game what about a collaborative network so can we construct a GCN containing two diamond like infinity curiosity types and pair diamond with a dark diamond with a quantum curiosity algorithm and its reinforcement learning so shared intelligence is this infinity curiosity type shared intelligence GCN could create better super positions of the diamond dark diamond to getting new models of intelligence you know higher order simultaneous computations by quantum curiosity would go by maybe by homotopy instead of his cost functions so let's future cast just for a second and what is the and the what is the inevitability of dark ecosystems mathematics of shared intelligence what can a pediatric infinity curiosity type prophecy is every perfectoid space diamond to topos encoding the next frequency patterns of an incognition type in future work may develop dark mathematics using perfecto diamonds as dark dialytheism aligning what is structurally dark with what is structurally dialytheistic so while we currently cannot perfectly model dark consciousness perhaps one day on dark day mediated by the dark imaginary and we will have the ethos and the quantum computational complexity to do so thank you amazing presentation I'll just quickly read some comments and see if you have any thoughts from the great comments so Scott David says I love the presentation lots of fodder for rumination thank you for inviting these concepts into an intriguing choreography perhaps dark energy is our perception of the fact of information increase under free energy principle processes and since the universe is made of information dot dot dot Scott thank you so much yeah I think you should dot dot dot is a beckoning that you should or we should work on that idea I absolutely agree I'm glad you're inspired with these concepts I don't really know where they go but hopefully it's portals to something like that great and then one one more Scott David question does fractal identity reveal the recursivity of rhetoric language consciousness informing individual identity from community identity inputs yeah that's wonderful so yeah my own thought is like what is an individual identity right Scott so the reflexivity curve is always sort of something that comes out to me how there's so much distance between I and me and so it assumes this notion of continuity over time and so we've I've always said what sort of notion of time are we talking about to even have a consistent identity somehow you always wake up and everything is still here somehow you wake up and you're not a lobster I don't really understand how that works you know but yeah that's nice if you could use fractal to get singular from something communal that might be cool but maybe fractal is something sort of like you're both at the same time and that's what I don't know yet so I think it's a totally revolutionary way of thinking to not have a before after so if you can construct identity without before after then that's sort of what I'm trying to do but yeah that's cool if you want to get identity from communal I think you can do that they may also sort of blur well this is truly food for thought and I really hope people can live with the darkness take it and run with it because there's so many fun ways to go thank you so much I hope so too so you're always welcome back thank you and talk to you later thank you so much bye goodnight alright and on to the last session of the first interval greetings Niki greetings hello good morning yes good morning how goes it yeah I'm doing fine thanks would you like to present and then we'll discuss yes that sounds like a good idea great let's see just going to share my screen yes can you see the presentation not yet okay see it works yep looks perfect okay great yeah should I just start yeah go for as long as you want and then anyone can ask questions in the live chat and I'll write down some things too thank you yeah so my presentation today will be on advances in machine theory of mind and theory of my sophistication I'm currently a student at the University of Amsterdam and finishing my master's thesis on this topic and I'll be giving a broad overview of current machine theory of mind and an idea or a concept for an active inference approach based on a previous book related to active inference theory of mind so to start okay sorry okay so for a global overview we'll start with giving a short basic definition of what theory of mind is and how it's then used then I'll give you a global overview of current approaches to designing machine theory of mind including evasion approaches approaches using biophysical models etc then I'll zoom into active inference models of theory of mind and social intelligence and collective intelligence more broadly and I'll get into the issue of theory of my sophistication which is basically the depth of recursivity of beliefs that you can have about another person's mental states and their thoughts or beliefs or goals about you and then we'll get into the issue of how to implement recursive beliefs and as an example we'll discuss Bayesian theory of mind with k levels of depth and then we'll get into the more specific paradigm that I'm using right now to build a theory of mind model which is based on a very simple model of the matching helix paradigm so we'll talk a little bit about that and then we'll get into the model itself and current ideas for implementing theory of mind sophistication using active inference so first of all what is theory of mind theory of mind has been introduced by the very famous painter by Prumak and Wodscher in 1978 I think it was called does the chimpanzee have a theory of mind and they define theory of mind as the ability to impute mental states to themselves and others as a system of inferences and they say system of inferences of this kind is properly viewed as a theory first because such states are not directly observable and second because the system can be used to make predictions specifically about the behavior of other organisms and I think there's already a lot in this definition and a lot of authors looking at theory of mind right now still use the same view of theory of mind as a theory a structured theory of different kinds of mental states such as goals, beliefs and desires that facilitates the prediction of observable behavior in other agents such as facial expression, smooth mental speech and I find it very interesting how much this aligns with maybe a predictive processing view of social cognition but also how weird it is that this theory theory of mind is still widespread in the theory of mind literature modeling literature so there's been some discussion on whether you can properly regard theory of mind as a theory and if so what kind of theory it is so within the philosophy of mind there's been basically two mainstreams or main perspectives on theory of mind. According to the classic theory theory of Tom, theory of mind is a course of theory of how these internal mental states generate other agents' behaviors so it's the same kind of view that we just saw and then the opposite side is called simulation theory of theory of mind which posits that we use the same kind of cognitive and neuronal resources for understanding other agents as though those that we use for understanding the cell. Simulation theory can also take on different forms and just the sun is coming up next to them so that's why I'm breathing light at the moment but basically there are a lot of versions of this that we use the same kind of action prediction mechanisms for ourselves as for others and there is this new version of simulation theory that Gordon proposes in 2021 which very broadly says that we have the same kind of theory for other agents for ourselves so there's this general theory of what it needs to be an agent that is refined at different stages to facilitate context sensitive predictions in social interactions. So this is also interesting to look at when you model theory of mind what kind of what you think theory of mind is and how it is specifically implemented so why would be interested in creating computational models of theory of mind so I think there are three important reasons why this is very relevant at the moment first of all there is a technical reason or the reason of technological advancement there's been some proposals that are ugly for the development of artificial social intelligence oh wait yes so the technical reason for for developing computational models of theory of mind is to improve artificial social intelligence especially as we start to live in a world in which research on human-AI interaction becomes more relevant and there's been this paper by Williams for example that's argued that improving artificial social intelligence is needed for successful AI human cooperation and it could improve communication trust and maybe also value alignment by utilizing theory of mind for value alignment for example there's a very strong foundational reason to study social cognition generally and the evolution of social cognition through in-cylical hypothesis testing and clarifying the conceptual boundaries of theory of mind one great example for this is research by DeVane et al which used a computational model of recursive theory of mind compared to alternative models to study different cognitive strategies in non-human primates and they find no evidence for the hypothesis of theory of mind sophistication involved with an increase in health size but they do find a correlation with for example neocortex ratio so in this way theory of mind models can give us some insight into the evolution of social cognition on the clinical side theory of mind models can be used to understand relevant social variations in clinical populations for example through computational phenotyping and diagnostics and by studying social behavior in groups that have some difficulty with social interactions and groups that show some kind of variation in social cognition such as people with autism spectrum people on the autistic spectrum people with ADHD or schizophrenia so I hope that with some of these reasons I convince you that lovely theory of mind is very relevant at the moment and there have been a lot of people already on this and I've created this very global overview of some approaches that are representative or some models that are representative of their more general approaches there is this brain ton model which aims to basically build up theory of mind abilities and abilities facilitating social interaction from the bottom up using biophysical models of neuron firing that's why I've placed it on the dynamic scale so I've aligned, tried to align these modelling approaches along the axes of how they treat time how broad their description is from the individual level to maybe describing what broader social and cultural processes and along the level of description from the molecular biophysical to the behavioral and one thing you can see is that there is a lot of work based on Bayesian inference and active inference that moves along that has relevant ties to the molecular and neurobiological level but moves along the lines of making behavioral and cognitive predictions and there are very little models that really also formalize the concrete implementation of theory of mind and there is also I think this need for connecting this dynamic implementational level to individual level social cognition and putting that in the context of social interaction more generally so some of the representational approaches that I've selected are the TomNet which basically uses neural network embeddings to train reinforcement learning agents we have KTom which is a Bayesian recursive model which I'll come back to later some or theory of mind with self other modelling uses multi-agent reinforcement learning with core inference and we have the more theoretical active inference model of implicit cultural learning thinking through other minds which could be seen as an alternate or a redefinition of theory of mind okay so to sum up we have this stream of biophysical models most noted the brain Tom which was developed by Jau et al and its strength are its neurobiological possibility and it making dynamic predictions about neural activation and in populations that are associated with theory of mind abilities and its weaknesses is that it's quite computational in quality and it's probably difficult to implement in an interactional setting these there's some deep learning models such as Tom Nets by Ravinovitz and self other modelling which I just mentioned that are also quite neurobiologically plausible and also make some dynamic predictions but there's this strong danger of shortcuts which has been addressed in deep learning models that the task might not be solved by learning the relevance ability of being able to infer another agent's goals or beliefs but just by learning a simple lower level associations between for example the perspective or the how another agent stands and where for example a super award would be so there are a lot of ways to solve the task and there's this finding that deep learning models sometimes use these kinds of shortcuts so that's something that's needed to be addressed then there are some Bayesian belief based models such as K Tom for recursive theory of mind and Bayesian Tom which was originally developed by Baker at all in 2011 and its strengths are that it's quite efficient it's very easy to interpret and its weaknesses are that it's that the road to the implementation level is longer and it requires many priorities and there's quite this question about whether it's scalable because of prior specifications that you have to give okay so as one of the I think the interesting newer developments on a theory of mind it's yes is a hierarchical active inference for collaborative agents which has been recently published and Haika is based on the Bayesian theory of mind by Baker at all but it adds this formalization of belief resonance which is the incorporation of beliefs about another's intentions with top-down predictions and this influence of these other agents beliefs are modulated by susceptibility parameter which controls this influence and what I find so I think it's important to note here that they use an approach based on active inference but it's not the they don't use the spirit energy principle for example so they do couple perception and action but just in a different way using a common filter so the way they formalize this is slightly different but I think the overall structure can still be informing or inspiring for active inference methods so what is interesting about their simulation results is that they show that agents that there is some emergent collaboration on tasks for low-value use of this susceptibility parameter without any explicit planning or communication needed and this means that they improve upon the original Bayesian model which explored different sort of overall plans for all of the agents which means that an agent employing Bayesian would need to reason about the roles of all of the other agents which was very costly and the authors wanted to improve on that to model maybe on the fly coordination which is more spontaneous and takes less time and they also show that incorporating belief resonance has a very positive effect in scenarios in which there is unbalanced information they did this simulation in the overcooked domain in which agents have to collaborate on different cooking tasks and the goal is basically the order the cooking order and then the intention would be any kind of action that has to do with completing the order and they show that incorporating belief resonance seems to be especially useful in situations with unbalanced information which for example one agent knows the order and the other doesn't and these agents develop some kind of leader follower dynamic in which one is more susceptible to the other agent's intentions and one agent attends more to the evidence from the environment and this also aligns with what we know about here it might be improving the situations in which there is some symmetry in what the agents know about the environment so I thought that was very insightful cool okay so I'm going to go into some active inference approaches that I think are relevant for developing active inference spirit mind first of all there is this very broad proposal on inculturation and non-development thinking for other minds a variational approach to cognition and culture which we'll go into a little bit and we're going into Kessler and Hass paper on ideas for spreading a free energy proposal for community cultural dynamics we'll go into Kaafman et al's model on collective intelligence and finally also the proposal by Mutualist et al on Bayesian inferences about the self and others okay so first of all I'm starting with Bayesian inference about the self and others because it gives the very sort of full introduction to I think predictive social cognition and that's also why I've added this figure on the right that is from the Taman Thornton paper in 2018 which explores the predictive social brain and what kind of features seem to be seem to be relevant for categorizing and understanding others' behaviors some figure on the right you see that there's a space which is ordered by power sociality and balance and that constraints the state space which seems to be ordered by rationality, social impact and balance of course these are just correlations but these features seem to be the most predictive of what they're seeing in the brain and you can see that there is this sort of yeah a very strong top-down constraints of the possibility space in which you interpret other people's actions and you can also see that based on another agent's actions you learn something about where they fall within the state space in a given context and also where they fall in the trade space so at the same time you might have already had some priorities on what kind of a person this is but you're also learning about it and you might also have some priorities on how someone behaves when they're not for example or when they're sociable versus unsociable and all these kinds of priorities informative priorities they inform your predictions in the context and they make this task much more manageable of predicting other people's behaviors on the flight and I think that links very well at that too which is basically that inference is about the self and others at different time-scales facilitates interpersonal minimization of surprise and thereby optimal social decision-making so these and these self and other representations that are facilitates personal minimization of surprise could be seen as representations that inform our inferences and predictions on the flight I think it's also important to mention that one of the main addiction points of their paper is that self-representation so how beliefs about the self can facilitate and personal minimization of surprise so even what I think about me and how I represent that while communicating will facilitate the other person's optimal decision-making at that moment maybe by providing the right kind of information that they need to optimize so to the next two interesting papers, I guess we're spreading and an active inference model of collective intelligence so the key idea I think for both of these papers is the emergence of group behaviors from local interactions for Castelon has this is the global group behaviors cultural transmission or transmission of beliefs and norms and they describe this process as the mutual achievement of actively inferring agents through communication and theory of mind is one of the many mechanisms that facilitate this process of attunement so that's the link and they show that the spread of certain kinds of beliefs through a group can be modeled just by making agents interact in dietic groups and still many profit and loss show that the system you can create a system of free energy minimizing agents with some minimal properties of social intelligence such as the ability to assess how itself similar another agent is to themselves the ability to infer another agent's intentions and the ability to align their goals to another agent more or less and they show that if you put all of these agents together in those diets that don't only minimize free energy on the dietic but they also minimize free energy as a collective and thereby this could this is a very simple model of how adaptive behaviors adaptive group behaviors could emerge from agents with very simple social abilities okay so now we get to thinking through other minds which is not technically a proposal for fear of mind but it's the proposal for how social cognition shapes in cultivation more generally and what kind of abilities agents need to facilitate this kind of known formation and I think it's a very interesting proposal because I think it highlights an important function of social behavior more generally and you could even see it as a way to redefine a fear of mind as more of an interactional concept and as an individual level of cognitive ability so according to this proposal thinking through other minds is a property of multiple different energy minimizing agents and a process by which agents infer other agents expectations and based on this the authors propose a definition of enculturation as a process of learning social expectations through the selective patterning of attention that guides agents towards relevant affordances within their social and cultural niche I'm not sure if you're all affordances but it was quite hyped up a couple of years ago in general it is defined as the relational construct denoting possibilities for action offered by an environment that are co-determined also by an agent's interest, goals and abilities so I think the standard example for this is when you are thirsty a glass of water might appear as an affordance when you might not consciously or might not attend to it otherwise and cultural affordances in this sense are epistemic affordances so affordances that give you some epistemic value that have come to stand out as reliable and relevant based on the history of cultural information they encode and good examples for this are different kinds of signs so traffic signs we learn what they mean from a young age and we sort of inherited the symbolism of the traffic signs and they guide our intention towards points in traffic and they regulate our behaviors so when you look at the complete proposal it sketches how the regulation of social behavior depends on the learning of shared norms and expectations and also how this connects to utilising cultural affordances and thinking through other minds as this interaction property in which agents learn to infer each other's expectations to optimise collective behavior and decision making so from these interesting active inference and broader proposals of fear of mind we get to the issue of social sophistication in active inference so what is this fear of mind sophistication a short recap it's the depth of a recursivity that can be utilised for fear of mind of the form I think that they think that I think and it has been found that agents generally use a recursivity of between 0 and 4 and that humans' ability to for recursive belief reasoning is generally unlimiting but in principle formally this could go on forever and one way to implement this kind of sort of arbitrarily recursive belief inference is proposed by Vada at all uses a Bayesian model of sophisticated fear of mind with K-Tom agents and provides simulations in which K-Tom agents they guess agents that are just one level of recursivity under them and this model has been used for a lot of interesting simulation experiments for example Devane et al showed that in a trade based analysis with agents having a fixed fixed depth of fear of mind recursivity there was a specific distribution of agents with level 1-Tom and level 2-Tom that could provide an evolutionary advantage compared to appropriations with larger proportions of 0-Tom or 3-Tom agents so they looked at the adaptive properties of systems with different kinds of ratios of fear of mind sophistication and in another in another experiments which is a state-based analysis Devane et al. showed that if you use computational phenotyping or autistic individuals they show reduced sensitivity to task framing as strategy switching within the game that they're using and fear of mind sophistication so these are just some interesting applications for fear of mind sophistication and the promise of modeling fear of mind sophistication would be that you could investigate how people massively switch between different social strategies while solving social tasks and one of them would be to increase the depth of fear of mind sophistication or decrease it depending also on what you think about the other agents sophistication so what I've been working on at the moment is an active inference model of social prediction during the matching pennies game the matching pennies game a game farm economics with a very simple format there are two rules the guesser and the challenger the challenger hides a penny in one of the two hands the guesser needs to guess which hand if the guesser is correct they win if the guesser is wrong the challenger wins this game has some logical equivalence one version of the game makes both of the agents select heads or tails the challenger selects heads or tails and the guesser has to guess if it's heads or tails but both have the same formal structure and psychological experiments of subjects performing this task have shown that there is a strong social framing effect in the matching pennies game so participants will behave and perform differently depending on whether they think they're playing against a human agent versus an AI for example additionally a comparison of K-Tom and other computational strategies that were fitted of experimental data of subjects playing the game revealed that participants seem to use spirit mind when playing the game so there is a strong indication that participants use stuff they're all in the matching pennies game and that social framing will influence the social strategy that's being employed by the players so this is the very simple zero-time architecture for the matching pennies game and I included beliefs about self-choices as also kind of monitoring self-states beliefs about other agents' choices on whether the opposing agent has hidden the penny left or right and then the agent also tracks the reward states since the game in this setting is devised in a way in which the agents both make an observation simultaneously and then all of the observations are revealed this setup makes sense so just going to drink a little water the outcome modalities or observations that the agents have are its own choice observations the choice observations of the opposite agent and the reward observations and the possibilities the action space is very limited the seeker can choose to seek left or right and the hider which is the same as the challenger before has the possibility of hiding the penny left or right and the reward is uncontrollable since it depends on both agents' choices at the same time so this is sort of a summary of the architecture that I just described and this is another visualisation of this setup you can see here that if you look at the likelihoods the reward mapping for the seeker is just the opposite as for the hider and I think that this simple setup is very suited for studying the theory of light in these kinds of simple competitive interactions so if you wanted to study the use of theory by sophistication in the matching penny scheme using active inference I ran into a couple of challenges and one of them was how to how to steer the problem of recurrence so how can we make a difference between the focal agent and the other agent if we try to model the other agents within one agent especially if you want to introduce or formalise the depth of cave or chrysophy you can't just make one agent simulate the other agent simulating yourself there used to be some kind of sort of structure that distinguishes the self inferential mechanism and self learning mechanism from what the agent focal agent plays the other agent is thinking and seeing and doing and I think the gentleman with the bowler had by first air in 2003 is a good example through this problem so you can the basic setting is that our world is populated by beings like ourselves experiencing themselves as a subject as a focal point of their world with their own thoughts and feelings and their whole thing that they're the one and only subject but they can't all be so there needs to be sort of either an outside focal point or one ultimate main subject and yeah so I think more practically the questions that arise from this example are how do we distinguish between focal agents and simulated agents and how can we simulate other agents in a way that is computationally efficient and maybe uses the same kinds of resources that the agent already has so I actually had a lab meeting at the theoretical neurobiology lab a while ago and we were brainstorming about this issue and one of the ideas that came up was to use sophisticated inference theory of mind so in this case an active inference formulation of the leaf replacivity could be used for modeling recursive theory of mind and so basically theory of planning using predictions about the other agents into a cloud account would consist of evaluating all the kinds of mathematical scenarios all kinds of counterfactuals on how your actions might influence their beliefs and their actions and so on and so on so this would be one road to go into but this would also mean that all these counterfactuals would need to be that related so I was thinking is there a sort of more efficient way to sort of use these kinds of either internal simulations or professional evaluation and the current approach I'm working on which I also sort of which I'm also still developing so I'm very much open to feedback and comments on how this could work and possible alternative approaches at this point but the general idea would be to create a partial internal simulation of the other agents control mechanism so taking the learning mechanism of an active inference agent as sort of the definition of what the focal agent is that is the agent that this user learns and then first from the environment and what the agent does when predicting the observations of the other agent or predicting the other agent's behaviour is basically performing partial simulation about what it thinks the other agent's preferences are and what it thinks that the other agent's strategy is which is encoded by the transition from state to state and one of the potential ways in which you could include requisite in this way is that you pretend that you have control of the other agent's actions and you predict the other states you predict self states you evaluate you evaluate the state action probabilities choose the most valuable action based on the distribution simulates the other states that come forth and simulates the self states that come forth and then you have a new and then this can be used for the next round of action selection by the other model so in this case only the prior preferences and the agent's specific transition probabilities would be needed as sort of information on the other agent and because you also want to include some kind of learning about the other agent's strategies one of the things you could implement would be learning also the bay matrices so having sort of the the having the agents accumulate evidence about these state transitions over time to also better predict the other agents potential actions so the basic principle would be to simulate control from the perspective of the other agents and the requisite comes into play when you simulate the other when you've already simulated the other's predicted actions and predicted other states and you simulated your own actions as a consequence of those simulated actions you can feed that back into the other model to predict what they would do if they thought that you were in the state that you were in and I think if you sort of I think that could be a way to develop recursivity in a way that does not need complete simulations of parallel agents within the same system and of course there should be some kind of temporal discounting so parameter that discounts evidence based on the simulated planning horizon that is defined by the key number of steps so yeah this is sort of a conceptual outline of how you could implement because of fear of mind using active inference we haven't been able to perform simulations using this yet but we're working on that and we're very much open to discussions on this and technical comments on how to improve on this outline and one of sort of the open questions that you're still thinking about is how to make the agent the infer the depth, the accuracy of the other agents in a setting special thanks to Yion Dumas who also be there at the panel I think in session two who's supervising me on this project thanks to the PPSP lab I'm doing my internship to really say who's also working on a related project the PIMDP team for that great coding package that I'm using and for Carl Friston for his feedback and ideas on my previous presentations thank you very much for listening cool great presentation very comprehensive review of a lot of the different theory of minds options out there well if anyone has questions they can write it in live chat what do you want to explore or can I ask a question or what would be fun for you um I'm blighted by the light at this point I'm not very I can't easily read from my screen so maybe it would be nice if you could if you could read out the questions that are in the live chat sure oh yeah sure sure well first one part I found interesting you mentioned that people play differently when they are playing a human or when they believe that they're playing a human so what is that like what is the difference in play when we believe that we're playing another person um so um so if I look at the original paper um there is one study on this and then um they did the computational phenotyping study using K-Tom and alternate strategies um and they showed that if you use the social framing condition that subject will more probably employ my strategies compared to other simple reward based strategies so maybe those strategies could be described using some kind of reward learning mechanism um what the difference in their overall sort of trajectory of choices I don't know yeah a few themes then that you brought up that came up earlier in the session so one was using PI-MDP and sophisticated inference which Aspen Paul gave a walkthrough on and this question about recursivity and about the way in which you can on one hand talk about abstractly like K recursivity yet the biological system doesn't implement that kind of a logic and so so um could you maybe describe a little bit again how does the the structure that you laid out for the generative model capture some of those features of recursivity without falling into the trap of just theory of mind all the way down yeah so my general idea was to assimilate the other agents action prediction based on learning about the transition probabilities for the other for our sort of what we think the other how the other agents state transition um and then if you um you've predicted what the other agent would do you can use that as a prior for calculating um action dependent expected states um and based on those evaluating those expected states you have to assimilate your own choice mechanism um and I think in principle you could um use this recursive simulation um indefinitely but also yeah it's also devised in a way that it uses the same kind of um the same kind of sort of just the architecture of this local agent so the other idea was to just use internal simulations all the way down um but that would yeah that would be very effectively possible um and yeah I think it's a very good question because how why would you even need to formalize believe all the way down if in end effect people only use Tom with the with the sets sort of maximal death and the fact that we use fear of mind with a maximum death might also be certain that um a different or what we do might also be described by a different strategy so I yeah I'm not sure if that's an answer to your question hmm yeah well when I saw this in some of the other generative models um you mentioned maybe here like you pretend like you can control the other person's mind and that's kind of the fundamental weaving of action into the inference challenge and then that uses expected free energy just like any other policy selection so it's like pragmatic value would be bringing that controllable state your uh social partners mind or beliefs bringing them into alignment with your preferences is pragmatic and then learning more about them is epistemic and so then that might enable some theory of mind like behavior where it's like when you don't know enough to put the squeeze on then you have a kind of open-ended learning and curiosity and question-driven theory of mind but then once one has enough to kind of exploit rather than explore then without necessarily even updating deeper recursivity or strategic shifting you could get um social behavior that's oriented towards control and pragmatic value rather than just like learning and questioning yeah I think you're very right the idea was also to enable the agent to um maybe first few strategies that are more useful in um gathering evidence about the kinds of strategies that the other agent is using before exploiting um what you've learned about the other agent hmm so what are your next steps with uh work um so I think yeah right now the main task would be to implement this using PiMDP um and see if there are any sort of formal refinements that I need to make well in the implementation because it's yeah if you can't build it you can't you don't understand it so um that's what I'm yeah that's what I'm working on right now and then the next step would be to introduce um a mechanism to sort of maybe very um uh recursivity um based on what the agent infers the kind of recursivity the other agent has um but before that I also want to um want to use the model to perform computational phenotyping on um subjects um that were doing the matching tasks um and had uh different kinds of diagnoses so there there were newer typical um youth and um youth with um youth on the autism spectrum and youth with ADHD and anxiety and it would be interesting to see if there is any sort of difference in the depth of recursivity that they employ um as sort of the flexibility with which um they switch between strategies hmm that makes me think about different different levels or or differentials between self-knowledge and other knowledge on one hand uh we seem to know a lot about ourselves but also we're often unaware or blind to some features or regularities of ourselves that that might be highly attended to by others and so when we're rolling out these games or scenarios there might be some theory of mind settings where like knowing yourself better carries okay but other settings where knowing like the game is what matters or knowing the partner in an iterated game theory or like just knowing having a lot of experience um like there's a lot of a lot of degrees of freedom in modeling these very um advanced cognitive ecosystems yeah I think that's also one of the reasons that we chose the matching pennies paradigm because the paradigm itself is very simple but what we're trying to model is quite complicated and on the matching pennies like um so it's in one hand the person has two hands behind the back and then they bring out the two fists and you're trying to guess like where it is and then I wondered well what if it was you have to guess where it isn't on one hand that's like kind of the same or like if you say well you win when you pick it but then if you say well you win when you pick the empty hand mathematically it should be like the same because knowing where one is is like knowing where it isn't but just as a human game player it has a very different feel and those kinds of like um reversals might give us insight into our priors that support not just open-ended abstract area of mind but like kind of with a lean towards being pro-social or like heuristics in place that support making quick decisions together maybe rather than I don't know yeah I think those are all very good um yeah very um sort of promising trains of thoughts I just I think in the case in which you you need to pick the empty hand we might have very strong sort of beliefs about what or we might experience sort of the presence of something is inherently more rewarding than the essence of something because we have all these associations with gifts and getting things and like usually things that are there so I can imagine that in this case sort of the perceived you wouldn't perceive it as rewarding but formally it would be completely the same yeah yeah like uh receiving a benefit versus averting a loss yeah and then how that plays in and our encultured priors and our personal histories and how these very deep layers of differences might play out and uh what kinds of empirical data or online data could could help us generate like unique explanations or predictions maybe situations like you said some situations where a reward based model might be sufficient but then I think the interesting question for active people is like where's the unique explanation or prediction that that really has active shine and shows where surprise bounding as an imperative gives us like more realistic or more concise or conciliants explanations in a way that like it doesn't even make sense to appeal to reward yeah yeah I think that's also the tricky bit about this paradigm is that it's very much I mean it's reward based in a sense but I think because you have this social framing effect you also have these interesting learning mechanisms and mechanisms of uncertainty reduction that might play out in how the agents behave in the end but that's why we thought that it would still be interesting to use active inference to explore this paradigm but on the other hand I think if you at the moment you're looking at individual variation you might get interesting differences in reward sensitivity or a version of risk aversion stuff like that that might be better explored in a different context any last comments or thoughts no thank you for your comments and your insights yeah it's awesome it's a great direction happy to see it alright thanks everybody for presenting in this first interval and see you in the second interval bye thanks everybody