 Great. Hello and welcome everyone to Act In Flab. Today is April 28th, 2021. It's Act In Flab guest stream number 5.1 with Martin Beal. Martin, thanks so much for joining us on today's stream. And we're really looking forward to hearing about your perspective and what you have to share here. So if anyone in the live chat would like to ask questions, we'll be relaying those to you during and at the end of the stream. Other than that, please take it away and thanks again for coming on. Thank you, Daniel. Welcome to my talk everybody. I will talk about, or I will talk about, at least in the background, about a paper I wrote with Felix Pollock and Rio Tecanai, which is called Technical Critique of Some Parts of the Free Energy Principle. But I will not go directly through this paper. I will kind of try to give a more updated perspective. Yeah, what else to say? I work at Araya in Tokyo. And this work is also funded by the Tibetan War Charity Foundation. And Felix was supported by the Monash University. Okay, so here's the content of this talk. First I'll make some preliminary remarks and then talk about my perspective basically on the Free Energy Principle and how it can maybe maybe split into two parts. The construction of agents and the identification of agents. And then I'll talk about what the FEP agent construction is and what the FEP agent identification method kind of is. And then I'll talk about the relation between those, or at least the claimed relation between those, and then I'll go through some issues with this agent identification method and also with this relation. Okay, so yeah, like I said, the main publication is the same name as the, has the same title as this talk and came out this year. And it resulted from an effort with Felix where we actually set out to do like a discretized version of the Life As We Know It paper from 2013. But we failed and then in the end we wrote this critique. So yeah, I will give this, in this talk I will give my personal view a little bit of the Free Energy Principle and why I find it interesting. And like I said, I want to go through the paper in detail and instead summarize how I currently see the issues, also the issues that we mentioned in the paper. But the FEP, the Free Energy Principle has, why we were writing this paper, it has changed and so my perspective has also changed. And but if you have questions, particular questions about the paper then you can of course ask me afterwards. Okay, so here's the, maybe a little bit more detailed overview. So in my opinion, the Free Energy Principle can be viewed as two parts. And the first part is basically what maybe you've heard is kind of the statement that Markov Blanket implies, Markov Blankets imply Bayesian inference. And I, so from my perspective, this is kind of a way to identify agents in stochastic processes. So this is one part that is contained in the FEP literature. Then another part is about how to construct artificial agents for, for example, for reinforcement learning or partially observable Markov decision problems. And I call this the constructing agents part. And the claim, the claim is that, like if you read papers about this construction agent part, the claim is that they, that this construction is based on this identifying agents part. The construction is based on this Markov Blankets imply Bayesian inference part of the literature. And I will kind of argue that this is, has not been established and I also kind of think it's questionable. At least in the current state. Then I also present some other issues with the FEP among those, those that I mentioned in the paper. And these issues. So one thing I want to say is that these issues, they kind of affect, would affect, I mean, if they are, if I'm, if I'm correct about those issues, and they affect the usefulness of the free energy principle or agent identification. So, so then we couldn't identify agents using the FEP and this also would question it as a theory of life is or of life general or life as we know it if you want. And then it would also affect this claim that these free these constructed free energy principle agents are somehow based on first principles, more than, let's say some other reinforcement learning agents. What, what all of this, what all of these points in my opinion won't affect is that free that these free energy principle agents like free energy minimizing agents that are constructed in the literature, that they work, or that they work at all basically, or that they can be useful for some research. I mean, I think, I think those. I think that these agents work is kind of independent of this whole Markov blanket implies Asian influence. But maybe I'm wrong. Okay, so what's this. What are these two things in the construction of agents and the identification of agents. So like I said, I think the literature can be split into these two parts and like how to construct an agent or a controller. And the other one is how to identify to be an agent in a stochastic dynamical system. And, yeah, so, and like I said also before the, the is first method how to construct agents or controller is claimed to inherit from the second method. These are still claims that are current so as a paper that came out just this year or actually a week ago. So, I think it's called sophisticated inference. Okay, so what do I mean by construction of agents or what is this. What is the problem of construction of agents. It's a, yeah, a different problem than the identification of agents. That's kind of the first point I want to make that these two problems are different in general. So, yeah, in, in, in artificial intelligence research what what people are occupied with is the construction of intelligent agents or some given environment or task. Or like, yeah. And, yeah, so basically what what these people do in artificial intelligence research or engineering is programming robots or virtual agents. And of course the robot is then kind of fixed, fixed environment or if you have a virtual world, then this is also fixed environment. And then if you have some reinforcement learning problem, then you have kind of a task. Okay, so, like I said, we in these cases, what we have is some environment that is given and I, I write this here as a Bayesian network. And then we construct a controller and then this what it basically does is we have some environment states that I call it. And sensor values that depend on it. And we have dynamics of it. So, so basically towards the right time increases, time step increases. So we have, we have some dynamics of this environment, which tells us how it changes under the influence of actions, and depending on its previous state. And the environment produces sensor values and what an agent does is takes these sensor values. And updates its internal state, which is called mu and produces actions accordingly. So as you see here, mute depends on both the sensor values and the actions. How exactly these edges look, there's some variability in it, but it's roughly one one idea. Yeah, most of them are roughly equivalent. So, like I said, or a bit more formally, maybe, either of T is an environment state, or an environment or a task is defined as by an environment state sensor values, some kind of action interface. So some dependency on the external variable. So this dependency of update of it on the action. And the dynamics, which means if we have, if we are in discrete time, then the dynamics can be specified as this Markov kernel or conditional probability of the next or the, yeah, the next internal state, the next sensor value conditioned on the past internal state, past sensor value and the action. Just the influence of the agent. And in continuous time, we could also formulate this and one way to formulate this looks like this we give the time derivative of the internal state as some function. And it's own state and the sensor and the action, plus some noise and similarly for the sensor dynamics. And maybe it's not so important, but in general, you can have an environment can have some kind of reward that it gives to the agent as part of the sensor value or it may not. Okay, and then what's an agent. So an agent has this internal or memory state. And it uses actions. And it also has an interface for accepting or sensors that are produced by the environment. And it's dynamics is can also be written as this kind of conditional probability. Or similarly, if time derivative is in the time continuous state. And maybe the, the main part that's going to be relevant for some something later in this talk is that we can combine the and we can combine such environments and agents. And then they form a Markov process. So then we have some joint variable X. And the product of these two conditional probabilities, the one for the environment and the one for the agent. They induce this Markov kernel of the joint process. Similarly, in the continuous case, we can see the whole thing as one, the combination of the environment and the agent as a whole as a single very as a single Markov process with state X. Okay. So that's kind of what we maybe should keep in mind. Markz on this. Usually we have, we construct these agents to solve some problems so we have some kind of reward. So this is the case in reinforcement learning for POM DPs. But we can also have agents that have just intrinsic goals and they don't need any reward from the environment. These things are usually called intrinsic motivations. The FEP agents from the literature, they actually kind of have both within them. So they contain some part that's like an intrinsic motivation and they also can, they have some part that is similar to a reward. So basically the, the intrinsic motivation part is, it's like a maximization of information gain and the instead of reward maximization, we minimize some divergence from a desired distribution. But at least on a high level. This can be seen as So these two things are kind of combined in the free and in the expected free energy. Yeah. I mean, there are some tricks to this, but roughly this is true. Okay. So, so that that's that's what I hope it came across what I mean by construction of agents. I don't know if there were any questions already. Not yet. Continue. Thank you. Okay. So then what do you mean by identification of agents, which is maybe a bit less common practice. So first, maybe let's ask the question what is an agent in general. And So very roughly, there's a definition by the And they say to be an agent, you have to be something that's individually that can act and make things happens and can act, which just means to make things happen basically. And it can act to achieve the these actions are performed in order to achieve some goal. So an agent has also a goal. Maybe something that has a goal and acts to achieve actively pursues this goal. And these, these agents that I just talked about these constructed agents that we constructed artificial intelligence. But also in the FEP literature. Because they are kind of by construction. They are individually because we kind of know where their boundary is. If you ask some engineer what they have to do to construct the agent, they know exactly what to do so they seem to know where the agent is or what the agent is. And of course, since we program into them. The intrinsic motivation or some kind of reward maximization mechanism they are also co directed by construction. Okay, so, but then this is not this just what what agents are so what's what's what about identification of agents. So the at least for me the origin of this question is the kind of the problem of agents in physics. So the physicists they tell us that everything is determined by the laws of physics. So that means everything that happens is made happen by the laws of physics and not by any thing within this. So in this, if they are correct then nothing within the universe can make anything. But that also means that nothing can act to achieve a goal because nothing can make anything happen so you can't act to achieve some goal because you can't act in the first place. So this seems to suggest that basically there are no agents in the universe like it must be some kind of illusion. But at the same time, I mean we are here. We are agents, and we kind of want to, we kind of have goals, or at least it seems extremely likely and it also seems like there are things that don't. And we built these robots, and they seem to be quite real things and have goals also. So it's kind of, so maybe there is some way there should be some way to make these two things compatible. So what we want is, at least if you're, if you like the physicist's perspective. And also you like, you like to think about agents then you maybe you want what you want is a formal theory of agents that is compatible with physics. And so one one problem that such any such theory should be able to solve is that if you have some kind of universe and in physics that's basically just a dynamic system. Then what and where are the agents so if you have if you have a formal definition of agents or a former theory of agents then this is the question that it should probably be able to answer. So here's the universe. And what we want to do is, we want to check where inside this long history of the universe. There are these things that are maybe can be called agents like this. A bit more formally. So we have some kind of in physics we have some kind of dynamical system we have a state space and states. And we have a dynamical law that is looked maybe looks like this, the time derivative of why is some function of the. The derivative of the state of the universe is a function of the current state of the universe. And then we have. So, these things have trajectories. And if we select one of those trajectories by some initial condition why zero. And then what we would expect from a theory about. A theory about agents and one that can also identify agents then what we would want is that this theory gives us like once we give it such a trajectory of a dynamical system, or maybe the trajectory and the dynamical system. Because what is the state of agents contained in this trajectory, and it should tell us this, including what is basically its body or its individual, what is the individual. What are the actions and what are the goals. Okay. So this is what I, this is kind of the agent identification problem. The FEP. That's something like this. They start, not directly maybe with a deterministic dynamical system as the physicists maybe like, but they instead start with something that's usually considered as a subsystem of such a deterministic universe. And the system should obey this kind of large of an equation. And then the FEP kind of tells us what. How to like conditions under which there's an agent inside such a thing. So, I'm not 100% sure how this relates to this. The formulation I gave here where you just get a trajectory and the system, and it tells you where the agents are. Yeah, some subtleties to this but but roughly, I think the free energy principle is definitely a step towards such a theory. And this part of the free energy principle is what I call the free energy principle identification. The FEP agent identification method. And it is, if you're familiar with literature. This is where the Markov blanket placed a fundamental role. The so-called Markov blanket I should maybe say. Okay. Martin, can I ask, what do we gain or lose by going to the Langevin equation? What is what is gained or lost in that maneuver relative to the more general phrasing you had on the previous slide? Yeah, good question. So I, well, I don't really know. I, all I can say is that, yeah. I mean, one thing you definitely gain is that if you I mean this thing, it's kind of hopeless. If you would really, if you, if we would need the particle physics equation, then we would probably never be able to identify agents because the particle, I mean, it's just too complicated and complex to say much about it. So here, this gives us kind of, we can kind of say, okay, I can ignore a lot of stuff. And all I have to find is some kind of part of the system that obeys some kind of approximate equation basically. And this can be enough. So this is, I guess this is one, one advantage. There's one other thing. I'm not sure. I mean, there's one other thing I sometimes think where this is might be useful for and that's. So what we, what we kind of want is that this agent theory, I mean, if you think about it, basically a billion years, like a few at some point, there probably weren't any agents inside the universe. So, so at that point, so so agents kind of like a full blown theory of agents, they should, it should allow trajectories with within which agents appear and basically disappear, or at least they have to be able to appear. Because that's kind of what we think the universe happened in the universe. And, but this is not this kind of appearance and disappearances actually not so easy to do. Especially if you would just kind of naively split up this system, the whole system into two parts and say is one part the agent and one the environment, because then you split it basically for all time. And this is kind of weird. Because how, because you, you would want that the split only works for a short time or, or once they actually is an agent. So sometimes I think, okay, maybe, maybe these, this approximate equation only works for a short time. And that's an advantage of this also that there might be short time frames where this equation works and where they, and others where it doesn't work and then we could get, we could think about whether this means the agent appears or not. But actually, I'm not so sure what these did about the details. So, yeah, that's all I can really say this. I mean, I think, yeah. But it's a very good question, I think. Thanks. Okay. Any other questions because it would be a reasonable point. Not yet, but anyone is welcome to do so. Thank you though. Okay. So how does the FEP construct agents? I'm not going to talk about this a lot, but just maybe I'm only going to maybe mention the things that are relevant later. And this is basically just the discrete time free energy minimizing agents. So in the free energy, in the FEP literature, especially the early ones, you also have continuous time free energy minimizing agents. But yeah, I'm going to mostly ignore them. Yeah, they're not really, there's nothing in particular I want to say about them. So in this discrete time free energy, in this discrete time FEP agents, the agent state mu consists of hyperparameters. Basically, hyperparameters for the conjugate priors of the models that this agent is constructed to use, basically. And these hyperparameters, they encode the beliefs over both the environment state, basically a model of the environment state and a model of the dynamics. Sometimes there's no model of the environment dynamics, but sometimes. So that's kind of important. I mean, they basically, yeah. So the actual state consists of these hyperparameters. So what happens, what happens with this is that in response to some observation and past action, the agent updates the beliefs, and actually that just means that it updates these hyperparameters. And this is done via variation of base. And then the agent basically it also has a belief over actions and this also get updated and this is done in such a way that it minimizes the expected free energy, which I think I've mentioned this before maximizes information gain and kind of minimizes the divergence from these beliefs. Maybe somewhat relevant is that if the model class that the agent maintains contains the true contains basically the true the truth, the true model then one exact model of the environment and so on then the belief update this variation of belief update actually is the same as the exact exact patient. Okay, so that's already all I say about the FEP agent construction. And now I'll talk a bit about how the FEP agent identification works. The main relevant literature on this agent identification for papers as far as I can see. And the first one is life as we know it. And then there's a monograph, quite a long paper, 380 pages something or 80 pages, lots of pages. And that's called a free energy principle for particular physics. And there's kind of an extract from that, but with small differences, and it's called Markov Bank it's information geometry and stochastic thermodynamics by Thomas Parr. And, you know, and also first and then there's also a paper, which is written in response to our criticism. That's called some interesting observations on the free energy principle. So question. Nope. Sorry. Okay, so. Okay, so this, this last paper was in response to our criticism. So these, I think these contain basically what the these contain all the things relevant to this agent identification problem. And basically they're the main papers concerning the Markov Bank. There are also some other ones, but I think actually these ones are the essential ones. Okay. Yeah, our the paper we wrote, it's mainly concerned with life as we know it because the second none of the other papers existed when we started writing it. This one, the energy principle for particular physics came out while we were writing it. But yeah, as I said, I will try, I will try to give my perspective that also includes these newer constructions or approaches or papers. Okay. Yeah. So what is the rough idea of this? I can't go through this in detail, mainly or among other reasons for this is that I don't really know. I don't really understand how it works in detail, like I am still not. Still haven't gotten through the whole construction. Because I get usually I get just stuck before I'm through. Okay. The rough idea is so we have our universe. And like I said, we find this subsystem that obeys the larger equation. Omega is some kind of Markov and white noise. And we also require that this subsystem is such that not only does it obey this equation, but also that this equation has a steady state distribution. P star. Okay. I think this is just going to be the same thing again. We already saw this. Okay. So now what we're going to do is we find the subsystem basically, let's say we have this system and it obeys this equation. I mean, so basically, you mentioned this environment with this animal. And now we further split up this system. Oh, I thought there was going to be something else. Sorry. So we further split up this system. Basically, we split it up into here. You basically, I hope you can kind of see that there's a central nervous system of this animal. And we kind of split split up this whole system into relevant into parts that are aligned with agency in some way. Let's go with this. Okay. Okay. Okay. That's what we're going to do afterwards. In one step. But first, it's a little bit confusing, but there is one nice fact about, so this should have been before these images. There's one nice fact about these kind of equations. Sorry, let me just go back here about these kind of equations if they have a steady state distribution. And only if they have a steady state distribution, if they don't, then you can do the following. So in that case, if they have a steady state distribution, then you can write f of x as this kind of gradient descent or ascent, whichever you formulated of the log probability of the log of the stationary distribution. And yeah, so then you also get these two matrices here in front. And are as anti symmetric and gamma symmetric. And also, yeah, you can write this here, the log probability, the log state probability. You can write it or turn it into this negative log probability. And that's the definition of this. I, you've maybe seen in the papers. The surprise. Okay. Yeah. Yeah, like this is not really trivial. This when exactly you can write this, it's not really trivial. Especially when this matrix and our matrix don't depend on the position X, which is something that is usually assumed in the FEP literature. Yeah, I mean, if, if the force is linear, or if F. The vector field, or sometimes also code flow is linear, then it works. And this also means that around fixed points of a nonlinear vector field, it works. Okay. Can I ask a question on that last slide, Martin? Yeah. Maybe giving an example of a kind of system that people apply it to what is X here and then what does it mean that are is anti symmetric and that gamma is symmetric. Okay. So I actually don't know. I don't think anybody applies this to anything. Let me think at least. Yeah, I mean, like that's actually one of the criticisms I have. But we can definitely apply it to some systems and maybe I'm going to look somewhere deep down there. Oh, there it was. And all these pictures there slow. Just get trapped in those. I saw it. Okay. Yeah, here. Okay. So here's a 2D example of such an equation. And with the gamma and the R. So we say this force, let's say this forces, or this vector field is linear. So it's just a matrix times the vector. So then exactly then we have this matrix times the state and we get this equation. And we can find gamma and R. We can compute them. There's a way in the literature. You can also see it. And it's also mentioned in our paper how to compute those. And you can. And we can compute the stationary distribution. It's some kind of Gaussian in this case. And it looks like this. And you see, here's the on the left, you see this vector field. And on the right, you see the steady state distribution here. It's called a garlic density, but that's the same thing. And how do these split ups look. So if you look at the component. Gamma times the gradient, you get this vector field. And if you look at the component, oh, this should be R. Sometimes this is called Q, but that's the same thing. Just in case you're wondering. If you look at the component R, then it has this. It's basically the circular component of this vector field. It's hard to see that this vector field here is has this circular component, but if you see roughly here, you see that they actually aren't. So this one is actually straight always. Cool. So the left one is that the hill climbing aspect and the right side is the, is it what's sometimes called solenoidal or the isocontours. Yes. So exactly. And so there is one interesting thing about this. And that's that this is the, so whether this matrix R is everywhere zero or not decides whether the steady state is a non-equilibrium steady state or an equilibrium steady state. And this is due to the, I mean, so the standard definition of non-equilibrium steady state and equilibrium or distinction between non-equilibrium steady state and equilibrium state is that in the equilibrium steady state, you have detailed balance and in the non-equilibrium steady state, you don't. And if you have the solenoidal flow, then you don't have, you know, that's exactly when you don't have. It basically destroys the detailed balance. Yeah, detailed. Yeah. So I mean, detailed balance means that basically if you, if you make a small square here, then the flow or two, let's take two neighboring squares and then if it had detailed balance, then the flow from one square to the other would always be the same as the square from the other to the first, from the second to the first. But since here this flow always goes in, in this horizontal direction, not horizontal in this solenoidal direction, what goes in from the one side is never compensated in the other direction. So then you can say, hey, but on the left side, it also does look like it's compensated and that's true. But it is actually compensated by the noise because this is only, this is only the vector field and without the noise factor. So this is only f of x without the noise factor. And the noise factor in this, because it's Gaussian, the noise factor is also, it has no curl. So this is why you, so the noise can compensate this, this component, but it cannot compensate this component. All right. Maybe that was a bit, maybe I shouldn't have taken that much time to explain this, but it's kind of nice. Yeah, we're very, yeah. Okay. Yeah. So this is what we get this nice decomposition and it's kind of needed. Oh, yeah. We only get it if we have this steady state and its place reasonably important in the literature. Okay. So this other thing that we do with our subsystem is we split it up in some way. And this is kind of what the Markov, the so-called Markov blanket does. So the, but here I'm just going to, I'm not going to talk about what the Markov blanket is yet. I'm just going to say what it's so about the work that it's supposed to do. So it will give us this split of the state into four of the state coordinates into four parts, basically a partition of the coordinates. And we will then call Eta the external states as the sensory states, A the active and action states, and new the internal states. Sorry, someone else just joined. Was that a question? Oh, no, no. Someone else just joined in. Not actually sure, but say la vie and I'll add a little password just so that no one else joins in. Yep, go for it. Okay. So, so then what the Markov blanket also does, basically it does all kinds of, it does a lot of work. Let's put it this way. Because all you need is that you have the subsystem and you have the steady state at the Markov blanket. And then what you get is that the internal active and sensory states in this system appear to. And this is important. Actively resist the dispersion of its Markov blanket. Okay. There's this word Markov blanket again. So here you should read boundary or basically you can also read individuality. That's what this means in this case. By engaging in approximate vision inference. So, so these, this statement basically has all these features that we kind of want from an agent identification theory. Because it tells us that if we have this under these conditions and by naming these variables in this way, then we see that these. I mean, it basically it, it makes sense to name these variables in this way because they. We have some kind of goal. The goal is to resist dispersion of the Markov blanket. So basically it's the goal is kind of persistence. And the agent acts to pursue this goal because it actively resists this dispersion. Yeah, and it is individuated because we have this. We know which states belong to it and which states basically don't belong to it. If this distinction between external and internal. Yeah. And then additionally, which is also kind of nice that this whole, this agent is also doing some kind of approximate vision inference, which is a bonus, I guess. So this is why I, I say that the FEP basically is contains a theory of agent identification. And yeah, and this is, yeah, and this is also the, I mean, this is basically the part that interests me the most about the free energy principle. It's, yeah, okay. It's kind of maybe a unique selling point compared to all other kinds of to, to most other approaches to constructing artificial agents. And it's also, I mean, there are not many, there are not many agent identification theories that I know of. And I actually hope that I know all of them. But yeah, actually, yeah, it's quite a, these aren't common. Okay. Yeah, I said what this, I said this. Okay. Yeah. So in summary, we just make three assumptions. We say some subsystem has large amount of dynamics. It has a steady state distribution and it has a mark of frankly, and we're done. We have identified an edge. Yeah. Nice. Sounds good. Okay. So let me, is there any question up to now? There's a few more general ones, but feel free to continue and we'll get them, I think at the end, because this next section will probably be very helpful. Okay. Yeah. So now let's talk about what the relation between this FEP agent construction and the FEP agent identification parts of the FEP literature are according to the FEP literature. So the, these, I kind of said this already, but the thing that makes FEP or active inference, those are actually the same thing as far as I know. Agent special is that are two things or at least two things. I mean, there might be more, but I think these are the main ones or the most important ones at least. FEP agents that constructed the constructed FEP agents that I talked about before. They can be seen as kind of a plausible or used. They are used kind of as a plausible model in neuroscience. And this is related to predictive coding and patient brain. But I actually am not qualified to talk about this because I don't know enough about neuroscience. But the other thing that makes them special is one of the main points of this talk is that they are claimed to be based on first principles. And when the statement that they're based on first principles actually means refers exactly to this, this FEP agent identification method. So the first principles are basically what I just talked about that the Najma equation and the steady state and the Markov blanket gives you this agent. So let's in order to make sure we're not putting words into the mouth of the literature, let's read this quote once. That's from the also sophisticated inference paper that just came out. The key distinction is that base adaptive reinforcement learning considers arbitrary reward functions while sophisticated active inference optimizes an expected free energy that can be motivated from first principles. Okay, actually here it doesn't say based, but actually in another, like you can look at this paper in another place, it says based on first principles. And also this is not only mentioned for the sophisticated active inference agents. You would see similar statements in the literature in other places. So this is, I mean, this is kind of in some way it's super cool if it's true, I think, but I'm not so sure it's true. Okay, so some notes. Yeah. Okay, so this is about whether actually this, how important is this, how important is it for the constructed agents to be based on these first principles? So in this also in the sophisticated inference paper, the authors say that the agents expected free energy, expected free energy, which is the thing they minimize, can be motivated from first principles. But they also seem to say that the expected free energy can also be, I mean, yeah, actually, I mean, they say in one place they say that it can also be motivated via information gain. Don't, and they, or I mean, there's one part that this refers to. So as far as I understand it, they are, there's a, they are two ways to motivate this expected free energy minimization. And one is via these first principles and one is via some other derivation that involves information gain. Now I wouldn't say I understand these derivations, this derivation completely, but I do know that there's a paper by Cohen and Hutter. And they show that if you, that there's a, that, that basically some combination of information gain and reward maximization is, you can prove that it's optimal in some sense. So since this expected free energy is kind of a combination of those two things, I kind of think, let's put it this way, I'm not surprised that it has, it shows good performance or something like this, because maybe it's actually so closely related to the method by Cohen that it just works. So all I'm saying, I guess what I want to say here is that in case the motivation via first principles fails, the FEP construction of agents may still be of interest and might still work. And this also means that all of these, all publications, I think there are some, but yeah, I think there may be quite a few, all these publications that, that are basically based on this expected free energy minimizing agents, they, they need not be without merit just because the first principles motivation fails. So, or in other words, they can still, they can still make as much sense basically, even if this free energy principle agent identification is wrong. If I could ask a question here, Martin, what, what kind of first principles are you most interested in? And then what kind of a theory of agents or agency would be satisfying given your interest in first principles? Yeah, I mean, I, I, I think so. So if you, I'm interested in the first principles that work, I don't know what the first principles are, but like maybe one, one first principle is that you can start from physics. Like that's, like you can, you, you start with some, I don't know where a whole five to go up. Yeah. You start with a dynamical system and you then derive using only notions that are well defined for dynamical systems or within such dynamical systems. To show that this thing, this system contains agents or not. Or yeah. So I guess the first principles for the first principles here are that, okay, all we have to believe is that there are such subsystems that obey the lingerie equation. And I guess since the lingerie equation has been quite popular in physics, it has, this has, this, this, this has some strong support. And maybe that's sufficient to be your first principle. Also having steady state is fine. It's well defined. There's no, I mean, that's a mathematically well defined thing. We can in principle check this. So yeah, then the other thing that's used in the literature is the Markov blanket. And I think that's also, yeah, it's a, it could be a well defined thing. I think it's a bit of a, like what it means in the FEP literature is kind of not, it's not really clear and it changes and maybe has to change again. But at least the original. So if it depends a bit, yeah, I guess I have to, I would have to show a bit more about the Markov blanket. But yeah, I would say, I, yeah, first principle for me means probably just that you have to be able to define it for dynamical system. It has to be like all the notions you use, you can only use notions that are defined for dynamical systems. Totally makes sense. It's like, if you're going to play a game, you're going to play a game, you're going to play a game, you're going to play a game. Totally makes sense. It's like, if you're going to play the game of dynamical systems analysis, you can only use those rules. Exactly. You can only, yeah, exactly. You have to, you have to construct everything from there. You don't get anything else. That's it. You get the dynamical system. That's it. Yeah. Okay. First principles. Yep. That's where I was now. Yep. I say this need not be there. Yep. Okay. So now, oh, okay. So that's a big part of the talk. That's kind of, that was kind of my personal, my current personal view on what's, yeah, and how I, yeah, how I look at the FEP literature and what part, yeah, how I split it up, like I said. And, yeah. So now I would go and talk about basically the issues I see with this FEP agent identification method. But if there's any questions now might be also a good point. That's it for now. Definitely after this we'll see if anyone has any questions, but feel free to type them in the live chat if anyone has it. Okay. Okay. So the first issue I have is that this Markov blanket assumption is known to be insufficient. By now, it's known to be insufficient. Maybe in the, maybe at the, for the very first paper, it wasn't known to be insufficient. Oh, actually, yeah, for the first two papers, maybe. No, yeah, I don't know. Yeah, I can't really say because it wasn't me. But yeah, I know it's, it's, it's fair to say that the first two papers, they might not have known this. Okay. Yeah, so, so, so I'm going to kind of go through the Markov blanket evolution a little bit, not, not, not in any detail, but the Markov blanket evolution in the FEP literature. Maybe I should say, like originally, I think I don't have this here. I mentioned it down there. Okay. So originally the, the term Markov blanket comes from Judea Pearl from this 1988 book. Yeah. So it's a, it's a technical term that had a definition before it was used by the FEP literature. Okay. So, and in this, in this original paper, the life as we know it, there is actually no definition of like, there's no, it's nowhere clear. It's, it's not clearly said what is meant. What kind of Markov blanket is meant. I mean, there's some kind of graph, but Markov blankets come from directed acyclic grass and the graph is cyclic. It's actually not clear what is meant. I mean, there is, but it's clearly stated that the Markov blanket should imply this equation that the vector field when we split it up into the four coordinates also actually. So it kind of implies also that there are these four, we can split up the fact, the coordinate into four sets of course, all the coordinates into four sets of coordinates. And the vector field then has can be has a certain shape form, namely that the internal and sensory states do not. The vector field components of the internal and sense sensory states do not. Directly depend on the internal states and the vector field components of the action and action active and internal states do not depend directly on the external states. Maybe I said something wrong, but yeah. And we call this the. So we, we, we call this the vector field condition or. Yeah, I mean, I call this the vector field condition because it's a conditional on the vector field. And in our paper, this is this conditional. Okay, so this is kind of. I think in life as we know it, this is kind of, this is the only real place where the Markov blanket plays a role. And by the way, this is not a Markov blanket. I mean, not technically, this isn't a Markov blanket. I mean, this is a Perlian Markov blankets condition. Then in. At least in, in Thomas Pars paper in 2020, the Markov blanket is said to imply this condition. Independence of the. External states. Given the blanket states which are the sensory and the action states. So we get this factorization of the stationary distribution or steady-state distribution. And that's also considered a condition of the Markov blanket. This thing is also implicit in the, this condition is also implicit in the. Free energy principle for particular physics. It's not explicitly referred to as the Markov blanket. So it's a bit weird, but this is actually a Markov blanket in the sense of to the approach. So this condition is a true Markov Markov blanket condition. And it has come to be called. Not only by me. The time synchronous synchronous Markov blanket condition. And in our paper, this is conditioned. Those two conditions, this one and condition one, they are independent of each other. Like you, there are systems that obey this one. And not this one and system that systems that obey this one. And not this one. And those systems are not complicated. They are actually linear systems. So there are these on-stand or impact. Processes where you can easily compute this. And it's a gosh. So you can look this up in the, in the paper. And I think, I think one of the, I mean, like this is why it's speculation, but. I think that. The, the authors. Our car. Because he's the only author. Kind of thought that this equal, that this condition implies this condition. I think it seems to me. There are various reasons, but it's kind of like, I'm not going to go into it. But yeah, this seems to me the case. And maybe you could think, I mean, I can, like, you could think this. I mean, it's not completely out of the question. But yeah. So then we wrote, we wrote this paper. After we wrote this paper in 2019. And complained about some. Some. Thing in the. Some things in the. Free energy. For a particular physics. Paper. Namely this thing called the conditional independence corollary. Which is supposed to follow from the. Back to field condition, I think. Is the, I mean, I think that's the claim. But we showed that it doesn't. And so then the Markov blanket. Or then. An additional condition was presented. In this paper called. Some interesting observations. And this is the response to our paper. And this condition is that the. Solid needle. Component matrix. R or Q. As this shape. Okay. That's why I mean, I'm going to call this the solar needle condition. If I ever need it again in this talk, but okay. Yeah. Since this came out after, I mean, this was in response to our paper. So we don't really talk about this. Okay, so. This is just some additional condition. And then I think. So. Yeah. So you can kind of see all of these, like these conditions are all used in. To derive this statement here that. That. This statement here. We can identify the. FEP agent identification method, basically. That you get to. Identify. That we actually get, we can identify an agent that. Actively. Acts to that acts to. Persist basically with a goal of. Okay, so we have this. This condition as well. And then in. In part in the. Was past paper. And also in the monograph. If we look closely, there is actually an additional assumption made. In the. For particular physics. It's claimed to be a consequence. Of the other conditions. But that's not true. We can show this. It's kind of easy. And in, in the power paper, it's, it's. Actually said that it's an assumption. So. I've known that this is an additional assumption. And this assumption is that there's some function sigma. And this function sigma relates to variables. And these two variables are. The most. Likely. External state. Given the. Blanket states as an A. And the most likely internal state. Given the blanket state as an A. According to the stationary distribution. The steady state distribution. And the, and so Sigma. Is. The assumption is that, that, that, that there exists. A function sigma that maps one. That maps. The most likely. Intern state to the most likely external state. And. That, that this function doesn't need to exist. You can see. For example, I mean, it's, it's not in our paper. We should have probably put it in our paper, but we didn't. Account example. Is. You can use basically. We, we show an appendix D. And then we show one of those linear. Linear systems again. And for this. The bold new is just. One half. A. And the both each is one half S. And since S. T is not a function of A of T. And basically. So they are kind of. So every combination of S and T can occur. So there is no functional relation between two. And that means there also cannot, cannot be any function. Sigma. So, so this means we kind of have to additionally assume. Sigma and also. Yeah. Yeah. And the thing is, in my opinion, this Sigma might actually be. A much stronger. I think it's probably the most. But since I don't really understand the whole picture, I can't really self-check for sure. But. So yeah. Just, just. On that. Sigma is sort of the proposed function. That's literally linking. So. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. The proposed function that's literally linking. Internal and external states. So when we're talking about agents and the way that their internal generative model is being linked to. Generative processes in the outside world. That's like the key link. That's what we're all about. So I totally. I totally see where you're coming from. Exactly. That's a good. Yeah. That's a good way to say it. Exactly. I mean. Yeah. I think I will. I would say this later again, but. These three conditions. They are all trivially satisfied by four independent stochastic processes. So. Like. It seems. Kind of. Unrealistic. That's what it is. That any of these conditions by themselves or even their combination. Would be sufficient to. Give us. Give us this kind of. Active. Active resisting to anything. It's like. Modeling of the environment or. Asian inference about the environment because. For independent stochastic processes. Don't seem to. I mean, like it would be hard to argue that. Any of those is doing anything. Is modeling any of the other ones. So, yeah, exactly. This is. But if you have such a relation. Then, yeah, then, yeah, there might be something going on. Okay. So in summary, we have these four conditions, the vector field condition, the TSMBC. The sunatal condition and the sigma function existence. And I would say maybe. Call those the FEP blanket together. Okay. So I would say this now. Apart from the signal function, all the conditions are certified by four independent linear stochastic processes. Yeah. So. Probably signals the most. Significant subject. And yeah, I mean, it's maybe a petty point, but. I kind of think that calling the combination of these conditions mark of blanket is. At least. I mean, by. By now it is definitely misleading. I mean, I think. The use of this word mark of blanket was misleading. Before, but I think. I should probably be dropped. But anyway, I'm going to call it FEP blanket. Oh, sorry. This is. So, and one more thing. I'm not. I cannot say that these conditions, these four conditions together together are sufficient for. The whole. FEP agent identification argument. Just because I don't really, I haven't really. Got. Got through the whole thing. But we know. That they are. Necessary. I mean they are used in these. Okay. So that's the first. That was the first issue. How am I doing in time? It's like already super long. You can go for as long as you want and we can. Go as long as you want already. At one hour. Wow. You can go as long as you want today, Martin. And then we can always schedule another time to speak. But this is really, really helpful. And I'm sure that a lot of people are going to digest it and have a lot to talk to you about. Okay, okay. Okay. All right. Next one. This point is not in our paper. I guess it has been in the back of our heads. But I mean, yeah, I think it's, it's really also, it's become much more. I've become much more convinced about it. And yeah, so I'm going to just mention it. And I think the, it would, yeah, I think it's a kind of challenge to the. Community, maybe. And I kind of, if I'm, yeah. I think they are kind of aware of this panel also. So the thing is that there is no example for which. All of the FEP blanket conditions, the four, I just mentioned the four conditions I mentioned. Have been proven to hold. So we don't know any. We don't even know if there's any system. That the FEP identification method applies to. Or that this statement that. Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, Uh, if there is actually something doing Bayesian inference, and in what way it's doing it, and that it's actively resisting dispersion. So this is kind of, it's, it's not, yeah, I mean, like, it would be kind of cool to, like, so in some way this says the whole Markov blanket story, or, yeah, the whole, as I call it, FEP, agent identification story might not, might be vacuous. Like this is what this says. And I think it's kind of a serious point, and it should be addressed, I think. Okay, so maybe some notes on this. So there's this primordial soup example in the original paper, and it's used, I mean, in life as we know it, and it's also used in the monograph. But there's no proof that this thing has a steady state, or that this steady state obeys the TSMBC, the time synchronous Markov blanket. And also, of course, there's no proof that it has a sigma function, because in life, as we know, it wasn't even any sigma function. Also, this, like you could say, maybe the constructed FEP agents are systems that obey all these conditions. But I mean, maybe there are even some of them. But this connection is never made explicitly. Like, it's always just, there's this claim that they are based on this, on these first principles and all of this. But it's never shown, for example, that the agent environment combination, which forms this Markov process, that it has a steady state distribution. And it's actually very unlikely that in all of them, it has a steady state distribution. And actually, you can kind of like, I will argue that it's actually not true, definitely not for all of them. And also, even if this agent environment pair would have a steady state distribution, we would still want to see that it obeys the TSMBC. And also that is kind of questionable. So I should mention that this is like the second point that checking that the TSMBC holds is an idea that's by Nathaniel Virgo. And there's some arguments about it, but I won't go further into it. So furthermore, it is never checked, like we never look at the Sigma function, which appears to play a significant role. But we never look at the Sigma function in these constructed FEP agents. What is kind of what I'm pretty sure that it says is satisfied in a reasonable way is the vector field condition. Okay. Yeah. And maybe this, I don't really know why I say this here, but yeah, like the, okay. So you could, yeah, okay. So you could maybe, so since me and Felix, we use the Onsen-Ulenberg processes to show various things about the free energy principle. And we, we know that it satisfies quite a few of, like it satisfies three of these assumptions. So it satisfies the vector field condition. We can make it so that it satisfies the TSMBC and the Solonary-Riedel condition. But there is no such system. Like, so basically I'm, like, if you want to, there's no such system that has a Sigma function. So if you wanted to search for an Onsen-Ulenberg process that obeys, that is basically an existence proof, then I can tell you it's impossible. So this proof is kind of unpublished, but, and it's mostly due to Felix. But yeah, maybe, maybe I'm gonna, maybe you're gonna put it on archive at some point. It's really not complicated. Okay, so that's this point. Now I want to make also another point that is not in our paper. And then we're going to slowly get to the end, because the last point isn't substantial. Okay, so yeah, what I want to say is that there are agents in the FEP literature for which the, the, the, the, the, the, the, the, the, the, the, the, the literature for which the, the system that they are contained in. So if we form the environment, if we look at the environment at the agent in combination as a Markov process, what I talked about in the very beginning, then, and, and the, the agent is this FEP agent, then for some of them, it's straightforward to show that they, that this combination, this combine, this resulting Markov process does not have a steady state. And the first thing to note is that if you have some kind of some monotonously increasing counter, it can never say if your state contains a monotonously increasing counter, it means you can never return to a previous state. And this means you cannot have a steady state, because the states are not recurrent. And that means there's no steady state. So as I mentioned before, the internal state, the internal agent state of FEP agents are hyper parameters and are updated using Bayesian inference, variation of Bayesian inference, but that also includes exact Bayesian inference. And so these hyper parameters of X exact Bayesian inference, they accumulate evidence, which means they count occurrences of events. And they only count up. And they count every ad for every event they call something. So they are counters that always that are monotonously increasing. And for example, in the sophisticated inference paper, there's, if you look at equation 4.5, there's one, there's an explicit equation for the updating one of the parameters, which counts the occurrences of transitions from expected environment states to observations. And no such counter can be contained within any Markov chain that has a steady state. So this means that this agent, this particular agent in the sophisticated inference paper is not within such a system that has a steady state. So it's not within a system that a subsystem, it cannot exist within a subsystem that obeys the conditions of the FEP agent identification. Okay, so this, so be aware that this statement also is not published. I kind of want to put it out there because people are in if I just talk about that's already published. And so, but I mean, there's some caution advisable, I think, but if you want to look at this at a very simple example of this, I published a paper last year on this. And so I'm pretty sure, it seems pretty straightforward this argument. And then one more thing that, so in this paper sophisticated inference also, there is some argument about steady state in the appendix A2, but that steady state just kind of, just it ignores the internal state of the agent. It only looks at the other variables. So I think, and I think that's actually necessary, like there's just no steady state if you include the internal state of the agent. So maybe be aware of this. And yeah, the two consequences of this are that FEP agents are not identified by the FEP agent identification method. And this also means that the FEP agents cannot be motivated by this FEP agent identification or by first principles, because they actually violate these first principles, or in this case, at least this particular age. Okay. I think that was, yeah, that was this point. The last point is kind of being a bit nitpicky, but I think it's also kind of important for some people. And that is that this identification method is changing in the literature. And you shouldn't think, for example, that the original life, as we know it, paper contains a method that's very similar to the neurons. I mean, these constructions of the most likely internal and external states, they don't appear, there's no sigma function. But in the later versions, these things, these constructions play essential roles. So these two, they are two completely different methods. And, okay, and, and yeah, one of these things we did in our papers, we showed that this original method in life, as we know it is kind of far, but wow, where did I go? This was, and okay, don't worry, we're not going to look at all these slides. Okay, here we are. Okay, so we kind of looked at this and we think it's wrong, at least the way it's stated. And yeah, okay, so, so be aware that if you look at this method, like don't maybe, maybe just ignore the life as we know it paper. That's maybe what I want to say. And also, like, you can't tell from the literature that this method is very different. I mean, it's kind of, they never say why they changed it. And it's kind of still represent like still mentioned as the basis of these things. So be aware, it's kind of, maybe it's a waste of time reading this paper. Okay, that's it. In summary, what I said is that we can separate the literature into agent identification, FEP agent identification and FEP agent construction, and that the FEP agent identification method requires significantly more assumptions than just whatever the Markov blanket was in the beginning, that this agent identification method misses an example that would prove that it that such systems actually exist, such systems with agents, that systems with such agents actually exist. And it requires steady state that at least some of the FEP agents definitely don't satisfy. The method has changed from its original. But also note, it could still be correct. I don't know, like maybe there are enough assumptions you can make, and maybe they are still systems that contain such agents. It could all still work out, but I haven't made it through all of these derivations. So there may also be, like I said, there may be even more assumptions necessary. Okay. At the last point, that FEP agent construction, the FEP agent construction is probably unaffected by any of these problems of the FEP agent identification method, or the Markov blanket stuff if you want. Okay. And it can be, yeah, okay, and this method can be probably independently motivated by some other arguments that are not first principles, if you like. Okay. That's it. I thank a lot of people for talking to me about this stuff. Yeah, and that's it. Thank you very much for your attention, if you made it all the way. Especially if you made it all the way. Thank you, Martin. Maybe you can unshare, and I'll just ask a few of the questions from the chat in closing. Okay. Hold on. Is this the, why is the unshare? Is it the same as? Maybe go back to the Jitsi and just unshare. Jitsi. Yeah, cool. Great. So this. Cool. Okay, perfect. So great. Thanks again. That was really awesome, and I hope people paid attention. So the first question is from Peter, and Peter asked, the distinction between agent identification and agent construction is very nice. If I wanted to use that distinction in my own work, should I cite your paper, or is this distinction made elsewhere too? I'm wondering if I mentioned this anywhere. I don't know, actually. I don't know. I mean, it's in some way, it's just, yeah, it's kind of like a, yeah, it's kind of like an observation that makes a lot of sense, and it's been, yeah, I don't know. If I think of something, maybe, I don't know, can I see you on Twitter? Maybe I'm going to post it on Twitter, follow me on Twitter. Sounds good. But yeah, I'm not sure. And I think it's kind of an obvious thing, that I'm not the only one who has, I'm definitely not the only one who has observed this. Yeah, probably there's somebody in the literature who did it in 1950. But yeah, I don't know. Like, I'm not going to be mad at you if you don't cite me. Let's put this back. Thanks for the response. Another question from Peter is, how much should the agent identification and agent construction depend on each other? In other words, how big is the problem if they don't? And how important is it that it comes from first principles? Anyway, can I say the first sentence again? I'm sorry. How much should the agent identification and agent construction depend on each other? Okay. So are those like two separate lines or one integrated whole? Yeah, good question. I mean, I guess, I mean, I guess ideally there would be some kind of relation. But on the other hand, so we want to construct agents, like so so I in my opinion, bacteria are agents. But when we do agent construction, we don't really care. Like we don't have to follow the principles of bacteria necessarily. If we don't like either we have, I mean, we might be, we might have to, because that's just pro physics works or due to the constraints of physics. But if we don't have to, then nothing should stop us just constructing any kind of thing that we want. I mean, like, like if we yeah, so I guess they are not necessarily dependent. I mean, yeah, at least I can, I can see this. So on the other hand, I guess it would be, it would also be nice if you have some kind of, yeah, if you still have some agent identification method, which comprises all possible agents, like all agents that you construct should be these examples of your, like should be detectable, identifiable by your agent identification method. So okay, now I'm going to take back what I said before they should be independent. I hope that I hope this helped a little bit. Yep. And I'll just give one. I think there's probably more to say, but that I can't at the moment I can't really think of it. Yeah. So here's one final question and is a question I had, but I'll use the phrasing of Peter here. And so this will be our closing question, hopefully the beginning of something new in the beginning of moving forward. Peter wrote, I suppose the mathematics in free energy principle and active inference could be taken as a work in progress. Are there suggestions for how to solve or work with the issues that you're raising? Just any thoughts on this could be nice. So where do we go as learners and doers and practitioners and communicators in active inference? So, so yeah, like I said, I mean, on the one hand, you can, if you, if you construct agents, you can just ignore all of the stuff I said, because in my opinion, you're not relying on this agent identification method anyway. So, so there's no problem. Like, of course, you could maybe wonder what this base optimal that is sometimes mentioned in the literature of the FEP, what this really means and where it comes from. Like, I think that's, I find it, I always think it's a little bit vague what this really means. And it's actually not such a trivial question what base optimize, but what optimize. But yeah, mostly you can ignore this FEP identification. But if you are interested in the FEP identification, then yeah, I mean, I guess the thing is, look at either, I mean, try to understand where it's actually going. I mean, like, I still haven't really understood these later parts of the monograph, where everything is supposed to come together, because I always get stuck before in some equations that I don't understand. And then I think they're wrong. I'm not sure they're wrong yet. As I would have mentioned them. But there's somewhere I don't, I suspect they are wrong. But yeah, I'm still not sure. And but yeah, I guess that's the, like, I don't really see, like, if you want to save something of this method, then look at this. But another way is maybe to think about, think about this for yourself. Think about what it, what are the conditions you think under which something is doing Bayesian inference or something like this? And ignore all the math by the, that's already in the literature. And then try, like, once you figured it out for yourself, maybe try to come back to the literature and compare it or something. Because I sometimes think people really like too much on what this, what the literature says. And I'm not really sure that actually what's in the literature is like, it doesn't strike me as a very straightforward argument. And I like straightforward arguments. I think straightforward arguments are usually better. Like, maybe there isn't a straightforward argument, but I kind of think I always think there's a straightforward argument. But that's just my opinion. Yeah, it's off. Well, it's a nice point that the literature is a scaffold for our own thinking and development. The literature is not a road that's been paved with gold. And it's just perfect in the past. It's actually always a work in progress and always a scaffold. So thanks so much for sharing this really work in progress, not that it isn't with landmarks that you've published, but it is a work in progress in the bigger sense. So also thanks for staying up late to bring us this information. And we hope that in future live streams, we can unpack this more and hear updates as you continue to progress on this line of research. Okay, thank you very much for having me. Thank you, Martin. Peace. All right. Bye. Peace.