 Welcome everyone. So, so far we have been looking at stochastic control problems of two different kinds. The first in which the state was perfectly observed and then again the stochastic control problems where the state was not fully observed and but we had partial information about the state through some observations. What we have assumed in either case is that we were looking at policies which were a function of the information and the information that we had at any time step was given by the history of observations that we had up until that time and the history of control actions that we have taken up until that time. Even when we were looking at problems where the state was perfectly known we had a notion there of history dependent policies where the information of the entire history was available to us in order to decide the action. Fortunately in that sort of setting what happened was that we realized that Markov policies were optimal. We realized that the entire history does not need to be retained it is sufficient to focus on only the, it is sufficient to just know the current state and so as a result of that we, if you remember I made this statement that Markov optimal policies are actually the same as optimal Markov policies. So in the case of partially observed problems the situation was somewhat different there we did, we did have to keep track of the entire history because previous observations and fresh observations need have no relation, no guaranteed relation in terms of the quality of information they provide us regarding the state of the problem. So one had to basically keep track of all the observations and it did not make sense to discard any part of the history. Of course we achieved a significant simplification in terms of computation because we said that instead of keeping, instead of taking the state to be the entire history or the entire information vector we can take the state to be the belief state where the belief was a probability distribution on the state space given the information. And that the belief state did not grow with time and the belief state had a fixed dimension and so it made our problem a lot easier. However in spite of this the point is that in either problem we have continued to make one key assumption about the information that we have. In the partially observed case we have made the following assumption if you recall we always wrote out, we always wrote ik to be comprising of all the control actions up until that time which is u0 to uk minus 1 and all the observations that we have up until that time. This was our information. Now notice that this would automatically mean that ik minus 1 is a subset of ik. In other words all the information so this is the information available to us at time k minus 1, information available at time k minus 1 and this is the information at time k, information available at time k. Remember I am not talking of observations I am talking of information available at time k, available at time k. So all the information available at time k minus 1 is also available is a subset of the information available at time k. So all that was available to us at time k minus 1 is also available to us at time k. So, as a result of this what we have effectively said is that information is continuous to accumulate over time, information is not being forgotten, we are not forgetting anything that we knew in the past. So, information is not getting forgotten, is not forgotten, so all the information that we had in the past continues to be available to us in the future. Now, as a result of this we remember we use this particular assumption in a very critical way and I showed that demonstration for you when we did when we were looking at linear quadratic problems. In the linear quadratic problem this assumption was actually used when we when we transform one expect an expectation of one kind over into an expectation of the other. We remember we transformed an expectation inside which was over i with respect to i k conditional expectation with respect to i k and outside we had a conditional expectation with respect to i k minus 1 and we were because of this assumption we were able to eventually write it eventually as an expectation only over i k minus 1. So, this assumption has in fact been used it is not only that this assumption was made for convenience and it has been has not really been exploited it has actually been used in deriving the problem. This sort of assumption about the information is what is called the information structure of a problem. So, here let me write this is an example of what is called the information structure information structure. Now, what is the information structure information structure is simply a description you can say is a description of who knows what each instant in time remember we are talking of information. So, really we must talk of knowledge in some sense right. So, when we say that i k is known to when we are taking the action at time k very effectively saying that that is our knowledge at time k. Now, you can think of the agent the decision maker or the controller or the agent that makes that is taking these actions as some kind of a algorithm or a device or a organism like a human being or a company or a firm or like a shop that is selling shoes or something any of this is alright as a model for the decision maker right I will write as a sort of mental framework for the decision maker. So, effectively you can visualize the decision maker as any such entity and effectively what this assumption i k minus 1 subset of i k is saying about that decision maker is that the decision maker has knows everything at time k everything that he knew at time k minus 1 is known to him at any other future time in time k and in particular any future times k plus 1 k plus 2 etc. So, in so this this here is a particular case of an information structure there is no this is not the only kind of information structure in the world you would you what I will argue now is that there are in fact many many other information structures also that arise and in fact those arise far more commonly than this particular information structure. In fact, if you think deeply about what this information structure actually means you will realize that there is something we are actually making a fairly strong assumption here about what are about the nature of information that we have. So, let us let us so we will what that is what we will do now let us let us we will think about what is it that this information structure actually represents. So, this information structure is basically representing that all that we all the information that we had received at a previous time instant continues to be with us at the present time instant. So, previous time instant is k minus 1 present time instance is suppose k so all the information that was received by us at time k minus 1 and at previous and at previous time since time immorally essentially is available to us at time k. So, effect what this means how is this information going to be available to us well for this information to be available to us this information has to be stored in some sense right somewhere it has to be stored. So, since when so for example when we when we remember what happened to us in our childhood we are effectively saying that there is some information about that which is being stored in our brains right or if the control or if your computer look if you open your computer and open a document that you had saved the previous day that information is there in the computer you are able to open it because the computer saved that document on its heart base at the previous day right. So, this so any when you are talking of information being available across time implicit in all of this is a means of storage that somewhere we are storing information right. So, this information here is therefore being stored at for us so that it can be used at a future time at a future time k. Now we can ask what happens as k increases as time increases the information that is being stored will continue to accumulate you keep having getting longer and longer information vectors like this you get you your information the information that you are storing will become will become more and more and eventually you will come to a stage where the amount of information you are storing is much more the is more than the capacity that you have for storing information because after all the world is finite, but you can talk of time instance that will continue to accumulate more and more information right. So, this here is therefore assuming that you have technically some kind of unlimited storage space right it is assuming one in some sense an unlimited storage space. So, this information structure which is i k minus 1 subset of i k is assuming assumes you can say unlimited storage storage. So, this is this is one way of thinking of of what is what is going on in this. So, as a result once you have finite storage right once you have a limitation on how much can be stored and how in your in your device let us say you are running a very parsimonious operation you do not have much storage space in that case this assumption would be violated there will be some information from the past which will be lost in the future right. So, this is something to bear in mind. So, this assumption will get is valid only when you really theoretically have unlimited storage okay. Another way in which this assumption can be thought of is the following and this is another way of thinking about the problem and which is something that is not I have not yet touched upon in this course. So, one we have always been thinking of k as instance in time or our thinking was that k is our times or decision epochs when decisions are to be taken and it is the same entity that is taking those decisions. So, it is a single controller or a single shop manager or single interviewer who is interviewing secretaries however, but it is one entity that is taking these decisions in time. But one can also think of another setting which is completely equivalent to this which is that there are not one entity but multiple entities. In fact, there is a different entity at each time instant. So, one can think that there is controller 1 at time 1 controller 2 at time 2 controller 3 at time 3 and so on. So, there are actually as many controllers as there are time instance and these are as physical entities different controllers. So, if these are in fact different controllers the information that is available to them is all local to those controllers. So, information ik is available with the controller at time k and that is that controller at time k is not the same as the controller at time k plus 1. So, the information available at time k is present locally with controller at time at the with controller k and this information has to then be somehow transferred or somehow communicated or transported to controller k plus 1 because these are physically two different controllers it means that they are they need to be information has to be sent from controller k to controller k plus 1. So, if I take this second view and which is also a valid view that there are actually not one but multiple controllers then this information structure that ik plus ik minus 1 is subset of ik is effectively assuming that the information of that is available with controller k minus 1 is being transported or communicated or transferred over space to the to another controller which is the controller at the which is controller k. So, if you are talking of so what does this mean? This means that as in if you think of these controllers as different controllers that means physically different entities then this assumption this information structure is essentially assuming that all that is known to one entity is known to the entity that acts next and therefore has to be for that to be true you need for that to be true it has to be communicated to the entity that acts next. So, they have now communication communication from one place to another is will always in will always have a some amount of noise. So, there is always a possibility that know there is always a possibility that whatever information is available at time with one controller is gets to the other controller in a somewhat correct corrupted form one can aim to correct this for these corruptions and I will come to that issue in a moment. But remember that even those corrections are not perfect and in general there will be some amount of noise when it when the information reaches from controller at time k to the controller at time k plus 1. So, when information so if we take the view that these are actually separate controllers then there is an issue of information being communicated and once it is once that if you is once the information is communicated and it is it will get corrupted with noise then the information that is received at the second controllers end is not the same as that which was sent what the controller will receive is what was sent plus whatever noise was has gotten added in the process. So, as a result this assumption here that I k minus 1 is a subset of I k is going to break down because what is what was known at the previous time instant is not anymore available at the next time instant. So, if these are actually separate controllers we are basically assuming that this is a perfectly noiseless means of communication. So, mathematically there are two these are all the two models whether you have one controller acting multiple times or multiple controllers acting at each time at separate time instance mathematically these two way views are actually equivalent. But the physical assumption behind them behind this behind this information structure has to be understood the physical assumption in one case is that you have unlimited storage right. So, if you have one controller. So, I had two particular two views or two ways of viewing stochastic control problems one controller acting multiple times multiple controllers acting controllers acting at separate acting at separate time instance. So, one controller acting multiple times this particular issue we saw what this we saw what the information structure the assumption of I k a subset of I k plus 1 what that information structure meant. So, the information structure would then in here information structure the information structure I k minus 1 subset of I k implies infinite storage or infinite memory whereas in this case information this the information structure implies perfect communication right. So, this is implicit therefore in our problem. So, what we need to what has happened as a result of this is that although we mathematically just wrote out of one particular problem and we made a certain assumption about the information structure it turns out we have not really thought very carefully about how this how this problem actually is physically getting implemented. If this problem involves a single controller then we have to really think of that well our assumption implies that that single controller has infinite memory. If this problem involves multiple controllers acting at separate time instance then our assumption is involves is a tantamount to assuming that we have perfect communication right. In either case you are seeing here a hint that communication either in the form of storage and retrieval or in the form which is communication in space or there is a communication across time which means from one time instance to the next time instance right. This communication this either of these two forms of communication is implicit in the underlying physical manifestation of this particular problem. So, this is something that we have completely overlooked so far as in our development and this is something that we now have to start grappling with as we try to think more deeply about information. structures. So, here this information structure here is has a name it is what is called the classical information structure. So, it has the assumption. So, the classical information structure is basically saying that I k is a subset of I k minus 1 is a subset of I k for all times k. So, so we have so far basically looked at stochastic control problems with classical information structure. So, we have so far studied stochastic control with classical information structure right. So, this is what we have we have been we have been studying alright. So, now lets the question then arises are there non classical information structures and are there interesting stochastic control problems which involve non classical information structure? Well, the the answer is actually evident right here itself. So, any information structure which which is not classical is called non classical any information structure non classical is called non classical right this is the different color it is called non classical. So, the question then arises are there interesting stochastic control problems with non classical information structures. Now, the answer to that is actually there right here on the screen. See remember we have realized that the classical information structure means the information structure where I k is a subset of I k plus 1 this this assumed either of these two cases it assumed either infinite storage or it assumed perfect communication right. So, if any of these two are violated then the information structure is not classical anymore. So, for instance if you did not if you had finite storage then you are compelled you are obligated to lose some amount of information and that that then makes it a non classical information structure. If you have imperfect communication between entities then then what is sent is not what is received that then becomes in becomes a non classical information structure right. So, let us ask now is there any is there anything apart from an academic interest in in in problems stochastic control problems with non classical information structure. So, what I will argue in the next lecture is that there are in fact there is a lot of a lot of practical interest in in such problems. In fact, there are many many interesting problem classes that have not been looked at or not been studied as stochastic control problems but actually are stochastic control problems with non classical information structure. So, this is now coming up as in the next lecture and in the rest of this course.