 Hello, everyone. Welcome to yet another week of our NPTEL on nonlinear and adaptive control. And we can't set them up systems at control. So, we are embarking into the final week of our course on nonlinear and adaptive control. I do really hope that you have learned quite a bit during this course on how to design and analyze robust and adaptive control algorithms that can drive systems such as the drone, the aircraft, the spacecraft, the neural network that you see here in the background. As I have mentioned before, I'm always very, very open to hearing more about the sort of applications that you folks are working on or plan to work on using the tools and techniques that you've learned here. And I'm hoping to hear more about that from all of you. So, what we were doing until last time was to look at a particular kind of adaptive control design, which does not rely on persistent excitation anymore, and only relies on having excitation for a finite amount of time. And this condition was of course called an initial excitation condition. And we of course saw how to do control design. We saw how it was a weaker requirement as opposed to your persistence excitation based design. We saw both the advantages and limitations of this method. We also saw that it's possible to alleviate some of the limitations by slightly modifying the initial excitation design to add a certainty equivalence adaptive control term. So, keeping that in mind, we do that, you know, this sort of design method helps us to, you know, give some improvements, not just in theory. Of course, we've seen a theory that there are significant improvements, especially in the sense that you have these negative terms in your adaptive update law, which was of course not there, you know, earlier. Right. So, we do of course hope that this sort of improvement is also something that you see in the performance of the adaptive control. Right. So, specifically the numerical performance of this adaptive. All right. So, again, please try it out on actual systems. And please do report back, you know, how it looks, you know, how the performance look compared with your traditional certainty equivalence controller. Yeah, and we would like to hear more from you. Now, where we want to go starting today is on discussing, you know, sort of modern ideas in adaptive control. And this, the end is here to make some connections with learning theory, which has become very, very popular. And of course, adaptive control is also an insisting of learning itself. So this is another, you know, case in point that we will try to make that parameter learning is essentially, you know, an ingredient or the key ingredients of what you do in learning, right. Albeit the algorithms that are used may differ. But but eventually the basic theory remains the same. So the so the key connection that we will try to make is with neural networks and so deep learning so multi layer neural networks which is essentially what deep learning involves. Before we do any of that, I want to point you and also talk a little bit about this particular paper. As we can see, this is a very, very recent article that came that's come up in archive online. And of course, I'll put this up in the course material also. This is Ben Radha, Anna Swami and Alexander Fratkov. These are, you know, rather senior researchers in the field of adaptive control and learning. So what I want to do is to sort of go over this paper, you know, sort of give an overview of this paper in this session and sort of get a feel for how adaptive control has evolved and what pieces of adaptive control we have sort of covered in this course. And of course, what is left and what we've left out of due to, again, lack of time in any course such as this. And we also want to see what kind of connections with learning has been made in literature. So this is what we call a survey paper. So I'm going to mark it here as lecture 12.1. So our last week of lectures. So, like I said, this is a survey paper. So we are not really going to do a lot of mathematics or discuss a lot of mathematics like we've been doing, but we look at how things have evolved. So, of course, you know, these are some of the highlights of course of this article. So there have been advances in the last 70 years in both in adaptive control in both, you know, deterministic and deterministic continuous time systems and also stochastic basically time systems. Okay, so these are the two sort of dynamical systems that have been looked at extensively in the domain of adaptive control. Right. And as you can see, we, we have covered of course, deterministic and continuous time systems, which is why I'm going to mark it in a different color here. But, but of course there exists a lot of literature and work on stochastic the state systems in adaptive control also. So, so this article of course is sort of trying to connect adaptive control to learning so it is it sort of looks at the developments that have significant intersection with parameter learning. And, and of course, I mean, as they usually do in survey articles, there's a lot of references that you can go through as part of different sections. And if you want to delve into a particular topic in much more detail. And so you will, you will of course get very good, very good idea of how you want to proceed to learn a particular topic if, if you're interested in that from here. So, so anyway, so, so we are, we start with this article starts with some kind of a chronology, right on how adaptive control went. So, in from 50s to 65, you know, you had a lot of work on deterministic continuous time systems. Right. And this is the sort of, you know, work that we also discussed. Right. So, there was a development of what is called the MIT rule, which formed the core of adaptive control. And, and this essentially was a way of updating unknown parameters by very standard gradient type formula and what is that the gradient type formula it is this equation of one that you see here. Right. So, now here that is of course the tracking error that we are so used to seeing. Between this process output y and the desired output y ym and the gradient is of course the gradient operator and the variant is taken with respect to the unknown parameter theta. Right. So, so basically the idea is, you know, sort of a way of, you know, sort of a steepest descent kind of an idea that you have particularly in optimizations like algorithms. So, this is essentially what was called the MIT rule. And this formed the basis of a lot of adaptive control algorithms. So, so basically a lot of motivation for adaptive control came from applications in aerospace for automotive for autopilot design. And flight control, and this still continues to be the case. A lot of new recent work in adaptive control has been applied to autopilots in fighter planes. And these have been tested, have been field tested successfully. In fact, of course, getting the, you know, FA clearances and security clearances, etc. The different matter altogether. But they have undergone significant field requirements. So, anyway, so, so basically, you know, so this is sort of what this section of this chicken talks about. So, so rather, rather, rather interesting, interesting set of topics on which there was a requirement to do adaptation of parameters and adaptation of unknown quantities. Yeah. So, so anyway, so, so this is a sort of motivation for what why or how adaptive control for continuous time systems did come about. There's also the discussion on this problem of pattern recognition and classification. And this was seen as a parallel development. Yeah, in adapt to adaptation. Right. And interestingly, they, and a lot of, of course, they were very, very different kinds of researchers who were working on these topics like Vidro, who was very, very famous. But then there were also folks from adaptive control and nonlinear control like Yakubovic, who also were working on it, Franckov Polyag. Yeah, and more recent works. Right. So, so basically, the idea is that there was a common element in to a diverse set of problems in pattern recognition and signal processing, which is the determination of this of a set of parameters or wings that leads to that was classification. Right. So eventually, there was also the notion of trying to identify parameters or weights and using basically just input output data. Right. There was no clear models per se, or the models were incomplete to a large extent. But the idea was to sort of identify parameters that would help describe the model. Right. And basically, Vidro's Adelene filters is one of the big, big foundations for neural networks deep in others. Yeah. So these Adelene filters became the foundations for what became neural networks. Yeah. So, so very, very old, very, very classical ones. Right. So anyway, so very, very interesting. Now, it was of course realized soon that this MIT will propose where we take a can result in stability. And of course, then there was a requirement to study the stability of this adaptive controllers. Right. I mean, it was not just enough to figure out a way to adapt for the practice, but of course to find a way to formulate a stability framework for analysis and synthesis of adaptive systems. And this is where, you know, there was a discussion, or there was a lot of delving into dynamics dynamical system notions. Many, many authors, I would, I would, again, point you more towards Narendra and Woodward, who are one of the seminal researchers in this area. And of course, Lyapunov's method was suggested in view of the gradient descent approach. And this is what we learned in this course. Most of what we did in adaptive control in this course was the use of the Lyapunov method. Right. And you can see the origin for this is, you know, going as far as the 70s. Right. So, not just Narendra, of course, you know, this is in a deterministic discrete time setting. This was being parallel to your address by Yakubovich in Russia. Right. So 68 and 72. So, so a lot of, a lot of different works. I mean, there was work by Landau. You have Wittenmark, you know, amongst others, I mean, you will find these names in, you know, very, very good books in adaptive control, Narendra and Anna Swamy, Astrum and Wittenmark, Ianu and Sun. So, Shastri and Watson, Tao, Kristic. So, Fratkov, Kumar, Varaya. So, who did the stochastic systems? So, so these basically address adaptive control architectures for a range of analytical systems. All right. With, of course, full or partial measurements and so on and so forth. So this was essentially, you know, I think you can trace it to about, you know, I would say, I mean, there's of course much more recent 2003, but you can trace it from 1975 to 8590 and so on. So almost 15 years of work, you know, actually laid the foundation for stable adaptive control. Right. So I would use the word, you know, stable adaptive control came about. Yeah, until then it was more like there was an update law for the parameters which was coming out of some kind of an optimization, but it did not really guarantee stability of the system. Right. So, so anyway, so this is, so we are, I mean, we are already aware of this kind of a structure for adjusting the parameter. This is what came out of, you know, most of the adaptive control laws. And in fact, we've also looked at update laws which look pretty much like this equation to here. Then, of course, there was this revise the suitable tone regressor. And then, you know, you have case with some kind of a normalization component or a weight if you may, which we call the adaptive control again. And then of course, there was the development of the motion of model reference adaptive control. Again, something we studied in the stochastic systems domain. You know, based on the work by Astrum and coworkers, you, we, the outcome of that was the development of what was called self tuning regulators. All right. And again, this is also being worked on by many other authors like Landau, Kumar, Clark, etc. All right. So, of course, then there were connections. In all these cases, there were a requirement for learning that is accurate parameter estimation, just like in signal processing, the Adeline filters and neural networks, you had the requirement for learning parameters and weights. Here also you are, we know we have parameters that we wanted to estimate. Right. So we in fact saw that in our, in several of our lectures. Right. So, this sort of connection was also made in stochastic approximation framework by Sipton in his work. Right. So, so the idea is that there was, you know, there was this parallel and similar evolution into branches of adaptive control and deterministic and stochastic systems. So before they were distinct rules, but this development was happening sort of parallel. Yeah. So, of course, this adaptive control terminology remained in vogue in deterministic systems, but then in the stochastic systems framework, it was termed a self tuning regulators. We of course did not cover any stochastic systems have tuning regulators in this course, but I will, you know, sort of strongly encourage you to look at that. Then, of course, 90s to present, you know, there were many love new novel flavors in adaptive control of deterministic and stochastic systems. So, one of the things that was evident, you know, was that both the gradient algorithm and the stability based algorithm in the Lyapunov approach had robustness issues to perturbations such as bounded disturbance and unknown dynamics. And this was figured out in 80s itself. Right. And so therefore, there were several approaches that were developed in order to nullify this issue. This was due to Narendra. We saw the Sigma modification, which was due to Eonu and Kokoto which was, sorry, which was due to Narendra. And the Epsilon modification which was due to Eonu and Kokoto which, right. So, so basically we wanted to ensure that these adaptive algorithms also provided robustness to withstand nonparametric uncertainty such as bounded external disturbances, right. So, so either of course they really, I mean, there was this connection to persistent excitation of exogenous signals, right. Or of course we modify the adaptive control in a suitable manner, which is what is the Sigma Epsilon modification, right. So, if you have persistent excitation, none of this was ever a problem. There was always nice convergence properties and all that. We saw this Narendra and Anaswamy's results and so on. And you know, if you did not have persistent excitation, we had to modify the adaptive law by Sigma and Epsilon modification. So, this is what we learnt and this is what was a sequence of development. So, again this, there was again a parallel, you know, in the learning framework and the learning domain on regularization in machine learning, right. And then again similar results in discrete time as well. So, so anyway, so this sort of highlights the parallels that you have in the discrete continuous stochastic framework that's happening, right. So, of course then there was a large, large requirement for addressing parametric uncertainties that appeared in different sort of cases, different out of situations. And a lot of special case nonlinear systems were started to be considered in the 90s, right. So, this was, you know, started with cristics work, you know, you start in and you need. And so a lot of focus on nonlinear system because of course most real systems are nonlinear. So, just focusing on linear systems or linearizing systems was not always a solution. And these were based on methods such as feedback deneration, back stepping and averaging. So, we did of course look at the back stepping based results, right. We did look at the back stepping based results mostly due to cristic. And of course some of them due to slotting and me, but then there were many, many authors, there's Leverets, there is Lottin, there is Anaswami, Narendra, Parthasarathi. So basically a lot of authors who did, you know, contribute to this area. So, by the end of, you know, 80s, there was this 80s and 90s. There was a focus on what is called reinforcement learning or approximate dynamic programming. So, and this sort of trend was highlighted by this paper in 92 by Satya Natal, which is basically reinforcement learning in direct adaptive concern. So, in usually the works on control based on reinforcement learning, typically a performance index is introduced, right. And then the neural network is used to approximate either the predicted optimal value of this functional or the optimal control policy. So, basically a neural network is a function approximator and we will see something later on. And the idea is that in this performance index, which is like a cost, which is usually an integral cost. The idea is that we in reinforcement learning is to use a function approximator to figure out the optimal value or the optimal policy that will give the optimal value, right. So, these, you know, they were made a lot of different terminology for this. And these are called either an approximate DP dynamic programming or neurodynamic programming or adaptive dynamic programming. So, all of these, you know, are sort of technologies that have been currently used. The thing to remember is that it's very difficult to give good analytical results in this setup because of obviously the complications arising due to dynamic programming, yeah. So, different sort of, as we are aware, I mean, we have seen there are many different kinds of problem statements. And I mean, this is the typical, you know, deterministic continuous time system where you want why to track some kind of a desired YC. I mean, or you may have some kind of a cost to be minimized. We usually do the first, I mean, we've looked at why tracking some YC. And then of course, we have these unknown parameters theta right here. So, this is sort of framework we have. And, and we, we've also looked at some kind of linearized versions of these, you know, with some regressor parameter types structure. Of course, the goal is usually to, you know, send the error to zero over time, right. And then this is what is the idea, right. Now, and the question is that if this, this can be ensured even when, you know, there are parametric uncertainties in theta, and also non parametric uncertainties and several non parametric uncertainties like modeling disturb modeling, unmodeled dynamics, disturbances, etc. Right. So, learning is always an important aspect of an effort. So, right, because we are always trying to learn some parameter theta. We, of course, look at, you know, this, the certainty equivalence principle, which found the basis of most of what we discussed in adaptive control. Right. So, so that's the idea. I mean, basically use a certainty equivalence principle is what came about of this. And a standard solution of an adaptive controller looks something like this. I mean, you have a controller which depends on some estimates of the parameters and regressor and time. Possibly. And then there is an update law, the parameter, which of course, again depends on the, would possibly depends on the estimate itself and the regressor. Right. So, so this is sort of the structure of what we have. Then you, of course, have the linear word, the model reference adaptive control, where instead of following the reference signal, you follow a reference model. So that's, we also, this is also something that we have seen. One of the points that this article tries to make is that learning is essentially equal to parameter estimation in adaptive control, very much so. The typical, you know, setting of learning for control, also this is the case. More often than not, whether it be reinforcement learning or deep learning, you are trying to learn some parameters. Right. So, and, and we did, I mean, many researchers established when such learning can happen, and these are connected to the persistence of excitation condition and uniform observability condition. So, we looked at this extensive, I mean, we looked at the work here. And in fact, in fact, mostly the Morgan Narendra paper in 77. So, this is what we looked at. So, this article is what we focused on mostly the results that you saw, but we also saw, you know, results for the results. Then there is, there have been some results on the notion of multiple models in adaptive control also. But anyway, that's an again, another parallel sort of a framework is what I would say. So, so yeah, so several, several interesting results on when parameter learning can happen. And we did look at this. I mean, again, so these were results from 70s and 80s. And we did look at some others. Beyond that, there was a notion of robustness in adaptive control also. So, the idea being that if there are unmodeled dynamics, time varying parameters and disturbances, how does the plant look? So, in the frequency domain, you would have something like this, where of course D is some kind of a disturbance and N is some kind of a noise. And Theta is some kind of a time varying quantity, right? So, you know, again, most of these results, these approaches on rely on properties of persistent excitation of exogenous commands. This is again Anderson, Narendra, etc. Or you modify the adaptive law. So, these are again the results that connect to, you know, I would, this is basically sigma epsilon modification. Yeah, this is what helped impart robustness. And so, of course, these are summarized in this result, right? There was also this, I mean, I'm not going to discuss this in too much detail again here, but there has also been a lot of development on stochastic and discrete time systems, adaptive control, in parallel. And as I mentioned, these are called, these have been termed as self-tuning regulators. And a lot of this work has been due to Kumar Varaya. A lot of different work connected to Varaya as strong, etc. So, if you search these authors, you will usually find some results. And so, there is also some little summary on what the self-tuning regulators, that is stochastic, discrete time, adaptive control results. So, this is sort of what you have. And the idea is that the stochastic, the self-tuning regulatory addresses the design of a minimum variance control, which looks something like this. And there is still the connection of parameter estimation and persistent excitation. I mean, that cannot, of course, be sort of given up, even in this case. I mean, the conditions for persistence may look different, but the requirement for persistent excitation still works. So, and then finally, there is also adaptive optimal control for LQG type of situations. And this was also studied where you have a discrete time system with unknown AB matrices. And you want to do some kind of an LQG control, that is, you want to minimize a cost which relies on the state X and also the control signal U. And you want to figure out how to minimize this cost whilst knowing that A and B are unknown and in the presence of noise. So, this is also something that has been looked at. So, has been looked at and formulated here. So, these are the sort of rather interesting sets of, or the interesting progress of how adaptive control has moved forward. Now, what we want to do in a subsequent session is to also look a little bit more at how this progress has happened. So, I mean, forgive me if you may for introducing you to some history in adaptive control. But what I did want you all to see is how the thought process of this innovation went. Yeah, I mean, there was a need based on autopilot aircraft flight systems, then came about some optimization based rules like the MIT rule for parameter updation. And then it was figured out that these are inadequate, so then stability mechanisms were developed, and then stable adaptive controllers came about. Then there was a realization that there is, you know, the stable adaptive controllers also do not guarantee robustness against disturbance and unmodified dynamics. So, then there was the notion of sigma epsilon modifications, persistent excitation for parameter learning, etc. And again, in parallel to all of this, there was the learning theory that was getting developed at the line filters. And there also parameter identification was essential, persistent excitation was formulated. And then there was, of course, the self tuning regulators in stochastic, notice great time systems, which we did not look at in our course. But that is also very, very large set of literature that's out there in adaptive. So I hope you got a good scene for how this adaptive control literatures evolved. I'm going to look at a little bit more at this article before we start into more discussion on the learning aspects in adaptive control. All right. Thank you and see you again soon.