 Welcome everyone, we will continue with our topic of optimization. If you remember, we were previously studying what is called real analysis. Real analysis is a careful study of real numbers and real numbers and their associated properties. So we studied the following topic so far. We have defined what the sequence is. We defined what it meant for a sequence to be bounded. So we defined a bounded sequence. We defined what it means for a sequence to converge. So all concepts related to convergence, what it means for a sequence to converge, what it means for a sequence to not converge and what the limit of a sequence is. And from that we also defined what is called a sub sequence. These were the four topics we covered so far in real analysis. Now let me take off from here. So it is very easy to see that if a sequence converges, then every sub sequence of that sequence converges. And it converges to the same limit. It does so to the same limit as the original sequence. Now can you tell me if the converse of this statement is true? So if a sequence, if every sub sequence of a sequence converges, then does the original sequence itself converge? Yes, no? Yes. Someone saying yes, why yes? That is the correct answer. So the sequence itself is a trivial subsequence of the original sequence. So the converse is actually trivially true. This statement is the non-trivial one. Now let us, we were so far looking at sequences that were in living in Rn. So these were vector value, the sequence of vectors. Now let us come down to sequences that are in on the real line only. In that we have a famous theorem which is called the Bolzano-Weistras theorem. Bolzano-Weistras theorem simply says the following. This does not necessarily have to be in, this is not necessarily true for real numbers. So I will let me, this is true for vectors also. So I will tell you a more general version of the same statement. So the statement is simply this that every bounded sequence in Rn has a convergent subsequence. If you consider sequences in Rn and make, look at those sequence, if you and consider a bounded sequence in Rn and look at all its possible subsequences. Amongst all its possible subsequences you will find at least one which is convergent, convergent to some limit. It cannot happen that all the, all these subsequences do not converge and yet the sequence remains bounded. It is not possible. Now the, this is actually a very deep fact. Essentially what it is saying is that if you constrain the sequence to be in a box, so if you suppose here are my axis and my sequence is living in this bounded box. Eventually it will, the very fact that you are constraining it to live in a bounded box would mean that you can extract some subsequence out of this such that the, that subsequence eventually starts accumulating around some point. The sequence cannot dance around in such a wild manner that even its subsequences are always dancing around and never will you find a single subsequence which is convergent. So this is actually a very useful fact because it, convergence of algorithms are make extensive use of this particular proof, convergence proofs of algorithms. I will just give you a general sense about this. See when we are computing something in optimization what we are effectively generating is a sequence of one, sequence of iterate. Sequence you have iterate at one then another you go through another loop straight to the next iterate then you get to the next iterate then you get to the next iterate etc. So what you get are these, is this iteration x of 1, x of 2, x of 3, etc, etc, etc. And what we want is eventually that this sequence, if you look at the limit of this sequence x of n, this limit should converge to the solution of your problem, solution of the problem under consideration. So for example if you are looking at this, if you want to, if you are talking of optimization problem, you want this to converge to the solution of your optimization problem. Now to show one of the big challenges that occurs in analysis of algorithm is making sure that this actually happens. And we may often the steps towards arguing that go through, you need first a way of arguing that the sequence converges at all to begin with. And that comes from Bolzano-Weistra theorem. You argue that there is at least a subsequence that converges and from there you build your argument. It is a common way of arguing. So this is Bolzano-Weistra theorem. Now let me come to sequences or sets in the reals. So now let us look at just the real line and look at suppose some set of points on the real line. So for instance, look at the set which is, I will just write this in words, set of positive rational numbers for instance. So this is a set of all numbers m by n, n not equal to 0, m and n, m comma n both positive and they are natural. So if you look at this set, this is a set of all positive rational numbers. Now we say that a number l is said to be a lower bound, l is said to be a lower bound if it is less than equal to all elements of s. So now if you take s to be the set of positive rational numbers, what can you give me an example of a lower bound minus 1, 0, these are all lower bound. Likewise number u in r and upper bound on s the opposite is true. x is less than equal to u for all x in s. Now again for the same example set of all positive rational numbers, can you give me an example of an upper bound, there is no upper bound. Now what we will, if a set is bounded, if a set which is a subset of r is bounded, then does it have an upper bound and a lower bound by definition. So it does have an upper and lower bound. What did we say, what did we mean by a set is bounded? We said we look at the norm of the elements in that set and all elements should have a norm less than equal to a prescribed number. So what it means, in this case we are talking of a set of real numbers. So the norm is just the absolute value of the number. So the absolute value of every real number, every number in that set is between minus m to plus n where m is some positive number. So if the absolute value is between minus m to plus m, then what does this mean? So mod x is less than equal to m, this means that x is less than equal to m and greater than equal to minus m. What does this mean for all x in s? What this means is that m is an upper bound and minus m is a lower bound. So a set if it is bounded will have an upper bound as well as a lower bound. Now upper and lower bounds are not unique. If m is an upper bound, then m plus 1 is also an upper bound. Likewise if l is a lower bound, then l minus 1 is also a lower bound. You can keep making lower bound smaller and smaller, they will continue to be lower bound. If you make upper bound larger and larger, they become continue to be upper bound. So here is the another concept which is called least upper bound. Least upper bound is the smallest such upper bound amongst all upper bound. What does this mean? That if I take a number, if I take a number even slightly smaller than this least upper bound, then it cannot be an upper bound. Let me write this in English first. The least of all upper bounds, what does this mean? If I for all epsilon greater than 0, there exists an x in S such that x is greater than, sorry, sorry, let us write this better, m is set least upper bound in A set S of R. If for all epsilon greater than 0, there exists an x in S such that x is greater than, x is greater than n minus epsilon. So m is said to be the least upper bound on a set in S which is a subset of real numbers. If for every epsilon greater than 0, you can find an x such that x is greater than m minus epsilon. You cannot make m even slightly smaller and still have it as a lower as an upper bound. So there will be at least one element whose value will be greater than m minus epsilon for every epsilon. Likewise, l is said to be the greatest if for all epsilon greater than 0, there exists an x in the set such that if I increase l even slightly, then I can find an element that violates my, is this fine? Now, there is a chicken and egg problem that arises in trying to define what this, trying to show that such a thing exists. The very fact that to show if you want to show that the asset must have a least upper bound, then what you want to what you need is that the set of upper bound must have a lower bound. Likewise, if you want to show that the asset must have a greatest lower bound, then it means that the set of lower bounds must have an upper bound, a smallest such upper bound. So, this is worked out using what is called the completeness axiom. Completeness axiom simply says that the bounded set of real numbers upper bound. So, if your set is bounded, then it does have a least upper bound and a greatest lower bound. If your set is just bounded on one side, means it is bounded above, then it has a greatest, it has a least upper bound, but may not have a, but does not have a greatest lower bound. If it is bounded below, then it has a greatest lower bound, but does not have a least upper bound. But if it is bounded, it has both least upper bound and greatest lower bound. So, this leads us to the definition. So, once you have with the axiom, you know that such a thing exists. So, the greatest lower bound has a name, instead of calling it the greatest lower bound on S, we simply say the supremum of S and it is written as this, sorry minus vector. We simply say the infimum of S and least upper bound called the supremum.