 Welcome to this next session on data structures and algorithms. In this session, we will discuss the notions of big O basically continue on big O from the previous lecture, introduce new concepts such as that of omega, theta and small O. In addition, we will also provide an useful framework for analyzing recurrence equations. So, recall our formalization of the first notion of rate of growth. We tried and bounded the rate of growth of a function based on something called big O and it says that the upper bound for Tn is C times f of n for all values of n greater than equal to n naught. So, asymptotically the function is bounded Tn is bounded by some other function fn. We also explained the benefit of the C. C makes us C makes it possible to ignore the multiplicative constants that may be associated with fn. On the other hand, n naught helps ignore additive constants that may be a part of Tn. So, the idea is to focus only on the dominating term which is n. So, let us understand this growth with an example. So, we claim that a particular function fn defined here as 5 into n plus 6 is order of g n where g n is defined to be n. So, what are we saying that we are saying fn is phi n plus 6 is order n which means for all n greater than equal to n naught fn is upper bounded by some constant C times g n. So, let us understand with this illustration. So, we find that the pink line fn is indeed bounded not by g n itself but a scaled version of g n which is 7 times g n. So, you could think of C as 7 and therefore conclude that fn is order of n phi n plus 1 is order of n. This happens for some value of n naught which is somewhere between 2 and 4. So, certainly for all n greater than equal to 4 you have this inequality. How about some other interesting functions? How about g n itself? Now, you find that g n is always below fn. Can we define something more interesting between fn and g n? It appears that fn is sandwiched between g n and a scaled version of g n. So, this is basically 7 times g n and the curve here is basically g n and here is fn. It turns out that one can formally define something called theta and omega. So, omega is basically stating a lower bound just like big O gives you an upper bound. One can say that fn equals omega g n that is for all grade n greater than equal to n naught fn is less than equal to some C prime times g n. Here the C prime happens to be 1. So, let me rewrite this. This is indeed C prime equals 1 and this corresponds to 1 times g n. A small mistake in my inequality fn is lower bounded by C prime times g n. And the fact that fn is o g n and omega g n sandwiched between two versions of g n prompts another definition called theta. So, fn is theta of g n if fn is o g n and fn is also omega g n. One could also draw some very other interesting conclusions from this graph. The fact that fn has been lower bounded by g n with a scale value of 7 should also give you a hint that g n appears to be lower bounded by fn for some value of C prime. So, in fact, one can say that g n equals omega fn with C prime value of 1 by 7, exactly the reciprocal of what was used here. On similar terms, one could talk of g n being theta f of n. This is based on the fact that g n is omega fn and you could also conclude that g n is o of fn. And what implies that g n is o of fn? g n is o of fn, C prime equals 1 or C equals 1 if you want to be consistent with o. So, g n is o of fn, g n is omega fn that gives us g n is theta fn. So, let us formalize the other definitions. So, t n is omega fn if there are positive constant c and n naught such that t n greater than or equal to c of fn whenever n greater than or equal to n naught. So, it is just that now we have found a lower bound beyond an n naught. So, it is an asymptotic lower bound or the growth rate of t n is asymptotically more than that of g n. t n is theta fn if they have similar growth rates that is growth rate of t n and f n are the same. Now we can also define a small o fn. So, t n is small o fn if for all positive constant c there is an n naught such that t n is strictly less than small c times fn when n is greater than n that is the growth rate of t n is strictly less than that of fn. So, you could think of specific instance of big o. So, if t n equals small o fn you can conclude that t n is also big o fn. Now what is really the difference between small o and big o? Well if t n is big o of fn it may still be theta fn if t n also happens to be omega fn, but this is not true with small o if t n is small o fn it will not be theta fn there is no way that fn gives you a tight bound for t n because even if t n had omega fn you do not get in fact you get a contradiction. So, let us give some examples to illustrate what we just discussed 2 n plus 3 can be showed to be order n. Now it is possible to choose a value of n naught and a corresponding value of 6 as c such that t n is less than c fn that is 2 n plus 3 is less than 6 n for all n greater than equal to 1. Now there is no unique combination of c and n naught, but all you are interested in existence. Now one might wonder is there a unique function fn which gives you big o for a given function t? No, so it turns out that t n plus 3 is also o n square it is also o n cube. In fact it is o n raise to k for k greater than equal to 1 actually, but what we are interested in in general is the tightest or lowest order. Note that o was basically an upper bound. So, while you can get arbitrary upper bounds in fact upper bounds for the function that is basically big o for a function t n are also upper bounds. So, if t n is big o of f n and f n in turn is big o of g n then you can show that t n will also be big o of g n and so on. But what we want is the tightest upper bound. We want to go as much close to t n as possible. So, generally by big o f n for t n we mean the tightest upper bound. So, going back to the example you actually can show that you cannot get better than order n for 2 n plus 3. So, t n is actually order n and in fact t n is theta f n where f n is given by n. So, in which case the other higher order terms actually become meaningless. The other point to note is you might have lots of training terms so 4 n square plus n plus 5. Now you can show that this is nothing but order n square. It is also order n square plus n again order n square is order n square plus n. So, n square is order n square plus n and therefore all these higher orders will also be order of a 4 n square plus n plus 5. But as before we are interested in the tightest upper bound and one more hint towards getting to this is to ignore lower order terms. So, you typically ignore these lower order terms. Well of course you are interested in the lower order terms for the original function, but not while stating the order. So, thus 5 n plus 3 log n is order n because log n is also order n. So, we do not formally prove finding of c and n not. That is the general convention you just write the order intuitively based on the dominating term. And again point out that for this particular example 4 n square was the dominating term or 5 n was the dominating term. So, here is an exercise for you. Try and build an alternative search algorithm and we call it interpolation search. Now imagine that you were trying to find an element in a sequence S, but a large sequence S. And imagine that you had the sequence S written across hard disks. So, let us say this part of the sequence S was written on hard disk 1, other part was written in disk 2. So, remember that binary search required you to find a mid element. Now how do you find a mid element? Even an array for an array that gets sorted across a disk you might have that the high element is here and the low element is in the sub array on the other disk. So, the idea is instead of looking for mid all the time you can sometimes look for the next element. The call for the next element this should be remind you of linear scan. So, what we are doing here is interpolating between linear scan search which was an order n and binary search which was order of log n. Now again menace is practical well it makes sense only if every axis is very expensive. So, retrieving this array from disk 1 and then retrieving the array from disk 2 and then going back to disk 1 and so on is expensive holding it in memory can be expensive. So, therefore, each comparison might require a disk access. The other requirement is that data must not only be sorted it must be fairly uniformly distributed. What you do not want is a sequence like 1, 2, 4, 8, 16. Sequence scan will binary search may not be productive. So, a phone book is fairly uniformly distributed. So, for example, if the input items are 1, 2, 4, 8 the distribution is not uniform. So, you can try out the exercise of analyzing such an interpolation search begin with you might want to formulate the interpolation search algorithm and then analyze it. We are going to next discuss a very important theorem that we could make use of to prove several recurrence relations. As you will see several algorithms such as divide and conquer have recursions they are recursive in nature. And therefore, recurrence relations become very natural in expressing the time complexity. So, what does this mean? This means that T n is T of n by a plus T of 1 minus 1 by a times n plus something, something and so on. How do you solve such general recurrence relations? It is enough to give some asymptotic characterization for associating the cost of an algorithm. So, we might not be interested in T n for very small values of n as long as we are able to characterize both the recurrence relation and its solution for n greater than equal to n naught for all n greater than equal to n naught we are fine. So, the master theorem gives such a tool for solving recurrence relations in the asymptotic case. More specifically, if T n is of the form a times T n by b plus f n, where a and b are constants greater than equal to 1 and f n is a function, you could invoke the master theorem to solve. For more details of the master theorem you could look at section 4.5 of the third generation of CLRS. So, what is the master theorem? So, let us say T n is a times T n by b plus f n. Now, this n by b here could be replaced with either the floor or its ceiling and master theorem provides a following asymptotic bounds for T n. So, it is all based on cases and there are three cases. If f n is big O and I repeat we are looking at an upper bound for f n which looks like n raised to log to the base b of a minus epsilon. This has something to do with the structure of the problem that is invoked to prove the master theorem, but many cases this just turns out to be something like square or cubes or n raised to 2 is a very classic example. However, you need a small epsilon which is non-zero. In such a case you can say that T n is theta n raised to log to the base b of a. Another example or another case is when f n is big O of n raised to log to the base b of a, there is no epsilon the epsilon is missing. It turns out that T n continues to have a theta component with n raised to log to the base b of a, but there is also an additional log n. So, this is a kind of price that you are paying for not being able to prove strict better lower bound that we had in case 1. A third case is when you actually have a lower bound for f n with a plus epsilon. You are not even able to prove lower bound with n raised to log to the base b of a. You need this epsilon shift. Additionally, if f of n by b times a is less than equal to c of f n for some constant c and for all sufficiently large n, then T n is theta of f n. So, now what did we find? We basically saying that f n started dominating the third case. In the second case and in fact even in the first case we somehow were able to get f n on power with T n. But in the third case, the f n actually just dominates the recurrence relation yields to a large value of f n. So, the intuition here is basically to compare function f n with n raised to log to the base b of a. Larger of the true determines the solution. The case 3 is where we f n was really large. So, case 1 is if the function is larger than f n, case 2 is f n and then 2 log to the base b of a are of similar size. Case 3 is when the function f n is dominating. And here are some examples. So, T n is 9 times T of n by 3 here. A is 9, B is 3. There is also f n which is n. So, for suitable values of a and b for these values of a and b and for a suitable epsilon, we can show that n raised to log to the base b of a is actually theta n square. So, log to the base b of a is nothing but square is 2. So, it is n square minus epsilon and this is basically theta n square. So, since f n is big o of n raised to log to the base 3 of 9 where epsilon is 1, we can apply case 1 to find that T n is theta n square. So, what we have here is n raised to log to the base b of a actually dominates. We can also consider another example. A different case here. This involves case 2. The previous was case 1. So, case 2 is about f n and t f n and n raised to the log to the base b of a being comparable and the log to the base b of a is log to the base 5 by 3 of 1. Now, you can again find out f n and f n turns out to be theta of 1. This is basically can be shown to be theta constant 1. In this case, you can actually apply the second case T n is theta log n. So, remember we have this additional log n component and we will have the n raised to log to the base b of a which turns out to be just theta 1. A third example is when we have an omega lower bound for f n. So, f n actually starts dominating. So, a is 3, b is 4. So, b the base for the log is actually larger than a. So, you can show that n raised to log to the base b of a is order of n raised to 0.793 and this basically shows that f n is omega times n raised to 0.793 plus epsilon. Now, this epsilon is whatever needs to be computed the remaining component is 1 minus 0.793 and you can easily show that this leads to the case 3 which basically gives you a theta n log n tight bound. There are several cases where you cannot even invoke master's theorem. So, for example, if T n is cos n not very clear how we could invoke master's theorem because the fundamental problem is T n is not even monotone. You could also look at other exponentiated forms of f n. So, f n could be 3 raised to n. It is not even polynomial because remember that we were comparing f n with n raised to the log to the base b of a where a and b are expected to be constants. So, you are not able to deal with cases where n sits in the exponent. So, we are in general interested in polynomial f n's as far as master's theorem is concerned. Now, well, you might have a polynomial T n is under root n square plus 3, but then b is not a constant. Again, you could have a case where a is not a constant. It all boils down to writing down these expressions here and identifying a possible exponents for n, but in neither case you will find n of the form n raised to log base b of a for constant a or constant b. And finally, here is an interesting case where f n is not even positive. And well, that is in the case that master's theorem can handle. Thank you.