 This material is ordinarily not covered in the first course on programming at the initial stages. However, I consider this notion very important because it tells you how careful you have to be while designing your algorithms and programs so that you write very efficient course or very efficient programs. We had introduced the notion of time complexity last time, we will quickly revisit that and continue discussion on defining the notion of algorithms which are costlier in terms of execution time and algorithms which are cheaper in terms of execution time. Naturally, our endeavor is to develop and write programs which are as efficient as possible. So, this is what we had remarked last time that while our C++ Dumbo works very fast, it does take a finite amount of time for every computation and therefore we must design our programs very carefully such that we do not make Dumbo do any work more than what is absolutely essential. And we had said that the order of magnitude of time required to execute any program is generally called the time complexity of the program or time complexity of the algorithm. So the steps in our program should be so designed that the execution time is minimized. This is the general concept, common sense, we would like to make somebody else do as little work as possible which is minimally required whether it is Dumbo or a human being. This was the simple example that we had taken where considered estimation of pi using estimate of an area of a square which was drawn on top of two radii. So we said that the area of the circle is pi r square, we consider a unit this it is simply pi and we said if we discretize this square into some n points then for i equal to 1 to n on x axis and for j equal to 1 to n on y axis if we figure out how many points lie within this circle we can get a good estimate of the area of this the value of pi because that area is obviously pi by 4. So this is what we had written this was a program to estimate the value of pi we had done this rate the value of n which could be 1000, 10000 whatever and we said that this is how we will estimate pi by finding out whether any point i comma j is within the circle or outside this if the value of i square plus j square was less than n square we said the point is inside the circle and therefore we would add 1 to the count. Essentially in this algorithm this is the instruction which was to be executed many number of times because this happens to be inside this iteration for j which itself is inside this iteration for i since each iteration will execute n times and this is a nested iteration that total number of executions of this if statement will be n square times and every time I am computing one multiplication another multiplication third multiplication one addition and one comparison consequently the number of computations which are proportional to n square will be very heavy. We had said later on that we could reduce the execution time first of all by taking the n square computation out because n was a constant value additionally we said that instead of considering the square we can consider only a triangle which is half of the square with area pi by 8 and this will reduce computations by half coming back to this slide then we notice that this is where the maximum computations are done but this time in the version 3 we said we will not start j equal to 1 but we will start j equal to i that means we are looking only at the points within the triangle rather than points within the square this will reduce the complexity significantly and we calculate the pi by multiplying this count by n 2 by 8 rather than 4. This in general is a commonsense way of reducing computations however when you have a very large and complex algorithm the analysis of the steps of that algorithm to find out how long it will take to execute is sometimes not a very simple matter it is important therefore for us to understand how do we go about it how exactly is the time complexity defined and what kind of care we ought to take while designing our programs these were some execution results you will recall that the version 2 was taking something like 36 seconds whereas this version takes only 17 seconds for 50,000 points notice that while every individual computation is done by within few microseconds by Mr. Dumbo when you make it do many computations even that would take sizable amount of time which becomes noticeable to you what is important to understand is that if a program executes in say 500 milliseconds another program executes in 1200 milliseconds for us the time taken is just about 1 second so it does not matter but one program is still faster than the other it starts mattering when the execution time becomes many seconds many minutes many hours or many days and that is why it is important for us to understand that every algorithm that you write should be as efficiently written as possible we had looked at the real time user time and sys time this is the same repetition of the last slide however the last line has been added during your labs now when you execute your programs it will be useful to use the time command so instead of saying dot slash a dot out you may say time dot slash a dot out so that the time command executes that a dot out giving you the system user and real time you might want to study the time command details it has various parameters which you can give as options and you can get the details of different kinds for that you might want to study the manual for time by saying man time it would be a useful exercise so we again look at time complexity of an algorithm there are actually two ways of looking at it one is what I call the micro view the nitty gritty view details because as we say they will lies in the details so we want to find out how many arithmetic operations are done how many comparisons are done how many assignments are done how many additions subtractions multiplications etc that is the micro view going to the nitty gritty we look at the time taken by each instruction first of all and the number of times that instruction is executed so addition may take less time than multiplication so we have to assign some timings to these different instruction and then we have to count how many times an instruction is executed in a program so while our dumbo executes instruction a few tens of microseconds or even in hundreds of nanoseconds every instruction takes a different amount of time depending upon its nature that is what is important to understand it is not that addition multiplication comparison assignment all take the same unit of time that is not true depending upon the complexity of the instruction dumbo takes different units of time to execute that instruction and it is important for us to compare these or count these so what we do is we simply define an arbitrary unit time the unit time could be a few microseconds for a fast machine a few milliseconds for a very slow machine tens of nanoseconds for a very very fast machine but whatever with respect to that unit all other relative times can be written down so it is a good exercise to do that this is some hypothetical comparative execution times for example if you are conducting an assignment a equal to five or a equal to seven we assume that it takes one unit of time whatever may be microsecond if I am calculating an addition or subtraction typically it might take two units of time so please note that if I say a is equal to b plus c to execute this instruction there are two operations that are happening b plus c is an addition that will take two units of time once the value is calculated to put it in a will take one unit of time a total of three units of time would be required similarly multiplication may take three units of time division may take five units of time comparison inside your if statement if you say if I less than j comparison may take three units of time these are all with respect to integer value but if you have floating point operations typically you may multiply by five the times that it takes for doing arithmetic operations for integers assignment however will take the same amount of time because assignment is taking a value and dumping it into a drawer it doesn't matter whether it's an integer value or floating point value but computations comparisons will take different amount of times so floating point arithmetic as a thumb rule is costlier than fixed point or integer arithmetic a special point to note when you make an assignment to an array element if you say a five equal to something a five is well known dumbo knows where the drawer for a five is but suppose you have an assignment which says a three into i plus j equal to something now that something may be a constant value but where to put that constant value is not very clear to dumbo so dumbo will have to calculate based on the existing value of i and j this expression this expression is called index expression and that index expression will take finite amount of time to calculate for example this expression has one multiplication and one addition so a multiplication assuming integer values will take three units of time addition will take two units of time consequently this index expression itself will take five units of time after doing this computation dumbo knows the location where to put the final value so that assignment will take one unit of time consequently a three star i plus j equal to some constant value also will take three plus two plus one six units of time as I said ordinarily it doesn't matter because each operation the unit would be few microseconds but we have seen in the program for estimating power that then when you are doing large amount of computations these differences start mattering immediately so how do you proceed in your micro view what you do is you find out totally how many assignment operations multiplication operations division operations comparison operations etc. count these number of units and find out a sum total however this can be done in an algorithm which does not have iterations which are executed some n number of times because n will be defined as input value you don't know it in advance so obviously such computations of execution time will have to be done in terms of n which is an external variable so if there is an iteration which executes n time where n is an input value you will agree then that n will largely determine the amount of computations in a program because the iterations will be dependent on n either you will have nested iteration or simple iteration and if n is large the computations done inside the iteration will amount to much more than what is done outside simple declarations and assignments initialization at the end computing some final value these all things would be done once only whereas what you do within the iteration will be done n times consequently n is termed as the size of the problem so you have n students you have n numbers you have n by n matrix etc. so n is the size of the problem and therefore you try to determine the total number of computational operations carried out by Dumbo in terms of this n for example for our pi calculations we already know that the nested iteration will execute n square times within n square times we are doing some arithmetic comparison multiplication whatever whatever I have given some arbitrary values here 24 n square plus 78 n plus 183 these are let us say the total number of units of time that this program takes how do I calculate these well the initialization such as setting up an iteration i equal to 1 or i equal to 0 j equal to 0 etc will take some fix amount of time at the end when I calculate the value of pi by multiplying the area by 8 that will take some amount of computation I am assuming arbitrarily that that is 183 the outer iteration is executed n times so whatever happens inside happens n times if there is something which happens between the outer iteration and external iteration so this coefficient is again arbitrarily put it could be 0 it could be 2000 it could be whatever and what you do inside the inner iteration which is executed n square times again I have arbitrarily assume that you execute instructions amounting to say 24 units of time these figures of arbitrary but they can be exactly calculated or counter based on the algorithm that you have what is important to notice that in general I will have for such algorithms which have two nested loops based on the size of the problem n the time complexity will be given by a formula such as a n square plus b n plus c where a b and c would be different values depending upon the actual instructions that you execute is that ok is no confusion so far now comes the important point if we want to take macro view a larger view after all n will vary but when I compare two algorithms we can get such expressions for each one and then compare for specific values of n so consider two hypothetical programs one is called program one the other is called program two let us assume that the time complexity of program one is say 24 n square plus 78 n plus 183 let us assume that the time complexity of program two is 12,586 n plus 6453 obviously this program two which is solving some different problem of size n not necessarily the same or it could be the same problem in a different way but it obviously has a single iteration it does not have double iteration because if it has double iteration nested iteration n square would have cropped in somewhere but if it has a single iteration which is run say i equal to 0 to n times or 1 to n times I will have only a factor n here now how do you compare these two expressions obviously the first expression will run faster than the second expression up to a certain value of n because the coefficients are small consider for example n is 5 if n is 5 n square is 25 25 into 24 plus 78 into 5 plus 183 is much smaller than 12,586 into 5 plus this you agree so then this first algorithm runs faster up to a certain value of n what happens if that value is crossed then the second algorithm will be faster what is that value can you do some arithmetic and find out very quickly two minutes you will have to do such things in the quiz day after tomorrow so you might as well do it anybody has found it 520 anybody else has found on this side how much 5 521 we have a we have a very innovative answer 521 point how much when we describe the size of a problem or even if you look at the calculation that we are doing n is always taken as an integer so there is no there is no meaning of evaluating expression for a fractional value of n so what you have to do is you have to say for suppose you get a fraction 525.6 or whatever then for values less of n less than that one algorithm will be faster for values of n greater than that another algorithm will be faster but you are clear on how to compute the time complexity of an algorithm by doing such evaluation now here is the point the point is that for all larger values of n greater than some threshold whatever 520 521 525 whatever beyond that the second algorithm will run faster for every value of n larger than that that means for truly large n the program 2 is more efficient than program 1 is that understood for truly large n program 2 is more efficient than program 1 so we should in general be looking for an algorithm which has which terms either constant values or a term involving n but if there is a algorithm which involves n square another algorithm which involves n cube then some time or the other no matter what the coefficients are there will be some value of n beyond which the lower order algorithm will be faster so this order of the algorithm time complexity it becomes an important notion when we take the macro view or the larger view of our programs so the macro view is customary to compare algorithms on the basis of their behavior for very large values of n theoretically the limit as n tends to infinity so as n tends to infinity the limit of program 1 will tend to something n square and limit of program 2 will tend to n so therefore we say that program 1 is order n square program 2 is order n is that clear is that simple there could be a program which is order n cube in general a program of the order polynomial order a into n raised to p is plus b into n raised to p minus 1 etc etc is polynomial order polynomial order algorithms will take larger and larger depending upon the order of the polynomial degree 2 degree 3 degree 4 will take larger and larger and larger and much larger by the way it is not just a linear increase so consequently what do we learn from all these analysis while those who learn computer science theory more will find out variety of ways of figuring out more details about the time complexity and in fact how to design algorithms with known bounds on the time complexity but for this first course it is adequate for us to understand the notion of time complexity in the large that is as increases what should be our endeavor will be first try to reduce the order of complexity so whether we are able to count the nitty gritties in how many comparisons how many multiplications etc exactly or not it does not matter as much as it matters to find out what is the order of our algorithm for a given problem size n if it is n cube we should try to reduce it to n square if it is n square we should try to reduce to n if it is n what can we try to reduce it to constant value so it does not depend on n there could be something between n and n square what about n raise to 1.5 there will be something up between n square and n cube n raise to 2.3 what is between constant value and n log of n for example log of n increases far less rapidly than n does so if I have an algorithm with time complexity of logarithm of n of the order then it is much better than even an order n algorithm in the long run so this should be our endeavor always remember when you very briefly in your mind you should get a hand that whatever program you are writing what is the order of complexity is it order n order n square order n cube less than order n etc it is a good practice to have although you may not be able to do much about it but you should know what algorithm you are writing and given two comparable algorithms of the same order say I have two algorithms which are order n or two algorithms which are order n square then I should pay attention to reduce the coefficients of the expression. So let us go back to the previous slide or previous to previous slide we have said hypothetically that our algorithm for calculating pi using triangle takes 24 n square plus 78 n plus 183 clearly the algorithm for calculating those point accounting those points in a square will take twice as many for the n square right so that that algorithm will be taking larger time. So we conclude that in general our effort while writing our program should be a design the algorithm such that it has minimum order of complexity in terms of the size of the problem n and b for a given same complexity if I have two algorithms I should try to find one which has smaller values for the coefficients of such an expression that we saw. So this is the bottom line I would expect all of you to be conscious of this because the computer works very fast we do not bother about the time it takes to execute the program. There is another type of complexity of the algorithm which is called space complexity how big is your algorithm not in terms of the number of instructions that you write but number of storage locations that you use. So if you use very large amount of storage it is possible to trade storage for time there are problems wherein if you have very massive storage then you keep things stored inside you do not have to go out and inside the machine to do the computation and you can get time reduced. We shall see subsequently what does space complexity of an algorithm is and how is it related to time complexity although we will just mention it in passing towards the end of the course but suffice it to say that right now I would expect each one of you to be consciously looking at your own program and at least making a mental image of what is the order of time complexity of that algorithm. Usually the algorithm that you will write will be polynomial time order n, order n square, order n cube but whenever we talk of variation 1, variation 2, variation 3 you will notice that our endeavor has been that even though the order of the algorithm is order n square we try to reduce the coefficient so that the amount of time taken by that program reduces. We are going to examine this particular notion further by looking at an example of locating an element in an array for our purpose we have taken these 8 sample roll numbers and their marks you are familiar with this is already there on your web I think I have added one more number because there were only 7 in that example there are 8 roll numbers and 8 marks. Please remember what we are discussing is in the context of having not 8 roll numbers and 8 marks but 800, 8000, 80000 and in fact we shall later on consider a problem which is so large that we cannot even define an array in computer's memory because computer's memory will fall short. We shall not discuss that problem today but we shall hint upon what happens and what we need to do when we are executing programs of that large magnitude of data that we have to handle. But currently the problem here is very simple these roll numbers and marks have been read into the arrays, roll and marks and then somebody gives us a roll number we call it a given roll our job is to find out within the arrays if that given roll number exists for example given roll is 1006 I may search 1, 2, 3, 4 and whenever I find 1006 I say yes found and I give this one but suppose the given roll number is 1007 I may search here, here, here, here, here, here, here, here, here not found so I will have to report I have not found. It is a simple algorithm simple common sense but let us find out in how many different ways we can make this search possible. So here are the program find marks dot cpp I have written a small variation of that we had 2 arrays roll and marks and we had n students we defined a variable called given roll and we define a variable for found marks we define a variable for position and we define an iteration index variable I you will recall that we had written a function to get data for so many roll numbers and marks and that will get us the value for n students for the time being we assume n students is equal to 8. So the array roll and array marks is filled up with 0, 1, 2, 3, 4, 5, 6, 7 elements here 0, 1, 2, 3, 4, 5, 6, 7 elements here. Now we read a given roll number and we have to find whether that roll number exists or not. Here is a simple iteration what I do is I just go through each and every element of the roll number array. If we go back to the previous slide effectively what it means is we take the given roll and go through this compare it with this compare it with this compare it with this and so on with all the numbers. Let us go back to slide 17 here so what we are doing is we are setting up an iteration. So I go through this array 0th element first element second element up to n minus 1 elements. If the ith element of that roll is given roll then I say found marks is equal to marks i. Then I say output marks for given roll are these and I return. There is one problem however in this particular way the way I have written the program that if the given roll does not exist. If I give roll number 1,945 which does not exist then what will this program do? It will print marks for 1,745 whatever are what will it print here? Found marks it will print some arbitrary value because found marks has never been assigned. Please note that found marks get assigned only if the iteration while executing the iteration some roll number is found. Otherwise found marks have no assigned value shall get some arbitrary value. The other problem here is that I keep going through all the elements unnecessarily even after I have found the fellow. So let me correct this step by step let us say. In the version 1 I actually create a new variable called found flag. It is like a green flag red flag. I start with a red flag whose value is 0. Found flag equal to 0 means I have not found anybody. So this is an arbitrary assumption that I made. In this program I assume that the arrays have been red. I just declare pass for a position where I find the fellow if at all I do. I assume that given roll etc. has been declared. I read the given roll and now I set up an iteration. So for i equal to 0 to n I check if it is given roll. If it is I set found flag equal to 1. The green flag is up because I have found him and I set the position which is the current position i to pass. So if at all I have found him when I come out the found flag would be 1 if I have found him. In which case if found flag is equal to 1 I say found at such and such position because the pass would have been assigned. Value of no it will not be equal to n because this statement is executed only if this condition is true. If roll i is equal to given roll then and only then I execute these statements otherwise I do not. So obviously the given roll will be equal to some roll number only in one case. In no other case. So only in the case where it is equal the pass will be set. In all other cases this if statement will compare wrongly and this whole thing will not be executed at all. I will simply go to the next slide. However even this program suffers from the fact that I am unnecessarily making comparisons even after locating a fellow. If I do not locate a person throughout these iterations then this makes sense that I have not found anybody. So I say else not found. However if I have found somebody let us say the first roll number itself was given roll number. Then I am unnecessarily doing the ghodagiri of looking at n minus 1 students because I have already found one. So I will change this now. This is a version 2. So here again I said found flag equal to 0. I read the given roll. But now I do it like this while found flag equal to 0 and i less than n. If roll i is equal to given roll found flag is 1 pass is equal to i. Then I said i equal to i plus 1. Notice that when I go back so I am actually simulating the for loop by actually incrementing i internally and checking if i is less than n. But the main condition of exit is found flag equal to 0 not equal to 0. If it is equal to 0 I keep doing this. Since I start with found flag equal to 0 I go on doing this. There is only one mistake in this program. Has anybody located it? Yes. What is the initial value of i? For the first time when I come in when I say is roll i equal to given roll what is roll i not known. So what should I do? I must be initialized here. Where could I initialize it? I could do that here for example. So I will start with i equal to 0. If roll 0 is given roll I have found this. Please note found flag becomes 1. So therefore I will exit the file loop when I will become actually 1. But that 0 would have been captured in position. Now this iteration do you agree is more efficient than the first iteration? If the first iteration was taking say something multiplied by n units of time because it was order n. Has the order of this algorithm changed? This is a question. Please read these two algorithms very carefully. The first one. So for the first one if I were to write the time complexity in terms of let us say a n square plus b n plus c. What is the order of this algorithm? Is it order n square? Is this algorithm order n square? No because the iteration there is only one iteration here and it executes from 0 to n. So obviously the number of times any computations comparisons etc are done. There will be a finite multiple of n only. So there is no question of n square. So this algorithm is order n. This algorithm is being executed n times from 0 to n. If I forget the earlier and the later instructions the behavior of this algorithm is determined by the value of n and it is a linear order. So order n but in this order n how many times is the iteration executed n times? 0, 1, 2, 3, 4, 5, 6, 7 n times. Let us look at the next algorithm. This is a version 2. This algorithm is a file iteration. So it is very difficult to determine how many times in terms of n the algorithm is executed. What is the worst case? Suppose the roll number does not exist at all then it will be executed n times. What is the best case? When the first roll number itself is this given roll then this loop will be executed exactly once. So best case you will execute all these instructions inside iteration once. In the worst case you will execute them n times. On an average how many times will this loop be executed? Ordinarily you will be searching for a fellow who is within your roll number list. If the roll numbers are between say 1,001 and 1,999 for this group it is highly unlikely that as a teacher I will be searching for a roll number 5,720. Although a few roll numbers may be missing I will generally search within the known range. Consequently most often I will find that person. So I am searching for a person who exists in that array. The best case is one search. The worst case is n searches if that person appears to be lost. On an average I will do n by 2 searches. Do you agree then that version 2 is more efficient than version 1? The first version was order n. Second version is also order n. But the coefficient is on an average how? Yeah you have a question. Yes this is position of the array in which the roll number is found. Yeah it will show as 0 that's okay. So what I am printing is index. Let's say position not as we count natural number 1, 2, 3 but we will say position in the array. That's right. But essentially do you agree that the second algorithm is better than the first one because on an average it will do less computations than the first one. In the worst case both algorithms will do roughly same amount of computations. Yes in the worst case both algorithms will do same amount of computations. Okay my answer is no. Tell me why. My answer is in the worst case version 2 will carry out more number of computations. Why is that? In the worst case you agree that both iterations will execute n times but please note that in the second algorithm I am checking for found flag equal to 0. I am making an extra comparison. I less than n was being done earlier. I plus I equal to I plus 1 was being done in the for loop. So all those computations are same but I am doing an extra comparison. So n times I will be doing extra comparisons in the worst case. So in the worst case second algorithm is marginally costlier than the first algorithm. But on an average because it will execute n by 2 times this is much faster. Can we make it better? Given that this is my array and if I assume that the roll numbers are stored in the array in increasing order of roll numbers. Can I do something better? Okay let me put it this way. Have you ever seen printed results of an examination where people who have passed their roll numbers are printed in newspapers? Do you apply the first or second variation where you say given roll number is the first one equal to this, second one equal to this? Do you ever read the complete list published in the newspaper? Why? Because the roll numbers are arranged in increasing order. So consequently you naturally go somewhere in between to find out where that given roll number is and if not you decide to go either up or down in your search. How do you search a word in dictionary? Suppose you are searching for zebra. Do you start reading a, n, something, something, each word? Is it zebra, is it zebra, is it zebra? In fact when the word starts with z you will automatically go towards the end of the dictionary. If the word starts with m you will go somewhere in between. Consequently there is a different natural searching mechanism that human mind uses when things are in an ordered fashion. If things are ordered alphabetically ordered for dictionary or ordered in increasing order of roll numbers then the search that you use is a different kind of search. That search is called binary search. This binary search in an array is easily understood if you recall the bisection method that we used for finding out roots of an equation. I will show you the algorithm and we will discuss the integrities of this algorithm in the next lecture but I just wanted you to understand the motivation for a better search algorithm. Do you remember the root finding problem that we looked at? We said that if we know a certain value at which the function let's say is negative and some other value at which the function is positive then assuming that there is a root between the two we will call this low, we will call this high and we will arbitrarily find the midpoint of this. So mid is nothing but low plus high divided by 2. You agree that this was the logic and what did we achieve by doing this? We simply reduce the space for search from this entire region either to this region or to this region. Our normal search that we examined in the two versions of the program amounts to the following. It amounts to saying that I start with low and I keep examining every point on this line, each point and I try to find out if the value of the function is close to zero at any one of those point. So if I have n points I am examining the value of the function at each one of those n points. That is what will cause n times the computations. However if I use this method where I do a bisection of the range and then try to calculate the number of computations I have to do will reduce drastically. This was a real line and therefore there is no notion of actual number of points here. They could be 1000, 5000, 10,000 it entirely depends on me. But imagine now that this entire thing is an array and I have real numbers here 1001, 1002, 1003, etc. And these positions are 0, 0, 0, 0, 1, 2 and so on up to say 7. If I take this to be the equivalent then I can actually see a tremendous relationship between the binary search method of finding roots of an equation and searching an array more efficiently where the contents are arranged in increasing order. If the contents are arranged in increasing order then instead of examining 0th element, 1st element, 2nd element which is equivalent of examining function value at each of these points I may choose to adopt the strategy of finding out the midpoint. In terms of number of elements of this array if total elements are 8 then the midpoint is 7 plus 0. In terms of the index 0th element is first element, 7th element is the earth element. So between 0 and 7 what is the midpoint? 7 plus 0 by 2 which is 7 by 2 which is not 3.5 because it is an integer element that I am looking at. So after truncation I will get the third element which where I will examine here this element. So what I am doing I am suddenly looking at this value. Now if the given role number is larger than this role then I have to search on the right hand side. If the given role number is smaller than this number I have to search on the left hand side. Consequently for the subsequent iteration I will be searching either here or I will be searching here just as in the bisection method I would have searched for the root either in this half or in this half. Effectively I am reducing the size of the search by 2 and this makes for a very powerful algorithm ok. We will continue this discussion but when the slides are uploaded I would request you to look at this slide which gives the algorithm for binary search. I will like you all to read this algorithm and come prepared to understand it to discuss it better tomorrow. Thank you.