 Hello, I would like to welcome you. In today's class, we will discuss insertion sort which is basically sorting using a sorted list as an abstract data type or a sorted list implementation of the priority queue. So, the idea here is given an input sequence S let us say 4, 5, 1, 6, 2. You create an auxiliary data structure a priority queue P which is initially empty. And the idea is to scan the list S and insert elements in the order in which you scan into P. So, this makes P 4. And you add the next element 5, you ensure that P has an increasing order of elements. So, which means you want to append 5 to the list, you will get P equals 4 and then 5. And then for the next element, obviously, you will need to prepend P with 1, that is 1, 4, 5 and so on. So, as you can see the scanning of the list will be order in. But the insertion into P can in the worst case take n square time summed up across all the insertions. So, for the first insertion, you will have incur a complexity of 1. For the second one, which is 5, you need to scan 2. In fact, you will have to incur 1, 2, 3 up to n irrespective of how well arranged the elements in S are. So, this is the order n square insertion time. As far as deletion is concerned, all you need to do is pick elements from P starting with the first element and pick the minimum element and insert them into S. Now, what will this mean? Well, this will mean just a linear scan because P is already sorted. So, you have 1, 2, 4, 5, 6 and all you need to do is push them in the same order into S. So, this is order n whereas, insertion was order n square. Overall, the complexity is order n square because it is dictated by the more expensive insert. Now, this naive implementation of insertion sort should ring some bells. You should probably consider getting the sorted content of P straight away into S without having to populate a separate P. Why? Because you will never notice that in the process of obtaining a sorted S, you have actually first created a sorted P. So, that gets us to the in place version of insertion sort and let us understand how this functions. So, as explained earlier, you will still have S which is 4, 5, 1, 2, 3. The output is to get the same elements in S but sorted in increasing order. So, what you do is as follows. You maintain the key that is to be need to be inserted and of course, you get these keys in increasing order of the indices. So, you start with key equals 4, i is 1. The goal is to insert 4 into a P. Now, what is this P? For us the P will correspond to a prefix of S. So, this prefix will basically be S elements from S ranging from indices 1 to i minus 1. So, your initial index was i, i was initially here at 4, i equals 1. So, initially the prefix P which corresponds to S 1 to i minus 1 will be empty. But going forward for i equals 2, this will be just 4. So, for the time being we focus on the key at 4, at 5. So, P equals 4 and K equals 5. This is again when i equals 2. So, what you do is set the index j equals i minus 1. The idea is to scan P starting from the right most element. So, as long as you have not hit the left most index of S or P basically. So, while j greater than 0 and till a point where you find that the jth element of S turns out to be greater than the key, you basically keep shifting the elements of S of P. Why are we doing this? Let us understand with respect to the example. This is saying shift the elements of P to the right to make space for K until. So, you do this until you find that S j is already less than equal to K. So, what happens in this case? Again let us see. So, for j equals 1, i equals 2, you are only interested in the prefix ending at 4 and you find that 5 is already greater than 4 greater than equal to 4. So, in this case what happens is as follows. You will need no shift. So, the array S after this iteration will remain 4, 5, 1, 2, 3. But for the next iteration, you will shift your focus from the i to be 1. Now, when you again try and insert i into the prefix P, preceding i, you find that i is less than 5, i is less than 4. So, what you will have to do is accommodate this i by right shifting the prefix members of S. So, this will need a following transformation. This will mean S will need to make space for 1, 4 and 5 shifted and then place 1 will be occupied by S and that is exactly what happens when you set S j plus 1 equals K, 2 and 3 are unaffected. So, this specific insertion is happening here, S j plus 1 equal to K. You stop when you have no element to the left that is greater than the element K that is being inserted. And in this case, there is no such element at all. So, the only place you can insert 1 is here. So, I hope you understand the functioning of this specific algorithm. Is this correct? What is its complexity? Let us discuss that next. So, correctness first and here is a claim for correctness. We discussed in terms of a loop invariant. At start of each iteration of the ith repeat loop, the sub array S ranging from i equals 1 to i minus 1 consists of elements of the old S 1 to i minus 1, but sorted in ascending order. This is what mimics the priority queue. So, what are we doing here? So, let us understand quickly what the correctness of this invariant at initialization. So, initialization, yes indeed this holds because we initialize with i equals 1 which means there is an empty list of prefix prefixing up to i minus 1. Initialization holds maintenance. So, if this invariant holds before the ith iteration or rather the i plus 1 iteration, will it hold at the i plus 1 iteration? The answer is yes. Please note that the i plus 1 iteration is only going to compare the key at the i plus 1th position with all elements preceding it and we are already assuming that the preceding elements are sorted. So, you scan from the right to the left until you find an element in S 1 to i minus 1 that is actually less than the element less than equal to the element or the key that is being inserted. And because you stop at that position and because you know from the previous iteration that elements preceding the element that was found to be less than equal to the key are already sorted you can stop and in fact this also ensures that the loop invariant is maintained. So, the maintenance holds since S from 1 to j is sorted this is referring to the the algorithm shown in the previous slide and we already found that S j was actually less than or equal to k and since S j plus 1 is going to be containing the key k and the elements after S j plus 1 that is S j plus 2 to i are already sorted and shifted and they were all found to be greater than k. So, this ensures that maintenance holds the third property is termination again this is very easy to verify typically is a maintenance property that requires some work. Analysis we going to look at the number of times each step of this algorithm is going to be involved. So, we have an initialization order 1 complexity you are going to iterate over all keys k except the last one actually I mean you. So, the complexity can be treated as n minus 1 the iterations require n minus 1 computations of k equals S i. Thereafter for each i you might have a variable number of shifts. So, what you see here is is a variable number of shifts by and specifically by shifts you mean right shifts until I repeat until you find an S j which was less than equal to k. So, this variable number of right shifts we refer to by W i of course, there is one more additional step which is inserting k into S j plus 1. So, this gives us W i minus 1 repeated n times and finally, once you have actually substituted S j plus 1 equals k doing this n minus 1 times we can go ahead and sum up all these individual counts. So, complexity given by this expression involving various constants the only thing that needs to be elaborated a bit upon is what happens to these summations in the worst case what happens to these summations in the best case everything else seems to be order n. So, best case running time what will be the best possible scenario for W i's which is is there a way that you can avoid shifting all together can the W i's be basically 1 and it turns out that if your S was already sorted. So, let us consider S equals 1 2 3 4. So, it turns out that when you try to insert 2 or 3 or 4 you end with just one comparison. So, as a result the best case running time is when the W i's are 1 for all values of i as a result these terms do not contribute anything at all there is contribution from this term what is again W i so just n times c 6. So, you will have n times c 6 the rest is as was discussed earlier this all turns out to be just order n how about worst case running time. So, consider a list sorted in descending values. So, 5 4 2 3 1 when you are inserting a 4 you will need to go all the way to the left in fact hit the left most wall and put a 4 there. So, for any of the elements whether it is 4 or whether it is 2 you need to go all the way to the left because you know you are dealing new smallest element. So, in all these cases the W i for i equals 2 will be 2 then W i for i equals 3 will be 3 and so on. So, W i will basically be i. So, making the required substitution you find that you will have a n into n minus 1 by 2 because you are dealing with summation i minus 1 i equals 2 to n likewise for the factor involving c 5 and so on. So, overall this is order n square a question is is there something in between what about the average case. So, discuss the average case we will need to understand when exactly does insertion sort do a swap. So, the most critical component of the insertion sort algorithm is a swap and what is a swap it is basically saying S j plus 1 is S j followed by S j plus 1 equals k. So, what are we doing we are saying that 4 3 2. So, if I had to insert 3 let us say the list was 1 2 3 and I was trying to insert a 3 here how far will I need to go. So, for every distance every element i cross for example, I cross 4 I am basically shifting 4 to the right. So, I have I need to shift or rather do a swap corresponding to every inversion what inversion is basically an i index less than j such that S i is greater than j. So, 4 happen to be greater than 3 this inversion led to a swap, but the swap is very indirect you attain the swap by shifting and then substituting a series of shifts and a substitution. So, it is important to note that swap is realized through a series a substitution and this occurs for every inversion for every pair of indices i j i less than j such that S i is greater than S j. We will see that there are other algorithms which are based on comparison which do not have to do a swap for every inversion, but unfortunately that is the case for insertion sort. So, the number of operations is therefore, the time required to scan each element plus the number of inversions what is the average number of inversions. So, let us say i is the number of inversions then the time complexity is order n plus i we will try and come up with an expression for i in the average case. We already know that for worst case the number of inversions i is order n square for best case number of inversions is 0. The claim is that average number of inversions in the list of n distinct elements is n into n minus 1 divided by 4. How did we come up with this number? So, here is some analysis. So, we will try and factor out redundancies in counting by clubbing together an array and its reverse. So, factoring out redundancies in counting by clubbing together s and its reverse. Let us see how that helps us. So, pair of elements e 1 and e 2 will be inverted in exactly one of s and s r this is obvious. So, if e 1 followed by e 2 is the correct order desired order it will reflect in s and the reverse will reflect in s r or vice versa. So, the total number of such pairs inverted either one of s or s r that is the advantage of clubbing we already see is n into n minus 1 by 2 basically this is n choose 2 because there are n choose 2 such pairs. So, they have to be inverted in one of them. So, what we do is we will consider the total number of such pairs inverted in one of them and therefore, the total number of inversions across all permutations is this n choose 2 factor multiplied by the number of such sets of sequences. So, this is number of permutations, but factoring out reverse each permutation. So, we do not want to count the permutation and its reverse separately that is why an n factorial by 2. So, what will be the average case? Well, the average will just be the ratio. So, take the total number this is the total number of inversions and consider the total number of permutations. So, this ratio turns out to be n into n minus 1 by 4. So, effectively all we are saying is the number of pairs inverted in one of s or s fact s r is n into n minus 1 by 2 and we know that this contributes to half to s r and half to s. So, therefore, half of this should actually be the average number of inversions that is an easier way of rationalizing. So, the average time therefore, the average number of swaps therefore, is exactly the average number of inversions this happens to be order n square and this is true about any swap based algorithm that basically removes one inversion in a single step. So, here is a summary the best case running time insertion sort is order n worst case is order n square average again turns out to be order n square and we are not too happy about this. Can we develop an alternative that gives better average and worst case running times? In other words, can we avoid so many swaps or rather a swap corresponding to every inversion. So, note that we used very naive property here that we wanted the prefix to be always sorted. Can we relax that condition and avoid swaps for corresponding to every inversion and in fact, implicitly attain restructuring of the array through something like a min element preceding all the other elements because that is what we really want we want that a min or a max element precedes other elements and the ability to deal with all these min or max elements at a later stage. So, that is exactly what we will discuss next it is the Heap sort algorithm and that is based on the heap data structure. Thank you very much.