 Welcome back. So, last class we defined some important concepts. We defined sigma algebra of events and we define measures in general and we also define probability measures as special cases of measures. So, the sigma algebra of events essentially consists of those certain subsets which are closed under complementation countable union and countable intersections and it must contain the null set. And elements of the sigma algebra this is elements of the sigma algebra f are known as events and then once you have the sigma algebra we defined measures on this measurable space omega f and we said measure satisfy two properties. One is that the measure of the null set must be 0 second is that countable additivity of measures. If you have disjoint sets in f and countable. So, the measure of the countable union must be equal to sum of the individual measures. Now, and then we also said that the measure is a probability measure if the measure of omega itself is 1. So, probability measures of three properties p of null set is equal to 0 p of omega equal to 1 and this and probability measures satisfy countable additivity of disjoint events. Today, we will derive using these three axioms of probability that I just mentioned we will derive some fundamental properties of probability measures. So, the first property. So, here as usual omega f p is a probability space and all probability space satisfy the following properties all probability measures follow satisfy the following properties p of a complement is equal to 1 minus p of a. For all events probability of a complement is 1 minus probability of a. How does this follow how do you prove this you have to use only the axioms. Remember a and a complement are disjoint a union a complement is omega. So, omega is equal to a union a complement. So, probability of omega is equal to 1 is equal to probability of a complement union a probability of a y because countable additivity of see the thing is there is let me just put down just to make this clear. So, there is also this is the simplest property this generalizes to what is known as finite additivity it is actually fairly trivial if you have countable additivity it is very trivially follows that you have a finite additivity. So, if a 1 a 2 dot dot dot a n are events and disjoint suppose they are disjoint then probability of union i equals 1 through n a i is equal to sum over probability of a i i equals 1 through n. So, actually this is a special case of this is not it. So, you have only a and a complement and it so happens that a union a complement is omega. So, if you apply this property you will get this very trivially I put this down first because it is so simple, but actually is a special case of if this follows from this which is finite additivity of probability measures. Now, how do you prove this we know that probability measures are countably additive, but you have to prove now that they are finitely additive it is actually quite trivial, but you have to do it properly. So, after a n you choose all those events as null sets. So, then you will have union. So, after all these guess you there are only null sets. So, you can prove that union i equal to 1 to infinity a i is equal to union i equal to 1 to n a i because a n plus 1 a n plus 2 they are all null sets and then what you do here on the right hand side. So, you will have sum over i equal to 1 to infinity probability of a i. So, the first n terms are whatever they are, but after that you will have probability of null set is 0, but now there is a problem. So, you are so essentially what you have here after this is the ghost term. So, to speak is i equals n plus 1 to infinity probability of null that is what is on the if you I mean. So, I guess what I am saying is if you have i equal to 1 to infinity union here and you put n plus a n plus 1 onwards as infinity null sets you have this remaining. Now, how do you prove that this is 0 you are summing an infinite number of zeros. So, after all what is the summation defined as the summation is defined as limit m tending to infinity i equals n plus 1 to m of whatever it is, but all these finite summations are 0. So, limit of the sequence which is 0 always is 0. So, this should be written as a limit. So, in order to do it properly do it fully rigorously you have to write this out as a limit then prove it. So, you should not get confuse saying over this is infinity this is 0. So, 0 times infinity I do not know what to do that is not right. So, you have to do it carefully. So, I mean. So, let me get rid of this. So, this is the actual result I just spoke out how you prove this right I suggest you actually write it down for your own clarity. So, these two are very easy right it is just finite derivative t and disjointness any questions. So, if you have two events a and b both in the sigma algebra and if one of them is contained in the other then the probability of a is less than or equal to the probability of b. How do you prove that seems perfectly reasonable right. So, you can write b as a union b minus a correct. Now, a and b minus a are disjoint. So, you can use finite additivity right. So, you can use finite additivity to write probability of b is equal to probability of a plus probability of b minus a right. Now, this is non negative why probabilities are non negative right they go from 0 to 1. So, that guy is non negative. So, you have that this is bigger than or equal to that. So, we already said that probability is a map from f to 0 1 right close to interval 0 1 right. So, the it only takes values the range 0 1 1 right. So, that is a part of the axiom right. So, this is non negative and therefore we have a and b belong to the sigma algebra you will prove it in your homework all right. After all what is I mean come to think of it what is b minus a right it is b intersection a complement. So, there you have it right any questions. So, these are very simple properties and the related property the probability of again all this I am not going write this again and again. So, maybe once I will write it a is an f b is an f then probability of a union b is equal to probability of a plus probability of b minus probability of a intersection b. So, this is saying that probability of a union b you can add the two probabilities and subtract the probability of the intersection and of course, a intersection b is also an element of f right. How do you prove this? Yeah you just do it it is fairly simple just try and do it as an exercise proof exercise fairly simple right to write out I think you essentially are over counting a intersection b right. So, you have to subtract that out proof is an exercise. So, a generalization. So, more generally if a 1 a 2 dot a n belong to f then probability of union i equals 1 through n a i is equal to sum over probability of a i minus sum over i less than j probability of a i intersection a j plus sum over i less than j less than k probability of a i intersection a j k dot dot dot minus 1 to the minus 1 to the n minus 1 n or n minus 1 n minus 1 I think n minus 1 probability of intersection. So, this generalizes to more events right. So, what we are talking about now is see when you had when you had disjoint events of course, we said the probability of a union b is probability of a plus probability of b. Now, we are saying that if they are not necessarily disjoint then the union you have to take out some you are over if you just add their probabilities you are over counting. So, you took out the intersection b. So, here this is similar. So, you are adding all the probabilities taking out all the pair wise intersections now then you take out too much. So, you add back three way intersections and so on. So, this rule is called the inclusion exclusion rule right it has a name inclusion exclusion rule or inclusion exclusion formula because you keep including more excluding some right. So, if you had probability of a union b union c you will have some of the three probabilities subtract three choose to intersection terms and add the three way intersections right. So, that is how you would do it how will you prove something like this induction right because it is a statement on the natural numbers right this is for any n right this is only a finite number of events right. So, you proved it you proved it for the case n equal to 2 by elementary methods assume that this is true for n is equal to k right and then prove it for k plus 1 right. So, it is fairly easy to prove by induction using induction on n, but it is not a pretty proof it is some ugly proof right you have to write out the induction. Induction hypothesis the basic case is easy n equal to 2 you have it write out the base case for k and you have to work through all the algebra to prove this right it is just may see. There is a much simpler proof using indicator random variables which we will study much later this is how you should prove it now to appreciate how much easier it is to prove it using the better you will study half way down the course. So, far this is all very elementary properties of probability measures I assume more most of you know this already right this is all stuff you have seen I think. So, I have put down some 4 properties that remaining ones are more advanced. So, these are very simple what are what is going to come is more advance you have to be slightly careful in proving this property number 5. If a 1 a 2 a 3 dot dot dot r and f then yes. So, this is a very important property this says that. So, you have a countable set of events all right a 1 a 2 dot dot dot it is a countable collection of events and of course, the countable union is also an f right because f is a sigma algebra right. So, this is well defined. So, we are saying that the probability of a countable infinite union is the limit of the probability of and even this countable this finite union right. So, what is on the left is the probability of a this is the probability of a countable infinite union and you have here a finite union and a limit right and we are saying that these two are equal. So, it looks as though there is nothing to prove right looks like. So, what you are doing is like you are in very informal terms you are just taking the limit inside right. So, in that sense it looks like. So, this property is known as continuity of probabilities continuity of probability measure that is what this property is called for obvious reasons right this looks as though you are simply. So, you had a if you have a continuous function you take the limit inside the function right this looks as though you can just take the limit inside right that is why this is called the continuity of probability measures. This is a highly non trivial statement this is by no means obvious actually it takes some serious thinking to even understand what this property means. Let me explain. So, first of all I told you that this kind of a union countable infinite union should not be thought of as a sequential operation it is not like unioning it is not like your unioning a 1 with you a 2 then a 3 then a 4 and so on that is not the way to look at this right this is there in the I think I made a remark in the set theory notes right this should what is this set of all omegas in at least one of the a i's. So, please do not look at this as a some kind of a limiting version of some finite union right after all right I mean what is the limit of a set there is no such thing as we know limit of numbers right. So, this is not like you cannot say oh this set is like the limit of that union or something like that there is no meaning to that right that is the first clarification I want to give right. So, this is the set of all omegas in at least one of the a i's and this is the set of all omegas in a 1 through a m at least one of the a 1 through a m's and. So, that is the probability. So, you can look at. So, you can look at let us say this as a function of this will be some call this some p m right this is these are numbers that depend on m right you are uniting the first m right. So, this p m's are. So, I claim that this p m's are monotonically increasing in m that is clear why a m is contained in. So, well this union is contained in the m plus 1 union right. So, you can say that this p m's are monotonically increasing sequence right the limit of a monotonically increasing sequence has to exist right it has to converge. So, this right hand limit exists as some real number right and left hand of course, is well defined right it is a probability of this countable union countable infinite union which we know exist because this guy is a element of the sigma algebra. So, this must be well defined. So, this property is saying that this number which is well defined is equal to some other limit which is also well defined right and, but it is by no means obvious why they are equal in particular it is not like you are taking the limit inside it looks like it, but it is not that yes it well no. So, this is correct. So, this is a monotonically increasing sequence. So, it has to converge right it has to converge to something right has to converge to either a real number or plus infinity right. So, this p m exists this p m is well defined right I am saying that this limit is equal to another entity which is well defined right now it is another probability right. So, in particular see if you look at this p m's there after all bounded above by 1 right. So, in this in this particular case they are this probabilities right. So, it will be bounded above and. So, it has to converge it cannot converge to plus infinity right. So, if these were ordinary measures this is still true except that this limit may not be finite it might be plus infinity right correct. So, is the statement at least clear the proof is non trivial, but the statement clear. So, this do not take this lightly this is not a simple is not a simple assertion right this is by no means an obvious property at all. .No, no, no have I said that no then I do not need to be right any other questions let us prove this. So, let us proof I am trying to prove this now proof define b let us do the define b 1 is equal to a 1 b 2 is equal to a 2 minus a 1 and. So, that b so that b n is equal to a n minus. So, a n minus union a i i equals 1 through n minus 1 and so on right. So, I am defining. So, I have this a i's this is a countable collection of events. So, I am defining another countable collection of events namely b i's. So, I do that. So, I keep a 1 b 1 as a 1 then b 2 I define as a 2 minus a 1 b 3 will be a 3 minus a 1 union a 2 right and so on right this is another countable sequence of events. Now, what can be shown. So, this is a very important step this is a very important step in the construction. So, what you can show is that claim b i's are this joint i b i intersection b j equal to null for all i comma j greater than or equal to 1. So, it looks intuitively obvious right except you have to prove it that is a home work. Proving this claim is a let me call this claim 1 this claim 1 proving claim 1 is a home work it is simple enough, but you have to be careful you have to do it properly it is not difficult. So, you start of the assuming without loss of generality is that j is bigger than i give you a hint assume j is bigger than i and you can prove it by contradiction assume that there is a common element to i and j and arrive at a contradiction. This is 1 claim the other claim is the following I am claiming that union i equals 1 to infinity well equal to 1 to infinity. So, I want to claim that union i equals 1 to infinity a i equals union i equals 1 to infinity b i this also you will prove. So, how will you prove that these 2 sets are equal. So, if whenever you have to prove 2 sets are equal you have to say that any element in this union is also in this union and vice versa right. So, if any omega is an element of at least 1 of the a i's it must be an element of at least 1 of the b i's and vice versa right you prove that you are done and. So, I think you can also prove and you can prove that the finite unions are also that is also true. So, what is this union this is the union of events in a 1 through a this is the union of elements in a 1 through a m and similarly for this you have to prove that this is contained in that and that is contained. So, these are all simple exercises you will do there is nothing very great about this, but you these are like lemmas in proving this result claims. Now, we can prove it I think now we are ready because I can write. So, this is what I want on the left hand side I have probability of that guy right now that is equal to probability of this union. Now, why is this union much better than that these guys are disjoint b i's are disjoint a's may be very complicated they may all be intersecting in very complicated ways, but b i's are disjoint. So, I can use what property sorry countability right. So, let us do this now probability of union i equals 1 through infinity a i is equal to probability of union i equals 1 through infinity b i that is because these 2 are equal and this will be equal to sum over i equals 1 through infinity probability of b i correct everybody this is because of b i's are disjoint and countable additivity of probabilities right. So, now. So, I am now in a good situation where I have. So, I had this probability of this countable infinite union now I wrote it as a summation. Now, this infinite summations are in fact just limits of finite summations see this countable infinite unions are not limits of finite unions that is not true, but summations this infinite summation is limit of a finite summation right. So, this I can write as limit m going to infinity i equals 1 through m p b i correct everybody happy this is by definition of this summation now again what will I do this I can use finite additivity right. So, now I can write this as limit see after all I want to maintain limit on the right. So, I brought out a limit on the right. So, I should maintain that limit right I should not get rid of it. So, I should maintain that limit and then I will write this as probability union i equals 1 to m b i which is equal to yeah. So, because of this right which is equal to limit m tending to infinity probability of union i equals 1 through m a i. So, this is a fairly non trivial proof right. So, you have to be very careful. So, this is by claim to this equality is by claim to this equality is by finite additivity right. This is by definition by definition of the infinite and this equality is by axiom right axiom of countable this is by countable additivity that is by countable additivity and that is by claim 1 claim 2. So, this is by countable additivity and claim 1 right. So, this is something we will use repeatedly this continuity of probability measures is this is a non trivial property. But, from now on we will be like happily taking the limit inside it is as though we are taking the limit inside, but there is some serious proof involved correct any questions. So, I will now state 2 more properties which are in fact corollaries of this continuity of probability theorem, but if they are also important in their own right. So, I am not going to I am going to state them explicitly this is property number 6 right. If a 1 a 2 or a dot nested increasing events i e a 1 contained in a 2 is contained in a 3 is contained in and so on. Then probability of union i equals 1 through infinity a i is equal to limit m tending to infinity probability of a m. See this nested increasing even means so a 1 then a 2 is bigger than that then a 3 is bigger than it contains a 2 and so on right. So, there is this nested sequence of containments if you have this situation then this is actually nothing but a very easy corollary to the previous result. So, in the situation we have union i equal to 1 through m a i will simply be a m because a m is the biggest guy right and this other guys are smaller they are all contained in them right fine. So, this is a very trivial corollary. So, limit m tending to infinity this term simply becomes probability of a m right this finite union is equal to a m right very easy corollary any questions and similarly. So, for intersection you get a similar result which is probably most useful. So, I will also state that explicitly property number 7 if a 1 a 2 dot dot dot nested decreasing i e a 1 contains a 2 contains a 3 dot dot dot. So, these guys are now nesting in right. So, a n is contain in a n plus 1 I will say that a n plus 1 is contained in a n right. So, that is this then probability intersection i equals 1 through infinity a i equals any guesses limit m tending to infinity probability of a m. How do you prove that intersection will be sorry. So, I want to prove this result how do you prove it section of a 1 through a m will be a m right that much is clear right because of this containment then what actually you can do that to right you can you have to see you have you are dealing with intersection now. So, you have to do what somewhere you have to demorgan right you been going from union to intersection you have to do a demorgan actually do a demorgan on this this result you know you do a demorgan on this what do you get a 1 complement will contain a 2 complement see if a 1 are nested increasing then the complements are nested decreasing right use that property and use demorgan you will get it fine. So, these are actually fairly simple corollaries of the continuity of probabilities actually this form is probably the most useful where you have this decreasing sets right these nested decreasing sets which are contain in with one within another and this is a countably infinite sequence of nested sets right. So, these sets are the colloquially that is called the Russian dolls. So, you have you seen Russian dolls they are actually dolls you open one doll there will be another doll open like you have this bunch of dolls inside one another right this is like that except there is a infinity of them. So, these are these kind of nested decreasing sets are called Russian dolls that is and then you have this property for Russian dolls any questions the next property is property number 8 is an important property called the union bound says that if a 1 a 2 dot dot are events then probability union i equals 1 through infinity a i is less than or equal to sum over the probability of a s. So, this is an important property that says if you have any arbitrary sequence of events then the probability of the union is less than or equal to the sum of the individual probabilities all right. So, they essentially this says that. So, in adding up all these probabilities you are over counting right. So, this says that this will be bigger than this. So, if a is a disjoint of course, we know from the axioms that equality holds, but if a is a not necessarily disjoint then the probability of the union countable union is less than or equal to the sum of the individual probabilities. This is always true, but note that what is on the left hand side is a probability. Whereas, what is on the right hand side may be something bigger than 1 this is just a summation right the summation of non negative numbers this may in fact be infinity or it may be some number bigger than 1. So, this is always true, but it is useful only when the right hand side is less than 1. So, the way prove this is to just again go back to your b i's. So, remember we define b i as a i minus union j equals 1 through i minus 1 a j right and we know from our earlier claims that union of a i's is equal to the union of b i's right. So, I can take the left hand side and say that probability of probability of what is on the left hand side is equal to the probability of union i equals 1 through infinity b i's. But, by construction we also argued that b i's are in fact disjoint right. So, this is a countable union of disjoint events right. So, this will be equal to sum over i equal to 1 to infinity probability of b i. So, this is because of countable additivity of probability measures right b i's are disjoint. Now, notice that b i's are. So, for every i b i is contained in a i right that is because b i is defined as a i and then you remove something right. So, this b i for every i b i is contained in a i right which means that probability of b i is less than or equal to probability of a i for all i for all i which means that sum over i equals 1 through n probability of b i is less than or equal to sum over i equals 1 through n probability of a i for all n correct for all n greater than or equal to 1. Therefore, by now you send limit n tending to infinity you will get that this summation here is less than or equal to the then you will get this infinite summation probability of a i. So, that gives us the proof of the proof of the union bound. Now, so a remark on the converse. So, you know that if a i's are disjoint this equality necessarily holds right the union bound holds with a equality if a i's are disjoint. However, it is not quite true that if equality holds for certain events it is not necessarily the case that a i's are disjoint. So, what you can show is that if equality holds in the union bound then the intersections between a i and a j will have 0 probability measure. It is not quite true that they do not have any elements in common they may have elements in common which have 0 probability measure. So, the converse of the union bound is not completely it is not quite true. So, the union bound is used in several applications. So, we will see an application in the Borel Cantile lemma also in digital communications if you are transmitting a set of bits the probability of let us say a i is the probability of the ith bit going wrong. For example, you bound the probability of making some error at all by adding up the probability of each bit being decoded incorrectly. And if this decoded this each of these probabilities small enough this union bound can actually give you a quite a useful estimate in digital communications and it has several other uses as well. Thanks.