 All right. Hey everybody, Mr. Gibson here with your next lesson in cryptography. Today, we're going to be starting down the path on trying to figure out how can we crack messages that were enciphered using a polyalphabetic method. And the first step along the way is going to be recognizing which ciphertext that we've encountered were actually enciphered using a polyalphabetic method. We're going to have to figure out, should we use these new techniques or can we rely on our old techniques? And if all we have is ciphertext, it can be sometimes hard to tell. So we're going to look at one specific statistical measure that can help us make that decision about which techniques to employ. The technique that we're going to look at today is called the index of coincidence, and it was developed by this person here, William F. Friedman. He was a U.S. Army cryptographer. He ran the Army's research division of the Signal Intelligence Service, so it kind of intercepting enemy signals or messages, they call them signals in the military. And then after that became the chief cryptologist for the NSA. And one thing that was just revealed somewhat recently is that Friedman initiated on the behalf of the NSA a secret agreement with a company called Crypto AG, which was a Swiss cryptography company, where the NSA and then the CIA had a partial ownership of that company. And for many, many years, I believe through the late 90s, they were selling cryptography equipment like cell phones, fax machines, things like that to governments around the world that had these secret back doors. So basically a secret key that the U.S. government could use to intercept and read those messages. And it was only just declassified that that happened. I think it was just late 2019 or early 2020 that that became known. So really interesting stuff. And it all began with this guy right here. So let's see what he did. Again, the goal is figuring out if we have a distribution of cipher text, how can we determine if it's mono alphabetic, meaning encrypted using something like Caesar affine, you know, things like that. Or is it poly alphabetic, vision air, auto key, trithymia is one of those ciphers. We've learned that by looking at the two distributions, it's pretty easy to pick out which one is which the mono alphabetic on the left characterized by high peaks and low valleys, not many things in between. Whereas poly alphabetic distributions are much smoother. You know, we might call them flat for poly alphabetic, whereas mono alphabetic are rough. We can see that there's not as many peaks, not as many valleys, everything is kind of lumping towards the in between, maybe four, four ish percent seems to be what things are trending towards. But it would be great if we had a single measure that could help us decide which of these two categories are cipher text is more likely to fall into instead of having to rely on a visualization. And that's what this method is going to give us. It's going to boil down these distributions to a single number that'll help us pick which ones which. So to get us started, we've generated two distributions to fictitious ones. Each of these distributions have 1000 characters represented in the counts. So on our mono alphabetic, we have 73 A's, nine B's, 30 C's and so on. And then our poly alphabetic cipher, we can see this was a really good one because the distributions are almost perfectly flat. Just a couple of characters have 39 where the rest have 38. And what we're going to do is we're going to calculate the frequency of each single character in the cipher text. So we can think about like running our character frequency function on the message we've intercepted. We get the following frequencies point 073.009 and so on. We're going to use the notation where m sub a means the probability or got the probability that you reached into that cipher text, you picked out a single letter and it was an a m sub B means the probability that you've reached into the cipher text, you picked out one letter and it was a B and so on. And it shouldn't be a surprise that if we were to sum all of those probabilities that you would get a one or 100% chance because what that's really asking you is like, what's the probability that you pick one a or one B or one C all the way down the line A through Z. It's essentially asking if you pick one letter, what's the probability that that one letter is an A, a B, a C, a D all the way down to Z. Basically asking you what's the probability that you picked any letter. So of course, it's going to be 100%. If you do the same thing with our polyathletic cipher, we get the same results, the individual character probabilities are going to change. But the end result is the same, that it's equivalent to asking, what's the probability you pick a single letter? It's one of our 26 100%. So this is not that interesting. But what becomes interesting is what if we don't ask what's the probability of picking one letter, but two letters that are the same, and we're going to see that this is where things differ. The difference between a rough distribution and a smooth distribution will actually change the result about what we get. Let's take a look at the mono alphabetic cipher first. So for a mono alphabetic cipher, we're going to get the following calculations just to kind of get a start. So what's the probability I picked two letters out of this distribution and that they're both a's? Well, that number is going to be 73 over 1000. So that's the probability I reach in the first letter I pick out is an a because there's 73 of them in 1000 letters. And now that I've already pulled that one out when I go back to do the next one, there's only 72 to pick from out of 999 characters remaining. And I want to do both of those things. So we're going to multiply those probabilities together to get our roughly five and a quarter percent. We do the same operation with the bees and we get a much smaller number, very small fraction of a percentage there. And we could do that for all of the characters using somewhat similar method, where we would take n sub i, meaning the number of a given character divided by n sub text, meaning the number of characters in the text, and then we'll multiply by one less of each. And when your messages are pretty long, there's a nice little shortcut we could do. You could just take the number of the characters that you're concerned about, divide it by the number of characters in the text and squaring it, as we saw previously. The numerator and the denominators don't change very much when the messages are very long. And those two fractions are going to be roughly equivalent. So that's what we use the approximately equal sign there. Those two values are not going to be the same as if you did it the long way versus taking that shortcut, but they're going to be pretty darn close and for our purposes are probably close enough. So when we use that shortcut here, we square each of the probabilities for any given character and sum those up a to z. We don't get one. We get a much smaller number, we get about 6.6% or 0.066, which is essentially the probability of picking two a's or two B's or two C's and so on when you reach in and pull two letters out of the cipher text. There's a lot of other things that could happen when you pick two letters, you don't have to pick two a's, B's, C's, or so on. You could have picked one a and one C, or you could have picked a J and a K. There's a lot, a lot of ways you could reach in and pick two letters and have them not be the same. But what the statistical measure that we're using, the index of coincidence, is the probability that the two outcomes coincide or are the same. We don't particularly care which outcome of the 26 it was, two a's or two B's or two Q's or whatever, but we're just kind of totaling up the sum of those probabilities to get this number. When we do the same process for a polyalphabetic distribution, we get a different number. We get 0.038 or about a 3.8% chance that if you pick two letters out of that type of cipher text, that the two letters that you've just chosen are the same letter. Again, not concerning ourselves which letter it happens to be, but any of the 26, that's our probability. And that might be surprising or maybe not not so intuitive, that a rough versus a smooth distribution would give you this difference in the squared outcome. So maybe let's think about this using a smaller distribution instead of 26 letters with all these numbers. Let's just think about flipping a coin for a second. If we have a fair coin, which maybe we could think about as our even or smooth distribution, the probability of a heads coming out is 0.5 or 50% and the probability of a tails is also 0.5. And if I asked you what's the probability of flipping the coin one time and getting a heads or a tails, you could sum those individual probabilities of 0.05 plus 0.05 gives us our one. Well, in our alphabet scenario, we had 26 outcomes. And if I just asked you what's the probability of any 26 happening, you'd say 100%. So let's go back to that original slide. But going back to our coins here, I said, if I asked you what's the probability that you flip the coin twice and you get the same result, you're going to calculate that slightly differently. I'm asking you what's the probability of having two heads as an outcome or two tails as an outcome. So we could calculate the probability of picking two heads in a row that'd be 0.5 times 0.5 to get our 0.025. And then we would add to that the probability of having two tails in a row, which you calculate the same way. And if we're concerned about either one of those things happening, we would add those probabilities to get a total of 0.50 or 50% chance that you get two heads or two tails when you flip the coin twice. If we look at the same outcome with an unfair coin, I've never seen one of these in the wild, but it's a wonderful thought experiment, we can think about our unfair coin is a rough distribution where different outcomes have different probabilities. In this situation, the probability of heads plus probability of tails is still one that has not changed. It has to be one of those two outcomes. But when we do look at what happens if we flip it twice, the probability of two heads is no longer a quarter, it's actually 0.49. That's much more likely to occur in the rough distribution. And the probability of two tails is much lower. However, the sum of those two things has increased, meaning there's a 58% chance that you're going to get either two heads or two tails when you flip the coin twice compared to the 50% chance with the fair coin. Both of these numbers we could think about as the index of coincidence for these distributions. And just like we saw with monoalphabetic rough and polyalphabetic smooth, in our coin scenario, the rough distributions have a higher index of coincidence than a smooth distribution. And we can use that to our advantage. This is a kind of a universal principle for any distribution that the rough distribution will have a higher index of coincidence than a smooth distribution. And going back to our examples previously, for English language text, a monoalphabetic will always be approximately 0.066 or 0.068 roughly, whereas a really good polyalphabetic will be roughly 0.038 or 0.04 ish. And we can use that here an example. Here's a cipher text that we intercepted. And if I don't know if this was done with affine or visionare or any which one of my ciphers that we are familiar with, I can calculate its index of coincidence. And when we do that for this cipher text, we get an index of coincidence that's roughly 0.042. And since that's much closer than what we saw when we calculated the same score for our kind of ideal polyalphabetic distribution, as compared to our monoalphabetic to monoalphabetic distribution, it would make much more sense for this cipher text to be more likely polyalphabetic than monoalphabetic. If instead we had calculated the score and it was 0.062 or 0.063, then I would probably think this was a monoalphabetic. But that's not the case here. So there we have it. We've got a nice single number that we can calculate relatively easily to determine what the likelihood is of a particular cipher text either belonging to a monoalphabetic or a polyalphabetic cipher. Thanks for watching. We'll catch you on the next one.