 All right, Mr. Gibson here with the next lesson in cryptography today. We're going to be looking at a method of scoring potential candidates for decryption To see which one of those is most likely to be our plain text That method we're going to use is a statistical measure called a chi-squared score that you might be familiar with if you've taken a course in statistics Let's figure out what it is how to calculate it and see how we can use this score to help us really do a better job Decrypting cipher text for today. We're going to be looking at this cipher text message It has nine hundred and ninety four characters in it and we don't know how it was created Could have been a Caesar cipher it could have been a multiplicative cipher or it could have been an affine cipher any one of our Substitution ciphers that we've covered so far we're going to try and figure out which method was used and then how it was generated so meaning which keys were used to generate the text Let's get started by just plotting or using a bar chart of the characters we have So what we've done here is we've counted the number of a's b's c's and so on and we've plotted them on a bar chart So we've you took all nine hundred ninety four characters And we have a distribution now of those characters and thinking back on an earlier lesson in this unit We can see that the order of our standard distribution of English characters has not been preserved I don't see that a and e close together even if they've been shifted I don't see two things two bars on a row that could have been an h or an i or an n and o and I don't see a triple spike of RST so we know this isn't Caesar Which leads us to believe it's either multiplicative or affine So we're going to figure out which one of those it is and then try and figure out. Well, which were the keys used So what we're going to do is we're going to figure out well since we don't know what the distribution of the plain text should Look like we're going to estimate it So we're going to take the fact that we know that there's nine hundred ninety four characters in the message And we know the standard frequency of each letter So if we think about the letter a if there's nine hundred ninety four characters in the message We know that eight point one six seven of those Percentage of those should be a's so we can run the numbers on that and eight point one six seven percent of Nine hundred ninety four characters gives us eighty one point one seven nine nine eight So we would expect there to be roughly eighty one a is in the message of this length Even though there's no B's in the ciphertext what we would expect there to be in the message are roughly fourteen because again We know roughly that one point four nine two percent of The English language is the letter B So that's where we can get our roughly 14 B's that we would expect to be in this message when it was in the plain text We can go on down the line We'd expect it to be about 27 C's in the plain text about 20 y's in the plain text and just under one Z in the plain text so we've kind of created here what we would assume to be based off of probabilities the distribution of characters in our plain text and Now we're going to try and create a bunch of candidates that we think might be plain text So the way we're going to create candidates is by deciphering the ciphertext Using an algorithm with some keys and I just randomly decided to do affine with a multiplicative key of 15 and an additive key of 7 Counted up all the characters and I created the bar chart on the left And when we look at that compared to the bar chart of what we expected in the plain text on the right We can quickly see that that was probably not the right key that these two bar charts do not look anything like But we could generate this here's another random key pairing doesn't look right Here's another key pairing doesn't look right and you can see that we're really no better off than brute force We're just trying all of these and we're again looking at two distributions as opposed to earlier Just looking for English in the candidate plain text. This is no better than what we've been doing before What we hope we could do is find a way to quantitatively meaning based off of calculations and numbers Find a way to compare the candidate text that we generate by trying to decipher with some key pairings To the expected text based off of probabilities. So we're going to find a way to score this and here's how we're going to do that We're going to start by taking this candidate So we're going to stick with this candidate I created by deciphering the ciphertext with a multiplicative key of seven and additive key of 12 I'm going to count up how many letters are in the candidate text So in this case and this this matches with the bar chart There's 11 a's 58 b's 18 c's and so on and we all the way down in the end we get 60 y's and 1 z And we can look at that and compare it to what we expected to be there and again The expected is based off of probabilities of what we would expect there to be in the plain text for a message of this length And you can see that they're they're pretty different and we can actually quantify that difference and we call that error So if we if we subtract off the expected from what we counted or the actual count in the candidate We can produce a value that measures the error in each category and what I mean my category is letter So for letter a we had an error of negative 70 for letter b We had an error of 44 and so on down the line And you might think well, let's just add up all the errors and figure out What's the total error that we had for our candidate? The problem with that is that no matter what and this is just the way that we are doing this um We add up all the errors and some are positive and some are negative they will always sum to be exactly zero So so that's not going to be helpful to our us to figure out which candidate text is better than the other If the sum of all their errors is always going to be zero no matter what So we're going to have to do a little something with that So in mathematically, we have a few operations that we like to use to make things all positive Because that's the reason why these are summing to zeros. We got some positive and some negatives Let's just find a way to make them all positive In a popular way that statisticians like to do that is to square the error So I'm going to move the bar chart out of the way here to give us some more root So we're going to go down this squared error column And the way we generate those values is by just taking the error in the column to the left and just squaring it So you can see that for a we have a squared error of 49 125.23 for letter b. We have a squared error of 1950 for c 83 and so on Let's think about what we expected. So if we look at letter a And we were expecting there to be 81 and we were off by 70 Is that better or worse than expecting there to be 14 and we're off by 44 I would actually argue that being off by 70 when you're expecting 80 Is actually it should be better than being off by 44 when you only thought there is going to be 14 The like the size of the miss compared to what you are projecting Is actually closer for a being off by 70 And the way that we can quantify that is by taking your squared error and dividing by the expected So that number 60.67 in the first row we got by taking 4925 and dividing it by the 81 So the squared error divided by the expected And this gives us a way to kind of directly compare how badly we we missed and by missed we mean Not create the correct number of letters in our in our candidate Between all the different letters so we can see that we did better on c We're closer to getting the correct number of c's in our candidate Then we were for y's because the the squared normalized squared error of 3.07 is much smaller Than the normalized squared error of 83 And if we were to total all of the normalized squared errors This is the statistic that we were trying to create this score Of 18926.41 is a chi-squared score for this candidate So again the candidate was generated using the affine decipher with a multiplicative key of 7 and an additive key of 12 And you want to think about what what what would be a good value for this score And you think okay Well, what makes a good candidate a good candidate would have very small errors Meaning the letters the number of letters we see in the candidate is close to what statistically we would expect there to be there So small errors would be small squared errors And small squared errors would lead to small normalized squared errors for each letter So if we total up all 26 normalized squared errors We'd want that they'd be really small a really small chi-squared score should indicate something That's very close to what we would expect in a plain old plain text So we could do this for all 312 possible key pairings with the affine cipher This is reminiscent of brute force. We're going to try all of the keys The only difference here is we're not going to be wading through candidate text and trying to find some English in it We're not going to be comparing a bar chart of the candidate compared to the standard distribution of English What we're going to be looking at is just the chi-squared score, which is much more numerical Which means that we can sort it So here's what we did. I took I got our top five candidates here based off of their chi-squared score We did it for all 312. I'm just showing the best five and by best. I mean the smallest chi-squared score So we can see that the top line candidate there with an additive key of four and a multiplicative key of seven When we deciphered using those keys The resulting candidate text had a score of 66.868 So that's the sum of the normalized squared errors for each of the 26 letters. It was only 66 And when in fact when you now decipher that candidate using those two keys we get English And you can see that that that score was far below the next closest of 1312 So there's our there's are going to be a really nice method here is this chi-squared score We can if we can run through all the possible keys for a particular algorithm Use all of those keys to decipher the message Creating candidate messages. We know hopefully one of those is the actual plain text But we're going to get a lot of other things that are just gibberish This method the chi-squared scoring gives us a nice quantitative way to measure How closely our candidates align to what we would expect there to be in the plain text And if we just trust the probability and the statistics When we when we choose the keys that generated the lowest chi-squared score More often than not those keys will result in the correct plain text That is it for today on chi-squared scoring. Thanks for watching and we'll catch you in the next one