 The next step after running, training, and testing the algorithm would be to improve it. And I don't know of any better way to do this than to inspect the results, see where it did well, see what it did, and then more informatively to look at where it failed. So there's an additional script, report.py, that compiles some of this. It generates the structure diagram that we looked at before that showed the architecture. It also looks at these randomly selected specific examples, and it generates a set of positive ones, ones that were correct, and ones that were incorrect. Here we can see this one, it was actually a one, it was predicted to be a one, and you can see the strength of that prediction over here to the right. It didn't predict it to be anything else at all even close. Same with the zero, and the nine, and the four. The five is interesting. It did predict that it was a five, but you can see by the length of that bar that it was a weaker prediction. It's like it's probably a five, I don't have any other guesses, so I'm going to go with five. In the case of the seven on the far right, you can see it strongly resembles a seven, but it looks a little bit like a three, so the neural network says. But seven won out. Now here are some examples of incorrect answers. So starting out, this thing in the upper left, the neural network said it's a three. The actual label was a two, and tellingly, it didn't really have any strong matches for anything. It looked at this, and it's like, I don't know what that is. I don't have a category for something that looks like that. That looks too weird. For the one below that, we can see that it had weak guesses for seven and nine. The nine won out by a little bit, but it didn't really love either of those. And you can tell by looking at it, it's like, yeah, it looks like a seven with a cross through it, but it also looks like a nine where part of it's been erased. It's tough to tell. The algorithm said, well, I'll go with nine. I'm leaning nine. I don't feel great about it. The actual answer was actually seven. Then in the upper center, we see one where the actual label is zero, and the algorithm predicted a six. If you look at the actual predictions, it was really 50-50. It's like, well, this looks a lot like a six, and it looks a lot like a zero. I don't know what I'm going to do. The six is a tiny bit higher, but I'm not at all confident that it's more six than zero. I'm going to go with six. I don't love it. As a human looking at that, I would be hard pressed to tell you whether that's a six or a zero. It looks like an incomplete six. It looks like a sloppy zero. I would not with confidence answer either one. I would say 50-50. Same thing with this one right below it. Here, you can see the algorithm gave very weak predictions for three and five, but it was, again, not at all confident that it should go one way or the other, but mostly it said, I don't think this looks like a number. This doesn't look like any of the things that I've seen and learned. As a human looking at that, I can see, yeah, it looks kind of like a five, but it also looks kind of like a three. Whatever it was, it was a little sloppy up top. I could see it going either way. Maybe even a hint of a six in there. This is not something that I would maybe expect an algorithm to be able to confidently categorize because I can't confidently categorize it. This one in the upper right is an interesting case. The algorithm says, hey, I think that's a seven. Looks very much like a seven. If you stand back and squint the overall shape is definitely very seven-like. The label was three. The algorithm did not think it was a three at all. Now, you and I, we look at this and we can see because of the slight offset of that downward slant, we can infer that to write that with a pen, you would have to stop and bounce and make a little divot in the middle, as we would do with a three, as we would never do with a seven. We can use that subtle bit of intuition that comes from knowing about the world and knowing how this happens to say, okay, I think that's a three. That's something that our neural network doesn't have access to. It has to go off a purely literal visual similarity. In that case, I can totally see how that would come out being a seven rather than a three. Similarly, down here with a five on the lower right, we look at that and I at least think that's clearly a five because I can see where the top hat is slightly disconnected as if it were made with a different stroke. I can see that the way that the bottom part was made, there was a downward stroke, a stop, a right turn, and then a loop, the way a five would be drawn, whereas a six would probably be drawn with a single smooth loop. If you just take a step back and look at the overall shape, that looks very six-like. Here, the algorithm's literalness bites it a little bit and notice that it doesn't make any strong predictions either way. This says whatever it is, it looks kind of different than what I've seen in the past, but it looks a little bit more like a six than anything else I've ever seen, so we're going to go with that.