 Hello everyone, I'm Kevan Kamali. This is the part three of the Deep Learning Tutorial series on convolutional neural networks. So as before, if you want to get familiar with Galaxy Platform, Introduction to Galaxy Analysis is a very good introductory tutorial. As for this specific CNN tutorial, I'm sure you've gone through part one and two of this Deep Learning series on feedforward neural networks and recursive neural networks. There is also an Introduction to Deep Learning, which is useful to review. So the questions we're going to answer in this tutorial is what is a convolutional neural network and what are some applications of CNN? And our objectives are understand the inspiration behind convolutional neural networks and learn the CNN architecture, learn the convolutional operation and its parameters, learn how to create a CNN using Galaxy's Deep Learning tools, and finally solve an image classification problem on MNIST digit classification data set using CNN in Galaxy. So the last two steps are part of the hands on part of this tutorial, which will be done separately. So what's a convolutional neural network or CNN? So there has been an increasing popularity of social media in the past decade or so. And as a result, image and video processing tasks have become very important. Previous neural network architectures, for example, feedforward neural networks, they could not really scale up to the image and video processing tasks in hand. So convolutional neural networks are specially tailored for image and video processing tasks. So this is a feedforward neural network. It has an input layer, an output layer, and a hidden layer. This is a fully connected network in that every node is connected to every other node in the next layer. Every connection has a weight that must be learned by the learning algorithm, the back propagation learning algorithm. So what's the limitation of a feedforward neural network? We're going to illustrate that with an example. Let's suppose we have a 64 pixel by 64 pixel grayscale image. So each grayscale pixel is represented by a value, which is usually between 0 to 255, whereas 0 represents black, 255 represents white. And any value in between represents different shades of gray. So since each grayscale pixel is represented by one value, we say the channel size is one. Channel size could be more. For example, it's 3 for color images. We'll get to that later. So an image is represented by 64 times 64 times 1 values, which is 4096. That's basically we multiply the number of rows by the number of columns by the channel size. So the input layer of a feedforward neural network then would have 4,096 nodes. Now let's assume the first hidden layer has 500 nodes. And our feedforward neural network is dense that is fully connected. So every node in the input layer, and there's 4,096 of them, is connected to all the nodes in the hidden layer, which is 500 of them. So the number of weights that we have to learn for just between the input layer and the first hidden layer is 4,096 multiplied by 500, which is over 2 million weights. So these are only the number of weights that we have to learn between the input layer and the first hidden layer. In complex problems, we usually have multiple hidden layers. So as you can see, this compounds the problem of having too many weights. So what is the problem of having too many weights? Well, first of all, the learning is more difficult because the search space's dimension is increased. So we're searching for the optimal weight combination in a search space that's huge. Also, the time and resources for doing such a learning task could be prohibitive. And finally, we're increasing by increasing the number of parameters. We're also increasing the likelihood of overfit because our data amount is fixed. So if we have a higher number of parameters with the same amount of data, we're more likely to overfit. So the problem is further compounded for color images. As I mentioned, for base scale images, we can represent the value of each pixel by one value. For color images, each pixel is represented by three values, red, green, and blue, or RGB color mode. So because each pixel is represented by three values, we say the channel size is three. And the image is represented as a 64 by 64 by three matrix. And we have 12,288 values, which is basically the number of rows multiplied by the number of columns, multiplied by the number of channels. So the number of weights is if we have a hidden layer, after the input layer, we would have 12,288 nodes. And let's suppose we have a hidden layer with 500 nodes similar to the base scale image network. Now the number of weights is 12,288 multiplied by 500, which is 6 million. If you go back, for base scale, it was 2 million weights. For color images, it's over 6 million weights. So the problem is even more, it's a bigger problem if we have a color image. So clearly, the feed for our neural net force cannot scale to larger images. So 64 by 64 is not a realistic image size. Usually images are bigger. So furthermore, we have more weights. So another problem with feed forward neural networks is that we represent a 2D image as a 1D vector in the input layer. So any spatial relationship in data is ignored. So a lot of times, like different parts of an image that are closer to each other, there's a relationship between what's in there. But if you represent this as a vector, you're basically losing all that spatial information. So we clearly need something better than feed forward neural networks for image processing. And that's what convolutional neural networks are. So how did convolutional neural networks came about? So in 1959, Hubble and Weizel did an experiment to understand how the visual cortex of brain processes visual information. So they recorded the activity of neurons in visual cortex of a cat while they were moving a bright line in front of the cat. They noticed that some cells fire when the bright line is shown at a particular angle or location. They called these simple cells. While other cells fired when the bright line was shown regardless of the angle and location. So these cells, they seem to detect a movement. They called these the complex cells. It looks as if complex cells receives input from multiple simple cells. And it looked like there was a hierarchy. So Hubble and Weizel won the Nobel Prize in 1981 for their experiment. Inspired by complex simple cells, Fukushima proposed neocognitron in 1980. It's a hierarchical neural network used for handwritten Japanese character recognition. It's the first convolutional neural network, if you will. And it had its own training algorithm. So in 1989, Locoon proposed CNN, which was trained by back propagation. So back propagation is the algorithm used for feedforward and recurrent neural networks. And now, 1989, Locoon proposed a CNN that was also trained by back propagation. That's great. Because all these different architectures are basically trained with the same algorithm. And Locoon won the Touring Award years later for his contribution to the field of neural networks. So CNNs became popular when they outperformed other models at the ImageNet Challenge. ImageNet Challenge, it's a competition in object classification and detection. There are hundreds of object categories and millions of images. And the competition has been running annually since 2010. So notable CNN architectures that won the ImageNet Challenge are AlexNet in 2012, ZFNet in 2013, GoogleNet and VGG in 2014, and ResNet in 2015. So AlexNet was the first CNN architecture that won the ImageNet. And it did significantly better than not only other models, but also the models from the previous years. So that started this convolutional neural network craze, and everyone started doing research and improving CNNs. So a typical CNN has four layers. There's an input layer, convolution layer, followed by pooling layer. And a fully connected layer at the end. So we may have multiple convolution layer, pooling layers, and sequence. And we'll get to that and show you an example. So in this tutorial, we'll explain a two-dimensional convolutional neural network. But we have to understand the same concept applies to one or three-dimensional convolutional neural network. So the input layer, let's say we have a 28 by 28 base scale image, unlike at feedforward neural networks, we do not flatten the input into a 1D vector. Inputters were presented to the network in 2D as a 28 by 28 matrix. This makes capturing spatial relationships easier. We don't lose any spatial information by converting a 2D matrix into a 1D vector. So that's good. We basically resolve one of the limitations of feedforward neural networks. So the convolution layer is composed of multiple filters or kernels as they're called. So filters for 2D image is also two-dimensional. And let's say we have a 3 by 3 filter, nine values in total. And values are randomly initialized to something, for example, between 0 and 1. There are different initiation techniques. So convolution is you place the 3 by 3 filter on the top left corner of an image. You multiply those nine values by the nine image pixels and add them up. And then you move the filter one pixel to the right and you repeat. And when you get to the top right corner, you start from the left-hand side and you only move the filter one pixel down. And you repeat this until you get to the bottom right corner of the image after which you're done. So let's explain this visually. So here we have a 4 by 4 image and the blue. And the dark blue is a 3 by 3 filter. So the 3 by 3 filter starts in the top left corner right now and then it moves to the right and then down and right. And then we're done. So when you go to the top left corner, the nine values of the filter are multiplied by the nine pixels that it is basically covering. And we add them up. So there are nine multiplications and then one addition and then we get one value. So as you can see, we repeat this four times and we get four values. So the green is the output of the filter being applied, a 3 by 3 filter being applied to a 4 by 4 image. The green, it's like, it's a 2 by 2 image. So the filter or the convolution operator has multiple parameters. There's five of them, the filter size, padding, sprite, dilation and activation function. And we're gonna go through all of them one by one and provide an example. So filter size can be 5 by 5, 3 by 3 and so on, larger filter size should be avoided because filters are basically weights. And if you have a large filter, you have more weights and as we know, we don't wanna have too many weights for all the problems. Increased learning time, increased computing capacity, neither for the learning and more likely it would have overfit. We should generally avoid even size filters. Odd size filters are preferred because they have a nice geometric property in that all the input pixels are around the output pixel. You can't have, you don't have this property for an even size filter. So odd size filters and small size filters are the preferred one, probably 3 by 3 would be a good example. So here in this example, we applied a 3 by 3 filter to a 4 by 4 image and we got a 2 by 2 image. So the size of the image was reduced. Sometimes that's what we want, but sometimes we wanna keep the size of image as it was, we wanna keep the size of the output image same as the size of the input image. That's why padding comes in. So if we want to keep the image size the same, we can use padding. We've had the input image in every direction with zeros before applying the filter. If padding is one by one, then we add a zero in every direction, up down, right, left. If the padding is two by two, then we add two zeros in every direction, up down, right, left. So here we have a five by five image in blue and we have a padding of one by one and we have a 3 by 3 filter and the result is a five by five output image. So the size of the input image or the size of the output image is the same, is five by five. Our filter size is three by three or padding is one by one. So stride is how many pixels we move the filter to the right or down? Stride of one, we move the filter one pixel to the right or one pixel down. Stride of two, we move the pixel, we move the filter two pixels to the right or down. So here's an example. This is a stride of two. So as you can see, we move the pixel two to the right and then two to the down. If you compare this to here, the filter, which you can see down here, moves one pixels to the right or down. Whereas here, it's like a jump of two. We could have stride of three and so on. So that's another parameter. As you can see, having a stride of more than one reduces the size of the output image. So next parameter is dilation. So when we apply a three by three filter, the output is affected by pixels in a three by three subset of the image. So dilation is when we want to have a larger receptive field. We want a larger portion of the image to affect the output. So if we set the dilation to two, instead of a contiguous three by three subset of the image, every other pixel of a five by five subset of the image affects the output. So let's see how that works. So here we have a three by three filter. So nine values affect the output pixel, but instead of these nine pixels in the input that affect the output being contiguous, they are, because the dilation is two, there's one gap between them. So as you can see, there's a gap here and here. So instead of the input receptive field being three by three because the dilation was two, now it's five by five, but not all the pixels in that five by five affect the output. So we've increased the receptive field. Sometimes that's desirable. That's another parameter we can play with. So the next parameter is the activation function. So after a filter scans the image and we get a bunch of output values, we usually apply a nonlinear activation function to introduce nonlinearity. So one of the preferred activation functions for convolutional neural networks is RELU. So RELU is very simple. If you have a negative value, it replaces it with zero. If you have a positive value, it leaves it as is. So here's one example. Let's say we applied a filter to an image and we got what's on the left-hand side. Now we apply RELU activation function. So anything negative is replaced by zero. Anything positive or zero, we leave as is. So let's say we are applying a three by three filter to a five by five image and the filter is being applied to the top left corner of this image. As you can see, there are nine values highlighted in blue in the top left corner of the image and the filter itself has nine values. And these nine values have to be multiplied by each other and then added up. And how it works is for the first row of the filter, you multiply zero by zero, one by two, and minus four by one. You repeat the same process for the second and third row. For example, in the third row, minus one is multiplied by one, four is multiplied by zero, three is multiplied by two. And if you add all these numbers up, you get a value of five. You get the value of one pixel. So if this filter scans the image, moves to the right, repeat, moves to the right, repeat, moves down and across until you get to the bottom right corner of the image, you're gonna get nine values as the output. So let's say, so this would be applicable if your image was grayscale and only your channel size was one. But if your image is color, then you have the channel size of three. Showing how, so if you have a channel size of three, your filter also has a channel size of three. So that would be three dimensional. And unlike this one, it cannot be displayed in 2D, in a 2D graph. So what we do is we have, we represent the three channels in the input as three separate matrices. And we represent the three channels in the filter, again, as three separate matrices. And this filter applied to top left corner of the image results in the value of five for channel one. This filter applied to the top left corner of the image results in the value of four for channel two. And this filter applied to the top left corner of the image results in the value of 10 for channel three. Then what we do is we add these three channels up And we get a value of 19. So this is a 3D representation of the same process. So just so you see how it works. And as you just saw, output of a multi-channel 2D filter is a single-channel 2D image. But nothing stops us from applying multiple filters. And for example, if the input image is 28 by 28 by 3, number of rows, number of columns, number of channels, and we apply a 3 by 3 filter with 1 by 1 padding, we get a 28 by 28 by 1 image. So we go from channel size 3 to channel size 1. However, if we have 15 filters, we're going to go to 28 by 28 by 15 output size. So the number of filters allows us to increase or decrease the channel size. That's very important. So the next component of a convolutional neural network is a pooling layer. Pooling layer performs down sampling to reduce the spatial dimensionality of the input. This decreases the number of parameters, reduces the learning time computation involved, and also reduces the likelihood of overfitting. So most popular type of pooling is max pooling, where you have a 2 by 2 filter with a stride of 2, and returns the maximum value as it slides over the input data. So this is an example of CNN. You have the input on the left-hand side. You have a convolutional layer with ReLU, multiple filters. And then you have max pooling. And then you repeat this three times. Eventually, you get to a fully connected network. And you have your outputs. Let's say you're trying to detect images 0 to 9. So you get a probability distribution on images 0 to 9. This one is obviously a 0, hopefully. We'll get the highest probability for the image 0. So typical convolutional neural network has several convolution plus pooling layers. And each is responsible for feature extraction at a different level of abstraction. So for example, the filters, the first layer detects lines, maybe horizontal, vertical, diagonal. The filters in the next layer detects shapes. It's a combination of lines. And the filters in the next layer detects a collection of shapes, like objects. So filter values are randomly initialized. They are learned by the learning algorithm. So CNN not only does classification, but it also does automatic feature extraction. And this is a very important feature. This distinguishes CNN from other classification techniques such as support vector machines. And this is part of a reason that CNNs are so popular. Because feature extraction is a difficult problem. If it can be automated and optimized by neural network and the classification is also done by neural network, then that would be great. So the last thing in a convolutional neural network is a fully connected layer. It's the last layer in a convolutional neural networks. All the nodes from the previous layers are connected to this fully connected layer. And it does the classification. Maybe we have a couple more connected layers. So the MS data, it's a data set of handwritten digits, 0 to 9, obviously, composed of a training set of size 60,000 and a test set of size 10,000 images. Digits have been size normalized and centered in a fixed size image, 28 by 28 pixels. It's gray scale. So each pixel is represented by a value between 0 to 25, as we discussed before. So MS data set is a standard image classification data set. It's used to compare various machine learning techniques. So that's a nice thing about having a standard data set because different techniques can be compared. So we define a CNN and train it using MS data set training data. The goal is to learn a model that, given an image of a digit, we can predict whether it's, we can predict the digits, whether it's 0, 1, all the way to 9. We then evaluate our trained CNN on a test data set and we plot the confusion matrix to evaluate our classifier. So references, as before, they are mentioned in the tutorial. The tutorial, you go to galaxytrainingattraining.galaxy.org. Go to statistics on machine learning and click on the deep learning part 3 convolutional neural networks. If you go all the way down at the bottom, actually there's a link here, you see all the references for this tutorial. And as before, the Galaxy training material can be found at this link, various topics, tutorials, contributors. For any help, please go to helpgalaxyproject.org or use the Gitter Chats. And for the events, you can go to galaxyproject.org and see what events are being advertised. So what's left is the hands-on section of this tutorial in which we implement a CNN using Galaxy's neural network tools and we train it using the MNIST data set to classify images. So that should be done shortly in a separate video. Thank you. Hello, everyone. We're going to be going over the hands-on section of the convolutional neural network tutorial. So as always, you can go to galaxytraining website at training.galaxyproject.org. Then you scroll down to statistics and machine learning and scroll down to deep learning part 3 convolutional neural network CNN. So if you click on the slides, we're going to do section 5 and 6 here. So we're going to go to get data section. We're going to upload the training and test set files to Galaxy. And what we need to do, so what I'm going to do is I am at usegalaxy.eu. I've logged in and I'm going to create a new history. Let's give it a meaningful name, GCC 2021 CNN. And then I am going to copy these training test files, the URLs from the tutorial. Then I click on the upload data. I click on paste fetch data and I paste them here and I click start and close. So this is going to take a few minutes to upload because these are relatively large files. So the jobs have started. So I pause the video while the files were being uploaded and may take a few minutes. So we have to rename all these files and make sure they are all of type tabular. So what we do is we get rid of the URL and the file name and the extension. We save it. Check the data type. It is tabular. So we do the same thing for the other three files. Check the data type. It's not tabular. So you click on this new type dropdown, type tabular, select it, and then click on change data type. And that takes care of that. So we start on the third one, the data type. Here it is tabular and the last one. And this needs to be converted to tabular. So change data type. So this basically takes care of the get data section of this tutorial. So I'm going to be following the steps in creating the deep learning model architecture and so on from the tutorial. So there are multiple steps. This steps creates a deep learning model architecture. This one creates a deep learning model. Then we do training and evaluation, model prediction, and the visualization. So these, I follow the steps from the tutorial. I have the tutorial open on a separate window. So I can look and make sure I do all the steps correctly. What you could do is you could download the workflow. If you go to the top of the tutorial, you can download the workflow. If you click here, you see the workflow, you can download it. And if you go back, if you go to the, what's it called, the use galaxy page, you can go to workflow, imports, browse for the file. You just, what's it called, download it and then import it. So that would take care of everything. You can run the workflow on the data set that we just uploaded. But what I'm going to do is I'm going to basically do all the steps that the results in that workflow that's saved there. So let's go back. So I'm going to go back to the classification of MNES. So I'm basically following these instructions and the, in the use galaxy EU page. Okay. So first we need to do, it creates a deep learning model architecture. We have a set of tools here. We can search for them here. I just know them. I know that these tools are in machine learning under ML and statistics. So if you, statistics and visualization, sorry. So you see the header here, statistics and visualization on the left hand side. There's statistics, machine learning and graph display data. So the created deep learning model architecture is here. So I'm going to find it. So here it is. So it is going to be a sequential model. The input shape is a 784. This is because we're using the MNES dataset, which is images of digits zero to nine. These images are grayscale and 28 by 28 pixel. So 28 by 28 pixel, that's a 784 pixels. So that's the input shape. And what we're going to do is we're going to select a reshape layer here. And we're going to change the shape into 28 by 28 by one. So our input data, the image is represented as a vector, one dimensional vector. We're going to reshape it into 28 by 28 by one, which is the number of rows, number of columns and number of channels. Because this is a grayscale image, we only have our channel sizes one. One value between zero to 255 can represent a grayscale value with zero being black, 255 being white and anything in between with shades of gray. If the image was color image, a color image, then we had red, green and blue values that the combination of these three colors can represent any color. But we have a grayscale image here. So it's going to be a reshape layer and the target shape is 28, 28, one. We're going to insert another layer. It's going to be a convolutional layer and a Conf2D. So it's a 2D convolutional layer. We have a two dimensional image. The number of filters is 64. The kernel size is three. So we have 64 filters in this layer and filter sizes are three by three. The activation function, usually for CNNs, we pick ReLU and we're going to make sure that our image size is the same after the filters are applied. And the way you do that is you specify padding equals same. So we insert that layer two. Now the next layer is going to be a max pooling layer. So if we go to pooling max pooling 2D, that's what we have and it should be a size two by two. We're going to leave everything else as the default. Do another insert. Now we have, so usually in convolutional neural networks, we have a convolutional layer followed by a max pooling layer and we could have multiple of those. So we just added one of those convolutional layer followed by max pool layer. Now we're going to repeat another convolutional layer followed by a max pool layer. So here, we're going to have a 2D convolution again. Number of filters is 32, kernel size will pick three. And let me look at the tutorial notes and the activation function we pick ReLU again and that should be it. And the next one is again, every convolutional layer is followed by a max pool layer. So we go to pooling max pooling 2D and it's two by two. We're good. Another layer. So this is, we've already, I think added like five layers. So you, this is obviously not a big model but sometimes you could have dozens of layers hence the name deep learning, which means there are a lot of hidden layers in the neural network. So after the second max pooling, we're going to flatten everything. So we're going to use a core flatten. So every node in the, you know, in our 2D space is basically flattened into a, you know, like a feed forward neural network. And that's that. And the final layer, we have a core dense which is a fully connected layer. And because we're predicting images, sorry, digits between zero to nine, we need to have 10 units here. And each of those, we're going to use the softmax activation function. So the output of those 10 units, their probabilities over those digits. So the probability, the digit that has the highest probability is the prediction of the network. So I think we're good in terms of the architecture. I mean, the number of layers, number of nodes and all the other types, node types, activation types, et cetera of the network, we can click execute. So that should create the model config, which is you can download it as a JSON file if you need to. So the next step would be creating a deep learning model. And it's the same thing, you know, if you search, you could search for it. I just know where the location is. So I'm going to go down to statistics and visualization, machine learning. And we're going to go to create deep learning model with an optimizer, loss function and fit parameters. So we click here and I'm looking at the tutorial notes again. So the choose a building mode is a build a training model. The selected data set containing my model configurations in JSON, it's the output of the current job, which just started. So we leave that as is. We do classification or regression, we're doing classification. So we leave Keras GClassifier as is. The compile parameters, the loss function because this is a multi-class classification, we're going to be using a categorical cross entropy, not binary cross entropy, but that's for the case when our classification has only two classes. So let's find categorical cross entropy. The optimizer, the default should be Adam. We pick that and the metric, it's accuracy, that's fine. Number of epochs, we're going to set it to two. This is basically how many times we use the whole training set to train our model. So if it's two, that means we use every training example in the training set to train our model twice. And the batch size, we set it to 500. And that means, let's say our training example, our training set has a size of 60,000. We don't want to update the weights in our network after 60,000 training examples have been presented to the network. That's going to be too slow. So what we do is we update the weights in our network after batch size, which is much smaller than the training set size. So after 500 examples are presented to the network, we calculate the gradient or the derivative of the cost function relative to each weight. We average those over the 500 values that we got and then we update the weights. Sorry. Okay, there was a problem with running the Keras model config job. After we tried a few times, I think this was like probably the server was being maintained or something because this runs on the Galaxy EU server. It's pretty late, close to midnight, I assume over there. So anyway, I reran the job and it succeeded. So the next step in the tutorial is we just created architecture. We have to create a deep learning model and we go again to the tools pane, scroll down to statistics and visualization here, machine learning, if you scroll down, we can find the create deep learning model with an optimizer loss function and fit parameters. So I click here, we leave the choose a building mode as build a training model, the default for selected data set containing model configuration, the JSON file, that's the result of the previous step, which is pre-populated. We need a classifier because we're classifying MNIST images because it's a multi-class classification. We need to select a categorical cross entropy optimizer. We use Adam and we leave the metrics as is. Accuracy is fine. We set the number of epochs to two. That means how many times each training example the data set is used for training the data. So in this case, twice. The batch size, we set it to 500. So after 500 training examples are presented to the network, we calculate the derivative of the cost function relative to every weight. For all those 500 examples, we average them. So that way we know the step size for updating the weights and biases in the network. So I think this is ready. We're just gonna click execute and wait for this to complete. Okay, so the model builder succeeded as well. So if you look at the tutorial, the next step is a deeper in training and evaluation. Basically, we're gonna train our model and evaluate it using the training data. So again, in the tool pane, you scroll down to statistics and visualization under machine learning. We're gonna be looking for deep learning training and evaluation, which is right here. We leave, select the scheme as train and validate. Next, choose the data set containing pipeline estimator. Pipeline or estimator object is a result of previous step. That's job number 10, which completed right here. We'll leave it as is. Then we go select input type. Our data is tabular and we're gonna select our training examples here, X train and doesn't have a header. We want all columns and the labels are in the Y train file, which is here. It doesn't have a header. Again, we care about all the columns. So train and validate, tabular data, X train, all columns, Y train, all columns, I think we're good. So we're gonna execute this and wait for it to complete. Okay, so the training completed. We get three files as output. One is the result of the evaluation. The other one is the model and the third one is the weights of the model. So going back to the tutorial, the next step is model prediction. So we used some training data to train our model that now we're gonna use that model on the test data and see what the model predicts. So let's do that. That's going to be a model prediction. Again, it's the same thing on the left hand side. We scroll down to a statistics and visualization on the machine learning. We're gonna look for model prediction. That's model validation. This is model prediction. So first it says choose the dataset containing pipeline estimator object. That's pre-populated with the correct file. That's number 12, job number 12 right here. And the next is choose a dataset containing weights for the estimator of what that would be, number 13 weights trained on the dataset. So I'm gonna pick number 13 here. We're going to do prediction. So we leave this as predict. Our data is tabular. Our data is X test. Doesn't have a header. And we're interested in all columns. So we're gonna execute this and wait for it to complete. Okay, so model prediction completed. If you view it by clicking on this icon, you see that for the 10,000 test examples, we have predictions. So the first one is digit seven, second one is digit zero, digit one and so on. So going back to the tutorial, the next step is basically visualizing how our classifier performs. So we have to go to machine learning, visualization extension. So let's go there. Again, we scroll down to the machine learning statistics. Let's see if it's under graph display or, sorry, I have to look this up. So the machine learning visualization extension is under machine learning. So we're gonna click here, expand that. We do have machine learning visualization extension here. So selecting a plot type, we're gonna pick a confusion matrix. That's what's used for classification. Select data set containing true labels. This is for test data. So it would be Y test that has the labels. Data set doesn't have a header. And we're interested in all columns. And so select data set containing predicted labels. This is the result of the prediction, which would be a number 14. And so the correct one is pre-populated. Does the data set have header? It does. So I'm gonna change this to yes. And let's see, I think that's it. So plot title is confusion matrix between true and predicted labels. So we're gonna click on execute and let this finish. Okay, so our job completed. You can click on this icon to see the confusion matrix. So going back to the tutorial, we can see the confusion matrix as a table that describes the performance of a classification model. It basically lists the number of examples that were correctly classified by the model, true positive and true negatives. It also lists the number of examples that were classified as positive that were actually negative. We call these false positives or type one error. And the number of examples that were classified as negative that were actually positive. We call these false negatives or type two error. So what we do is usually given a confusion matrix, we can calculate precision and recall. So precision is a fraction of predicted positive that are actually positives. That's equal to true positives divided by two positives plus false positives. Recall is a fraction of two positives that are predicted. That's equal to true positive divided by true positive plus false negatives. So after we calculate precision and recall, one way to deal with just one number is to combine those two via F score, which is basically a harmonic mean of precision and recall. So this is a formula for harmonic mean of precision and recall. And if you look at this confusion matrix, the rows are the true class labels. So we have 967 plus two plus one plus eight plus two digits of zero and 1132 plus one plus two digits of one and so on. The columns are the predicted class labels. So column zero means that we predicted 967 zeros that were actually zeros. And same thing for column number one, I'm sorry, same thing for digit number one all the way to digit number nine. So the way, so true positives are these green diagonal points. These 967 zeros, true zeros were predicted as zeros. 1132 ones were predicted as ones. 1002 twos were predicted as twos and so on. But what about what's on the first row and first column? So these are the predicted classes, the columns. So these were predicted as zero, but they are not zero. 967 of them are, but four plus two plus eight plus seven plus two are not. So for example, these four were predicted as zero, whereas they're class two. These two were predicted as class zero where they're class five and so on. So these are the examples that were incorrectly predicted as zero, they're false positives. So if you add these values up to seven, eight, two and four, you get the false positives. So true positives is 967 false positives. If you add up these numbers and then false negatives are actual zeros that were not predicted as zero. And those would be all the other entries on the first row. So two plus one plus eight plus two zeros, true zeros were not predicted by our classifier. So that's our false negative. So we have true positive, false positive. True positive is the green circle here, 967 false positives. You add everything up in the first column, false negatives. You add everything in the first row and then you can plug them in into these formulas, calculate precision, recall and F score. And we can repeat the exact same thing for all the other digits. So you just have to do the second row and second column for precision and recall and F score for, you know, digit one all the way to digit nine. So as you can see our true positives are large, like 967 out of maybe less than a thousand. So that's pretty good. So that means this is actually a heat map. So the darker, the diagonal elements get better. And actually we're doing pretty well here. So in this tutorial, we explain the motivation for convolutional neural networks. They are really good for image processing and they have fewer parameters than feed forward neural networks. We explain their architecture. You know, there's this concept of filter or kernel. It has certain parameters. And then we have some pooling layer, like filters or convolutional layer followed by pooling layer. And we can have multiple of those in sequence to do basically feature extraction, automatic feature extraction, which is very, very powerful. This is an excellent feature of neural networks. So finally, we use the Galaxy's neural network tools to solve a image classification problem using CNNs on MNIST dataset, which is a standard dataset for image classification. So this concludes part three of the neural networks tutorial. This hopefully will give you an introduction to various neural networks that are in use. And if any of you have any questions or if you want pointers to more advanced tutorials or topics, feel free to contact me. I also hope to expand on these tutorials in the future. One idea is to come up with a tutorial for autoencoders and variational autoencoders. Those are non-supervised machine learning techniques that via neural networks, and they have a lot of applications in different domains. Okay, thank you, take care and bye.