 So in the previous video we looked at a few ways just to improve training a few concepts You can still read the rpops documents down at the RMD file from GitHub have a look at those What I want to do in this video is just to show you how we can implement at least a few of Those improvements that we saw in that video. I'm also going to introduce you to a package called TF Runs Let's just move down here We can see the TF Runs install it via your package manager and It is going to help us to look at the effect of changes that we make to our model Now this demonstration and this demonstration I'm going to use just a normal r script so that would be file new and then we see our script here Just an r script and there are two files here You see the file that I'm working in but there's a second file that just contains the model It should be familiar to you. There's our Keras model sequential We compile our model and then we fit the model So that's all that is in the second file that just the actual model Look, we're not even importing Keras here or TensorFlow or anything. This is just the model in the second file so let's go back to the first file and Let's get things started first line of code. I'm just going to set my working directory and I'm just going to click hold down control and hit enter or command and hit enter on a Mac and That just executes that line of code and we can see at the bottom here in the console that line of code has been executed so my files both of them Live inside of this folder or this directory and now it's easy just to reference the files that are in there I'm going to import Keras and import the reader packages Reader remembers to read a CSV file Keras because we need to do some machine learning Now the training and the test sets For this example exists in two separate spreadsheet files comma separated value files so they've already been split and I'm using the computer variables train dot import and train test dot import To hold those two CSV objects read underscore CSV not Dot CSV because this comes from the reader package. So let's do that control enter and The second one control enter and we'll see we have two Imports here test import train import That's what I call them and you can click on those two buttons and you can see we notice that they're 27,020 observations over 13 variables in the training set and then in the test set is 2,943 observations Along the same across the same 13 variables As per usual, I want to change these into matrices So I'm going to say train dot import and that's going to be as matrix the train import let's do that and Then I remove the names They were column headers and I'm going to remove them and I'm doing the same for test import and I remove the column headers So I'm left with these two matrices what I want to do now is to create my train and test data and For columns 1 to 12 That was the feature variable and the target variable is in column 13 Remember it's row comma columns square brackets indicating that I'm using indexing So I'm creating two train and two test sets here that to Two of each a train data and a train labels I should say and a test data and a test labels just to hold the features and Then the target variables separately. So I'm going to execute all four those Now remember in the previous video we spoke about now just normalizing Those with the way that we're going to normalize in this video is take the mean and standard deviation of each of the feature variables so each of those 12 columns and What we're going to do that's for the training set remember we're going to for each value Subtract its mean and divide by its standard deviation for that variable So I'm going to create this empty variable this empty vector. I'm going to call it feature dot means It's a vector and it has a length of the number of columns in my training data, which was 12 So it's this empty vector of 12 spaces control command enter and We can see it here feature measure It's just 1 to 12 and it's all false false false. Is this this empty? It's just this empty a vector of 12 values Now what I want to do is I'm going to iterate through each of the columns So for in I So a for loop I in 1 to the length of the feature means remember they're 12 of them So I'm going to iterate from 1 to 12 and I'm going to start populating feature means remember they're 12 empty spaces So during the first loop that will be spot number 1 will be the mean of My training data's all the rows in column 1 so I'm looking down column 1 I'm calculating the mean and I'm putting that in position 1 in my features mean Then it'll be 2 so it'll look at column 2 in train data Get its mean and place it in position number 2 and So until we get to 12 and then we see here we have feature means up here It's 12 numbers numbers 1 to 12 in the indexing and it's the mean of each of the columns And I'm going to do exactly the same for the standard deviation So standard deviation and run my for loopers as well. So I've got 12 means and 12 standard deviations Because what I want to do now is to create this normalized feature set So first of all, I'm going to create an MT Matrix that's going to be the same number of rows and columns remember it was quite a lot in In in when we first imported it it was 20,000 27,020. So rows over 12 columns. I'm creating this MT Matrix to hold that and Then I'm going to do two for loops a nested for loop. So on the outside for loops I'm going to go through all the columns and In the inside the inside for loop. I'm going to go through all the rows So what I'm doing with this I'm going to populate all of these MT values in this matrix And I'm going to start with row one column one and that is going to be that position train data subtract from that the feature mean Divided by the feature standard deviation. So the end refers to those columns So I'm going to go down each and every step each and every row in the first column Each one of those sample data data point values and I subtract from that the mean for that column or variable Divide by the standard deviation for that column or variable And once I'm done through all of the first columns all of its rows I Come outside back to the first for loop and we go to column number two And we repeat it for all the rows and column two Etc etc etc, so we just run through all of those very quick Now we're going to do the same for the test data So I'm going to create this empty matrix of the same number of rows and columns as my test set my Feature test set and then I'm going to loop through all of those this nested loop same story But I'm using the feature means and the feature standard deviation That's from the training sets mean and the training sets the standard deviation So I'm subtracting the test data So from each individual sample value data point value in the test data I subtract the mean and the appropriate mean and standard deviation by from the training set now let's import T of runs and You see we're using the training underscore run function from training runs. I don't have to put this Training runs and two colons T of runs and two colon symbols But just to show you that's that's where this function comes from And now I'm going to reference The model that I actually have so let's run through the model a model is a I'm going to call a baseline model It's a carous sequential model It has a dense layer another dense layer So two hidden layers and then an output layer the output layer is a single node because this is a binary classification So using the sigmoid activation function there be nicely to go from zero to one Let's have a look. I've got 48 units and 48 units So quite a bit more nodes there than the 12 nodes of my input vector I'm using rectified linear unit as my activation function in both and remember in the first one I have to input the shape which is equal to the number of features 12 But I've got this new thing here called kernel initializer And remember the first time that we get the first multiplication during forward propagation It's just going to initialize random values for the weights is the first time we go But we can also set those we can have a specific one and this time I'm going to specify kernel initializer and I'm going to set it to in it underscore w which I created up here So instead of this typing this here. I could have just put in all of this Right here, and I've just done it separately to show you that's possible So I have this in Initializer and this is a curious function initializer random normal So it's going to use a random normal this to a normal distribution for all the weights in that very first tensor that is going to be multiplied by My input vector and I'm saying that the values for that tensor must have a mean of zero and a standard deviation of 0.05 and I'm setting a seed value here one two three That means every time I run it and you run it you're going to get exactly the same weight values By default the bias Nodes they are set to zero all the biased values are zero initially and you see yeah I'm not even referencing it here, but you could Put it in here as well and just initialize a zeroes that is for for the Bias notes, but it's zero doesn't matter So what I want to do is just bring up a page for you here Let's bring up this page. So this is curious that our studio comm and I've moved all the way long And I've clicked on the reference tab at the top and you can see the contents here of of This whole long page, but you can just go to the right hand side and you see there initializes You see regular rises there. You see losses there Optimizes, let's have a look. So here's One of the initializers we used initializer random normal You can see how it works. There's a mean a standard deviation and a seed and you can set those Anywhere you want and you can also see all the other initializers so you can try any of these and if you click on them here, we have a Likun normalizer from young Likun from a publication and You can see you can read more about those read this efficiency backprop by young Likun I think yeah, that was and We can try that normalizer. It just has a seed. So there's no other argument to set there if we look at Some of the optimizers This is the RMS prop optimizer. You can see we've discussed some of these That Arguments in the previous video. You can read up more about them. Let's get back to a model So we're going to get to the compile step And I'm going to use the optimizer the optimizer RMS prop and I'm only setting two things the learning rate I'm setting the learning rate And I'm setting row value here So 0.01 and 0.9 those would be the defaults My loss is going to be binary cross entropy. I'm dealing with two categorical variables. That's a good loss function And then we're going to fit and we pass the training data normalized and The training labels to the fit and our validation set is going to be test data normalized and test labels now This is a data set that contains a lot of noise very difficult to learn from this data set so let's go back here and All we're going to say is training run and the file equals the name of this file because it lives in the same Directory or folder and up here we set The working directory. I don't have to refer to its full address. I can use it as such Now it's going to open that file and it's going to run that file and then something else is going to happen Let's have a look We see that the model starts running and we can see there's a big gap there's a lot of variance here a bit of overfitting I would say and We can see the model run over the 40 epochs that we have set There we go, and then this page opens up very nice We can see Some information here on the right-hand side. We can see an accuracy of the training set of 75 So we can certainly let this run a lot longer and then 70% for the Validation accuracy so quite low there. So a few things that we have to do we see the model here at the bottom The model with the the layers created and if I go to output there We can see what the model was and the code will give me the code of my model and the code of the file from which It was called as well So very good Let's close that Now we can go back to improvements and we can say let's make some changes to this. I Want to make some changes So let's change the Let's make this a smaller network. We're going to put 12 in each of these Let's put only 12 nodes in each of these and Let's make our learning rate slightly smaller 05 Let's set the road to 0.5 Just changes you can make look at those pages on the RStudio Keras website Think about some of the content concepts that we've discussed before I'm just going to illustrate Some for you here. So let's just save this go back to our original and Let's run this again just to show you if you do latest run Let's just run that it'll just give you some information down in the console as to what had just happened But we saw that in the overview that came up So let's train this again same file, but the changes have now been made. Let's run that There you go. It's run and we can see what has happened as we did before What I do want to show you though is just instead of latest run. Let's use compare runs and if we do that Very nicely we're going to see both of the two previous one ones come up and We can see the changes that were made So where the units were 48 before they are now 12 when the second hidden layer was 48 units And now has 12 the learning rate You see the change and the change in the row there for the RMS prop So you can see the changes that were made and you can do a side-by-side Comparison to see if you've improved things So we can see we can do quite a few more epochs because this was still going down So I'll certainly do that. We see that we've certainly improved our Variance here our high variance in that these are closer Pay attention to the scale though because that might trip you up that the scale make might might make this look a lot Better than than what it really is the improvements really are so have a look at these But that's a beautiful way to go about it. Just to do these Make changes and compare the runs make changes and compare the runs. Let's try something else Let's go back and for the weight initializer. We're going to use this Likun. Let's bring that back up again Likun, and you see what it what happens here. It is going to be a normal distribution but it is going to vary according to the number of input units and we spoke about that in the previous video So it's going to do this according to the input units That the input units for this weight tensor So let's just have a look at that all we have to do is say initialize a Likun normal. Let's do that So let's save this We go back and we're going to run our model again And there we go. We can see what I would perhaps do here is introduce some dropout or regularization And I would certainly let this model run for quite a bit longer, but let's have a look compare runs now and Now we can see from the previous one the changes that we have made we had we had the normal random normal initializer before now we have the Likun normal and We've to see the change that we made there and we can see the difference that it had caused So it is a beautiful way to play around with your model and see the changes that are made