 Hi everyone. Thanks for coming again. I'm glad I didn't scare you off yesterday. There was a few questions that came in overnight. I just wanted to sort of summarize those first and maybe address those first but actually before we do that let's just go a really really quick overview of what we did yesterday. So you know the purpose of the purpose of this clinic is to introduce you to deep learning workflows for image segmentation. It's a fairly difficult clinic because there's a lot of code and you know that unfortunately you're being exposed to probably more code than you would like or that you're used to at this point especially if you're starting out with Python and especially if you're starting out with machine learning but this is like a real real world quote-unquote workflow that you know it's sort of typical. Deep learning is still new enough that it doesn't have a lot of canned software that you can just sort of install on your computer and run with. That's becoming a little bit more popular but for the most part people are accessing these tools, these algorithms, these ideas by making their own Python scripts and so it does require a certain amount of Python knowledge in order to do this stuff. But with that in mind you know there's a lot that you can do without understanding every part of this code and you know I wanted at least to show you a sort of a working example of how you might do land cover landscape type classification using the sorts of imagery that GMFologists and ecologists that you know are now more sort of routinely acquiring such as from drones and from planes. So that's the sort of scale that I'm sort of addressing here and it doesn't mean that this workflow is not transferrable to smaller or larger scales. We talked a little bit about transferability of these techniques to satellite imagery. Yes that's possible. There's a few different, there's a few considerations that you need to take in terms of how you organize your data but in principle these methods do apply to any type of sort of spatial problem. And the same goes for a very small scale imagery as well. I've used, personally I've used these techniques on very close range photographs and they do an awesome job in general at many of the tasks that you throw at them. But that isn't to say that you don't need to be very careful about what data that you give it and how you frame the problem. This is just a tool like any other that you need to understand how to use. So just to recap then of what we did yesterday we downloaded a publicly available data set called Aeroscapes which is a set of RGB photographs of various types of landscapes, various places, various types of categories of land form, sorry land cover for the most part. Those are the categories that we looked at or sorry that it's contained in the imagery. The category that we looked at was vegetation and we just designed a neural network to essentially segment the vegetation pixels from the imagery. We weren't too concerned about any other classes. Today we're going to look at some of those other classes. So we downloaded the data, we made lists, we had, we looked at workflows to plot the imagery with sort of the label sort of semi-transparent over the top. And we started to develop an understanding of how to put model workflows like this together, starting with the concept of feeding models batches of images which is typically how models are trained. And so we first looked at this sort of concept of generating batches of images by making our own generator function with a yield command at the bottom so we can generate random batches of those imagery and their corresponding masks. We then, well we just explored that generation then we built a model. We talked a little bit about the unit model. We've looked at, hopefully looked at the video that explained it but we also talked about sort of the basics of what it did and why it works so well. And today I'd like to touch on a few of those topics again because we didn't fully explain what the residual part of the residual unit meant. So I'd like to talk a little bit about that. We talked a little bit about class imbalance. Class imbalance is the situation that's very common for landscapes which is that you have many, many more pixels of one class compared to another or of one class compared to many others. So in our case, vegetation wasn't, it was quite dominant but it wasn't the most dominant class in our imagery and lots and lots of imagery only had very small pieces of vegetation and some of the imagery had very large patches of vegetation. Generally it's a class imbalance problem. There are fewer pixels of vegetation compared to others. If you were to look at other classes and hopefully a few of you did, looking at boats and people and animals and things like that, there's an even more, there is the class imbalance issues even greater because those objects tend to be very small compared to the to the landscape elements. So we looked at this function called, we looked into this using a dice coefficient in order to try and get around some of those problems. And I'll talk, if you want, I can talk more about that. We made our model, we sort of saved that model to disk and we had to look at that model in terms of its graph structure and we looked and we talked a little bit about this diagram here. We talked a little bit about these very long skip connections which are these very long arrows that you can see that sort of connect the top of the model to the bottom of the model. And we sort of covered why those occur or why those are important. Those are essentially how units are able to discover low-level features as well as high-level features. It takes the image, it passes it through a series of layers to extract the very low-level features and then what it does in the end is it just sort of merges the feature maps from the low-level features with a set of feature maps that have been inherited from the image without much filtering and that preserves some of the higher-level features or the finer detail features within the imagery. And that's essentially why units are so popular and why they're so useful. We train the model just for a few number of training epochs with our own custom callback function that we use to plot an example of a validation image as we went. The validation images are those images that are used by the model to evaluate how good it's what you use to evaluate how good the model's doing as it trains. The training data are those images that it uses to actually set the weights within the model to optimize and converge towards a good solution. There's a third set of data that we sort of set aside that we call the test set and that's for testing. It's an independent set of data that's sort of drawn randomly like the other two sets and that's used to evaluate how good the model is. We train the model just for just five epochs as I said. We stored the weights inside a container, an h5 file format and sort of format file. We train the model and oh this is a hang up from yesterday's code. We train the model, we had to look at it, we found it was pretty good and we talked a little bit about the units. We looked at some of the training history. The training history here, what you're looking at if you remember on the left hand side here is the blue dashed line is the training dice coefficient. That takes a value between 0 and 1, the closer to 1 is better. You can sort of think of that as an accuracy score. It's the degree to which the pixels are actually overlapping between the prediction and the ground truth mask. We saw that the validation set was bouncing around and you would have got different curves. You would have got different curves to me because we were training on small batches and they are randomly drawn from a very large training set. It's very unlikely that we actually trained on the same individual images and I would say it was impossible that we trained on the same sequence of images. We're going to have slightly different results. All neural networks train iteratively and they converge towards a solution can be a very long time, typically several hours if you're training over tens to hundreds or thousands of training epochs. Which is also probably why when you ran the evaluate command you probably all got slightly different results because the model that it's using was trained using different, it saw different images. That's a very, very quick overview and recap of yesterday. Before I field some questions, today I wanted to move on to a common situation especially for our scientific fields and that's where you don't have enough imagery to start with. What strategies are available for us to sort of kickstart a model or to use another data set to examine whether our data set is going to be useful within this model framework? One really cool thing about machine learning is that the practice of taking the outputs from another model and directly using them in your model is totally acceptable and it's encouraged even because one of the real problems with applying these deep learning algorithms is of course that you can almost never have enough data and that they are data hungry and they require a lot of examples to learn from because the training process is essentially unsupervised. So we are going to move on today and talk a little bit about that in the first sort of hour and then hopefully in the second hour we're going to move on to the second part which is contained in the second notebook and in that one we're going to talk about what's called a multi-class segmentation problem and all that means is that instead of just having one class of interest now we've got many. So we're going to look at vegetation but we're also going to look at a bunch of other classes and combine them all together in the end. So we have a semantic segmentation which is a segmentation where every part of the image is something that we can ascribe meaning to. So before I go on to the transfer learning does anyone have any questions about what we covered yesterday? I do have a couple of things that I'm going to address that I've written down here I'm going to bring up as I go. One is going to be how do you take this code and actually use it on your own data. I'm going to talk about that at the end. I'm going to talk a little bit about the transfer learning. The elevation data that came up was a couple of questions from folks who wanted to use these techniques with elevation. It's more experimental. There's fewer people who have convincingly demonstrated that elevation is enough to segment landscapes in meaningful ways but that's not to say it's not been done. It is technically possible. You can feed these neural networks raster stacks of anything between one and whatever however many dimensions you have in your data. With elevation data you typically have just one band which is elevation but you might also have derivative products like slope and aspect of things like that that you may throw at the problem. Other people have done that many times within classical machine learning frameworks like random forest and support vector machines and things like that. I think it is worthwhile trying but I don't have any particular perspectives on it other than it's more experimental. You have to choose your classes correctly because if you're just using elevation data obviously you have to make sure that your class is zoned out by elevation which is obviously you know that and it's an obvious point to make but the more you can do that obviously the better. If you have two classes that exist in the same elevation zone then they may be in different spatial locations so the model has something to work with. The model knows that this particular class is only in this particular portion of the image but in general you probably want some other covariate sort of use within the model as well as elevation to actually bounce that idea off internally if that makes sense. So before I move on here let me have a quick look at the chat and I'll take any questions you have at this point. I'm going a little bit faster today because I realized we have still quite a lot to cover. A couple of people have got errors I'm not too sure where those errors are occurring. Let me get back to those. Jason says did your workflow include code to look at the total area of different classes across all the training imagery? No that's a good point that's a good thing to do so what Jason means there I think that's referring to the class imbalance problem so one way to actually verify whether you have class imbalance would be obviously to enumerate the number of pixels that you have for every class and so that would be a simple matter of loading in all of your imagery and just counting the number of pixels that you have in each class and then perhaps making a histogram or something like that. I could certainly make a note to update that that would be a simple update that I could make for this workflow. So thanks for that. Hopefully that answered the question but yes there would be ways that you could examine the total area covered by every class. Obviously you need to know what the pixel resolution was but you could just do it from the purposes of the model training it doesn't know how big every pixel is. There isn't any way to build that information. There's no easy way to build that information into a model framework like this so it's not really using that information to come up with a solution. But that's not to say that space is not important as I said yesterday. The convolution filters are set up in such a way that they share weights spatially and so that's how they're able to upscale the image from the small bottleneck that's created which is the sort of essential features of the image that's useful for prediction. And if everyone's still with me then I'll move on to the next thing. These errors I'm not entirely sure. Chris and Paola are not entirely sure where these errors are coming in. Hopefully I can find them. Don't worry about that now. It's the first time you train the data and I just ran the file from the top and got that error. Oh really? Okay. Oh and Paola's fixed hers. Awesome. I'm not having the same problem but I'm just trying to figure out where that is exactly. All right so transfer learning. So this is this in this part we're going to take a during the during the training process yesterday we wrote out a file. It had an h5 it's an HDF5 format file. Binary format you can't see it with your eyes but you can read it into almost any software and it knows what to do with it. Karis knows what to do with it because the model weights are sort of stored in a very prescribed way so when it reads those files in it just literally allocates the weights as it needs to as it goes to the different model layers and there's obviously many many many of these different weights that are essentially controlling the way that this neural network comes to its solution. There's also a bunch of biases in there as well which they go hand in hand. So in general those those are called parameters of the model. The parameters are the things in a deep learning model that the model figures out through training. Hyper parameters are the things that we have decided that that would include for our case yesterday that would include the batch size. It would include the the image size. It would include the the optimite the particular optimizer that we use the loss function that we use and the metrics that we use and things like that. But they also would refer to decisions that are made within the model itself such as how many convolution layers that we want how many cooling layers and things like that. But in general hyper parameters are just those things that pertain to a specific model that you can control. The parameters are the things that are learned by the model and that's those are the things that we're transferring over. It only works because all of the hyper parameters stay the same. The the batch size is going to be the same in this time. The the image size is going to be the same. We're going to use the same model obviously because that's how we transfer the weights over. So this is typically how you would start with your own problem. You would take I mean or I don't know if it's typical but that's certainly what I would encourage. You would take a similar data set that someone else has maybe looked at or has labels for and you'd play around with that data set if it if it seemed like it was going to be pertinent to your particular your particular task or problem. So if you were looking at vegetation in in Landsat imagery or whatever you might take some vegetation classifier that had been made for modus or centenal or something else and you just might want to transfer it over before you start this from scratch. If you've got UAV imagery like we have then obviously you know if you wanted to segment out water or a particular plant or a particular feature of interest there might be already someone who's done something like that or there might be a set of classes out there that are similar for you that look similar with respect to your spatial resolution and things like that. But another way that people tend to do this is actually to just transfer the weights from models that have been trained on imagery that is completely different and you see oftentimes you see people sort of transfer weights from a from what's called ImageNet which is a you know a very very large set of a million images of everything that you can possibly imagine it's got a thousand different categories. Oh apparently everyone several people were in the ah sorry about that I didn't realize everyone was in a few people were in the waiting room and yeah sorry about that. So so a lot of people use weights that have been transferred from this thing called ImageNet million images thousand categories and that the the intuition behind that is that you transfer weights over from a convolutional neural network that has the ability to recognize stuff in images. Low level features high level features and mid-level features that are you know sort of generic things that you find in imagery you know corners lines intersections of lines vertices and things like that as low level features and then that sort of might translate to higher level features such as specific textures of a specific spatial structure or specific spectral structure for that matter. So transfer learning is extremely common we're not going to take the weights for that have been learned on a generic set like ImageNet we're just going to take the weights that we learned that we had our model learn yesterday and we're going to transfer it to a different data set for this for this essentially the same classification task. The data set we're going to be using is called the semantic drone data set I did talk a little bit about it yesterday but just to recap it has it's somewhat similar in that it's you know sort of UAV or drone imagery that's been taken from about 10 to 30 I think meters off the ground there's more there's more of the there's fewer images but there's more categories many more categories it turns out and the landscape isn't really a natural landscape it's more of a housing development or something like that there are natural elements in it like vegetation water and things like that but for the most part it's sort of a residential suburbancy but we're going to see how well our vegetation weights transfer over so hopefully um oh and of course because I was talking so much I didn't actually finish running my own notebook so maybe I'm just going to discover the same errors as everyone else had um as I do this bear with me just a second as I just skip through some of this stuff so I'm able to then take the rest um so essentially what we're going to do we're going to download the data like we did before um it's I've I've hosted it on a google drive like I did before um it's a subset of the data um just so you know the full set of data you can acquire online it's much bigger um both of these datasets I've provided links to so if you were to download them yourselves um obviously you cared but they're much bigger um in terms of like there's a few different versions of these data that exist um for example uh different subsets and things like that but you'll you'll get a sense for it if you find these data useful but otherwise you could just use the subsets that I curated for the purposes of this class so I'm um apologies here I'm just getting down to the point where um you probably have already exceeded me and I'm just trying to get to this point here um so maybe I can just go straight to this this is going to take a while isn't it oh of course I'm training my model apologies for this this is uh unprofessional let me let me go let me just go straight down to I think where I can revisit this this is another difficulty with using google colab is obviously you have to sort of run things sequentially and oftentimes if you don't if you miss a cell you're scratching your head for a few minutes thinking hold on what the hell it worked a minute ago it's probably because you just didn't run a cell up up above um so what I'm doing here is just um lab- downloading a zip file that contains the the images and a zip file that contains the labels the images are different in this respect because they're much larger but we're going to actually use the same size of image uh partly because I just wanted to demonstrate that you this this model can still be effective if even if you're uh radically sort of downsizing your imagery which some uh some hardware configurations or most hardware configurations sort of necessitate um and so what I just did there um is just unzip those two folders into um a couple of different uh places if you use the the files uh tab over over on the left hand side of your window and you might need to hit refresh and you'll see a new folder that's been made that's called quarter and each one of these jpeg images is is an image of uh from the dataset and you can click on on one of those it might take a minute to load but you can click on these these images and it will load them up um and you can see that the imagery is a bit different um in terms of its sort of resolution much much higher resolution imagery and the labels are color labels they've been made um in color and so each one each each individual class here has a has three unique uh color values that we're going to have to unpack and so that's the that's the first task that we're going to do um so I'm using the uh I'm just using this uh so OS stands for operating system and walk just means it's just going to walk through a top level directory and give me everything under it so what this piece of code does essentially just uh cycle through just this uh folder and gives me all of the images within it but if they if it had been um organized with subfolders it would do essentially the same thing and preserve the parts for me so um because I've got that in such a way um I can now just use this this type of list comprehension in python to just go through the images and figure out which ones are the sample images the color photographs because those are the ones that have the jpeg extension and the labels which are those uh the labels which are the the categories which have the png extension so in this in this case it's fairly simple to just break that out the original data set consists of 400 images um and the reason why this is called quarter is because we're only using 100 of them and that's just because these um because uh just for the purposes of speed really but uh the original data set has 400 and so if you wanted to work with this data I encourage you to download the original um these are our classes I've got them in a class dictionary um for a specific reason uh so uh that will become apparent later on a dictionary um in python is actually something that this might be confusing for folks a dictionary is actually something that's like this so uh you might have uh the the value that you've got the name of something and then you have the value of that something so your value there would be some variable that's in your python workspace um but what I mean here is uh is it's just a class uh let's just call it a class data structure because it's actually not a classically a dictionary because it's not uh it's not got these curly brackets um and then this is a tuple anything in python that has the it's sort of surrounded by uh these parentheses is a tuple so basically this is a list consisting of many tuples and dictionary was a poor choice of word here I'm just extracting only the names because I want to have a variable for later on that just contains the string names um and later and in a minute I'm going to extract these rgb color values because I need those values to decode um and and um make a make a two-dimensional label image after my three-dimensional label image it's three-dimensional because it has three bands rg and b um and so I want to take those three bands and those many colors those tuple colors and basically uh condense them into a two-dimensional array so we can feed it uh or matrix sorry so we can feed it into the um into the neural network so here um I'm at I'm prepending this quarter to uh to my file names here because it's not uh because it's not going to know that it's in the quarter directory and here I'm defining a function that is just going to read an image into memory and a label into memory and resize them to the size that we want we've got rectangular images that are several thousand pixels in each dimension but the version of the images that we're going to give to the model is actually square and and only 512 um we're going to see that uh that's that doesn't that's not going to be a problem like it sounds um lots of conventional geospatial work would obviously this would be really a bad thing to do you don't want to downsize your imagery um but for the purposes of the neural network we have to make sure that our all of our data fits with uh with onto our GPU memory and as I said yesterday our GPU memory is very limited so unless you have very very large GPUs um or very very small high resolution images that you've chopped up then a common strategy would just be to take your big image and just sort of make it smaller uh it's not necessarily ideal but it does work okay and hopefully i'll convince you of that um but an alternative workflow of course would be to take these large images and to chop them up into smaller images um we did have a question yesterday from zoltan who asked you know whether there would be consistencies between the tiles so i'll try and remember to get to that but yes there should be and i'm gonna show you a technique right at the end of today hopefully we get there um that will show you how to deal with any type of discontinuities that do exist if you do chop up your imagery and use a model to predict so um i realize i'm going a little faster than yesterday uh but hopefully i'm hopefully i'm still making sense really we're just at the point here where um we're just defining our own color map uh based on these rg rgb values that have been given to us by the people who made the data we're just um we're reading in um the that that array of of rgb color values and we're dividing them by two five five because we want to turn them from an eight bit integer into a float between zero and one um because only because this particular function from map plot listed color map is expecting the the the colors to be in rgb but in zero to one format and here we're making a a a plot of an example image drawn from the catalog with a semi-transparent label image on the top and you can see that these labels are quite detailed they've gone in and they've done a very good job uh being sort of careful about not mislabeling um and so this data is is a very good dataset to sort of cut your teeth with some of these different algorithms because it's it's probably the size of images that you're sort of used to you know a sort of a more than 10 megapixel and it's many classes which is also probably more like what some of the tasks that you might anticipate needing these techniques for um but the the problem we have with this type of dataset right now is that these colors are in these are colors they're in rgb color space and we need to turn them into labels that are just integers that we can feed to our model the model is not going to understand what to do with um a color vector um it just wants an integer to know what every class is and so without going into the details because we don't have the time this function here really just takes that rgb image and it just looks up um what the colors are within that within that sort of list of colors that I made and it just designs an integer to each of the colors in such a way that um you have as many integers among the entire label collection as you do labels that you're interested in or is containing or is that that are contained within the data um I will stop in a minute to ask answer questions but I just want to get to this point here where um I'm essentially I'm first of all I'm removing the unlabeled category um unlabel is difficult to deal with because what does that really mean uh is it unlabeled because they just didn't get around to labeling it or is it unlabeled because they couldn't fit it into their classification scheme I imagine it's probably the latter um so it's really unknown and we don't know what to do with unknown so I'm for this purpose of this class I'm just going to get rid of it we're just not going to use it for the purposes of what we're doing we're just trying to see whether we can replicate the type of vegetation segmentation that we achieved with the last data set by using those weights but in the next lesson we'll revisit this a little bit more in terms of the unknown class and why that's a problem but also why it's not really a problem because it really does depend on how you frame the problem um so let me um let me uh so this is this is where we have taken our label image our RGB label and we flattened it down to two dimensions so it's now just a single matrix of of integers and here um this is just a handy way to just grab an arbitrary color map but in discrete form instead of like we did before where we sort of defined the color map based on the actual RGB values given to us and yesterday when we gave it discrete colors that were sort of intuitive to us you know cyan for sky and green for vegetation things like that here I'm showing you a third way that you may get at this by making a color map and that's by just using this um this command get cmap and what it does it just takes um any of the existing built-in color maps or color ramps um in map plot lib and will just give you a discrete a discrete number of colors and that's just the length of our of our the number of classes that we have so I've got a new color map here that when I plot this image now it's going to just give us this semi-transparent mask um no of course it's not because I didn't run the code so this time it's going to give us our semi-transparent label but it's going to be in a different color space and this time the label is just integers so before I go on to preparing for the model training um does anyone have any questions while I have a look at the chat I realize I'm going a little faster today but um that's only because I do want to try to get to the end um okay so there's no new questions on the chat so I'm going to assume you're okay but if you did want to unmute yourselves and ask questions that would also be fine all right so if we go back to our class list up here then one second there is a question oh okay comparchi it's in the chat I can read it out um okay sure I've got it yeah so any pre-processing required for shadow that's a good question uh possibly yeah it really does depend on how bad your shadows are I mean I would say that the the imagery that you see is not the image the form of the image that is actually used by the model to do the class to do the prediction you know the the models the models are set up in such a way that they are quite good at extracting features even in the presence of significant noise um and even in the presence of significant shadows but to a point um it really does depend you know a little bit it depends on you know that's sort of the bit depth and the color space and the compression and the various factors that went into making your your image you know like um we're not using raw imagery we're using sort of jpeg or something like that that's been heavily modified and compressed so there's that you know there's that to pay attention to and obviously the the best version of the image that you have um to give to the model is the one that you want to use but also the model is quite good at extracting features in the presence of noise um I've worked a lot in Grand Canyon there's a lot of almost every photograph you ever take anywhere in Grand Canyon has a very dark shadow in it but even even even in the presence of those shadows you can still use these type of techniques to to see what the landscape is composed of but the short answer is yes if you can but I don't even know what you're doing otherwise let the model do it and see how well it works but obviously if it's really really deleterious then you need to sort of address it before you give it to the model but it's a good question thanks and you know shadows you know things like that they come up a lot with the types of imagery that we might be using as ecologists and geomorphologists because obviously you know we have the the vagaries of weather and light to always deal with a lot of the um you know the lot that's also a lot of the the limitations with some of the the the materials that you see presented you know blog posts and in papers and especially those categories and models that are trained on sort of anthropocentric classes you don't have the same sort of issues necessarily with lighting variation things like that as you would at landscape scale so those sorts of considerations are definitely are definitely good okay so if we if we go back to the list of the classes that are actually contained within this dataset I guess we can just look at the the the color bar here um you'll see that there's a few different vegetation types and so you know without really going into too many details about you know what type of vegetations in there I thought we would just sort of throw them all in together as one generic vegetation class but I would encourage you to you know to to redo this oh and re re go run run through this workflow again with different combinations of those or with different classes obviously it's only by doing that that you're going to sense of how I get a sense of really how powerful this model is for this particular problem um but for the purposes of today we're going to combine the grass the trees and the vegetation classes together and you know the other reason why I wanted to show that is because that's also a fairly common workflow that you may encounter um you know where you're the the the model that you want to transfer weights from to your dataset and they may not have exactly the same classes as I said but they may have something similar you might combine them or separate them out in a way that suits you so here I've just made a list of these class class foregrounds I'm calling them the classes that they're interested in vegetation trees and grass um but obviously that could just be any number of those classes then you could just give it one class if you wanted but I'm going to give it three and then this next little bit that just goes through and it's just a piece of of um code that will just extract where that class is found within the list of classes that we made earlier the text list of just the names of those classes and it will just append them to this uh this end list that I have and so n in the end is just a list of three numbers um those those classes as they are represented in the in the in the in the set of labels and then numeric value so we're going to be looking we're going to basically tell the model to treat as vegetation any pixel within our label image that is either a three an eight or a nineteen and we're going to call them all one thing vegetation here we're using basically the same batch generator that we used yesterday but this time we've modified it a little bit because uh specifically because we've got slightly different uh way in which our data sets are organized uh if you recall yesterday our air escapes data was contained in these two folders jpeg images the segmentation class so we had a little lining there that said uh grab an image um and and sort of replace uh this folder with that folder this time we've got everything just in one folder so we're just basically telling it to just grab it grab an image and replace it with the thing with the with a different extension because you'll see that the images and the late and the corresponding labels are identical except the extension so that's actually a really convenient way to do it and to organize your data and then um the next thing it does it just cycles through each of our classes in n and um and uh just appends them to a single binary mask essentially so what we get in the end is a binary mask where uh the ones in the image are associated with any one of those three or all of those three vegetations and then um the zeros are everything else so it essentially functions in exactly the same way as as the previous batch generator that we used um and so and so here i just generated um a set of images using the same batch sizes yesterday which was eight i believe um and so x and y are x is x's are the images and y's are the labels and so the length of those two should be eight but the size of any one of those will be two uh five hundred uh five hundred and twelve by five hundred and twelve by three if it's an image um and five hundred and twelve by five hundred and twelve oops caps lock five hundred and twelve by five hundred and twelve if it's a label but it actually um appends this extra one to the image as well because it's expect the model is expecting that and that appears in this expand dims uh expand dimensions there and i'm saying expand my third dimension the first the zero dimension here is just the uh the the sequence of where that the image is in the batch so the first so if i were to treat this like that then i've got my um my individual my my first or zero label um and my first or zero um image and this little bit of code is just telling me well which which one of those image contains my class and it turns out that in the in particular one that i ran you're going to get something slightly different because this is randomized but the one that i ran told me that each one of my images in my batch had that class in it and i can verify that by obviously plotting them and this also gives a sense of what the data looks like you can see that because this is randomized um it's actually grabbed the same image a couple of times which is interesting never seen that before but um it's also because the image set that we have is a lot smaller than yesterday we're only working with a hundred images this time yesterday we're working with a couple thousand okay just quickly consult the chat kerry says can you give an ecology geomorphology example is training a new model to distinguish between plant species feasible oh that's a really specific question that's part of my own research um i don't know yet i think is the is the honest answer to that question whether or not these didn't mean to derail you i just was wondering when you're using all these training images of drones and pavement if this can move beyond that yes it can if it's too long of a question i won't derail us yes i would say i'm confident that yes it can um but it does depend on how you you know you frame the problem and how much data you have i think the from what i've seen so far is that these these residual unit models are very powerful discriminators of objects that are um very obvious to your own eyes at the end here i'm going to show you i developed another class that's sort of a more advanced version of this class that actually does go through a really specific ecology example using this exact exact same model to identify intertidal oyster reefs and i'll show you that at the end um if you wanted to take that further and that's that's sort of an example it's the best example that i have for how well this model might work on a noisy difficult data set that may be more what you're used to if you're a remote sensor working in ecology because obviously the sum of the signals that you're looking for are pretty small that's not going to i'm not going to i'm not going to extend my confidence to saying that it's going to be able to identify individual species of plants because a i don't i really don't it's not my area and b it's entirely possible that there's not enough distinguishing colors and textures um at every part of their growth cycle or whatever that would give you that reliability uh but i would encourage you to just look into it for sure so um here i've i've like i did yesterday in order to try to um get us looking at roughly the same thing even though we're training on random batches we're going to be using at least the same starting point the same set of training files testing files and validation files and they have been allocated by myself but randomly using this code that's commented out at the bottom if you wanted to try and replicate that um and again it's like i did yesterday it's a 50 25 25 split i i want more i want more training data than i than i want validation and testing data um but that's not to say that that's always going to be the case you you may find that um giving it different splits is going to be better obviously you want to try you know the the most ideal situation is that you're giving the model um relatively few training examples and you're and you're reserving the majority of your dataset for testing because that's more natural that's a more uh that replicates the actual um application of the model in the real world because you have obviously uh lots of unseen imagery that you need to predict but in this case i'm using about double my training files um that's also fairly common uh if you if you don't know what number to use then i'd say use use twice as many training files as you have uh validation and testing files and if you are able and if you have enough data that you can actually split it into those three different groups then that's best just to reiterate from yesterday the training data is used to set the weights and reset the weights in the model iterative the validation data is used internally by the model to report how well the model is doing as it trains but those data aren't used by the model to set the weights so they're not actually used in the solution um so they are an independent set among themselves and they could be used for testing too but it's even cleaner i think to use and it's even more convincing to use a third set where the model has never seen it as your testing set if you're able um so once again we're going to train this model so this time we're going to just use um five epochs again like we did yesterday just because we don't have time for more um and i'm going to skip through to uh to actually get it running um because it takes a few minutes and then i'm going to go back and explain what i just did okay so you'll you'll hopefully recognize this code from above um this time we're giving it a a a file path with a set of weights uh that it's going to write to uh we're creating a training generator from our set of training files like we did yesterday and a validation generator and we're evaluating how many steps we want that model to to pass through during every training epoch which is simply just the number of the files that we have divided by the batch styles we want the model to see every image in the catalogue during every training epoch but we don't want them to see them in the same order which is why we use the batches and we don't want them to see them all at once which is why we use batches what we're doing here is um we're training a model from scratch so we're not in this this is the version that we it's our baseline that we're going to compare to later we're so we're not using transfer learning right now because we haven't loaded the weights to the file the um if we were to to to load the weights at this point we would use uh the same construct as we used yesterday and we'd give it you know some file that we had there we haven't we haven't got that to that point yet um what we're doing is is training a model that is i'm calling a cold start because it doesn't have any um it's it's essentially starting like um it's training with uh random numbers for weights and it has to figure it out all from scratch um how to do it and you'll see that as it trains you know you'll have you'll have different images to me because again it's randomizing the batches um and it's randomizing which one it's showing uh in in the validation image here but hopefully you get a sense that it's even even uh from the first training epoch it's it's pretty good at finding the vegetation and distinguishing the vegetation from the rest of the categories um and vegetation here is obviously a category that i have chosen specifically because it works um but i would encourage you to again redo this with other categories to see how well this works in general for other categories but but you can see that it's going through here it's gone onto its third epoch already and you can already see that um it's it's doing an okay job at finding the vegetation so our baseline in this case is going to be fairly good already and what we're going to do um in the next couple of examples is we're going to give it um a set of weights that we trained the previous model on the previous data set the aeroscapes data set on vegetation uh we're going to use the uh the version of those weights that were the outcome of five training epochs and we're going to call that a warm start and then at the end we're going to use a version of that same of those same model weights for this time uh from a model that was trained over a hundred epochs and we're going to call that hot start and we're going to see what the difference is between the cold the warm and the hot models as a demonstration of of what you might be able to achieve if you use transfer learning on your data um but while these models are training i thought it would be a good uh juncture to talk about the the a little bit more detail about the the model that we are using um just to recap from um from above where we had the the image of the unit um i realized this is a this is a difficult image if you're not used to looking at them but essentially on the left hand side you've got your input image um which is your rgb image and that's why it's got three bars there are three channels they said oh well actually no in this case um not sure what that's representing but that that first bar would be your image that then got sort of went through a convolution block um you know that's why there'd be sort of several lines stacked vertically here because it's going from one thing to the next to the next before it then goes through a a significant pooling event um which is essentially crystallizing or condensing the features that the previous layer has extracted but the the main point really that i want to to illustrate here is it's it's called unit because it's u-shaped um this first part of the of the model is called the encoder down to this point here this point here is called the bottleneck and then this this this expanding part here is called the decoder and so it's a it's a it's a version of an auto encoder decoder network it's auto because it figures it out itself just like all neural networks do and it's and also because it's encoding itself um and the encoder because it's encoding that image into smaller and smaller and smaller feature representations of those features until it gets to a bottleneck feature and then it uses that feature along with all of the spatial information that it inherited from the the progressive downsizing in a reverse way to essentially interpolate back up again using the spatial um the spatial relations that it that it used originally the gray lines to recap are those skip connections um those long skip connections that essentially merge um high level feature representations with lower level feature representations to provide um to provide a sort of a hybrid of those two features that the model can then use to predict um that's what's called the vanilla version of the unit that was the the original 2015 paper uh implementation of the idea and then what we're using is actually a slightly different version of the model um that has what's called residual connections inside it and so that's why it's called a residual unit and I just wanted to sort of go over that at a high level at least before um we sort of moved on here but I can see that the cold model has now stopped training so actually what I'm going to do real quick is um get the next model training because that's going to take another five or 10 minutes and that will give me enough time um to explain the the residual model okay so um if you're still following then what we're doing here is um we're making an entirely new model so we're just calling this uh this function yet again um we're giving it a new path to store its weights to this the same as the previous one except it's propended by warm and not cold and then what we're doing here this is this is the point this is where the magic happens this is where the transfer learning happens happens essentially here load weights because we're loading weights um to the same model framework same architecture same everything same hyper parameters but with a different set of weights and these are the weights that we learned from the air escapes vegetation problem over five epochs um like I said so we initialize the model we load the weights oh of course we don't have those weights because um they were they were generated by my model training above um so what can I do here so if everyone's not following why I just got that error it's because I didn't run that model training so I don't have that file so that was an unanticipated thing on my part so let me see if I actually do have that file um for the purposes of moving on here oh damn it no I actually don't have it so we're gonna have to skip that part unfortunately uh but you you should have it if you've run the the training above I just don't have time to do it so I'm gonna I'm gonna move um oh do I have this one yeah no I have that one right oh damn it damn it damn it damn it okay something's gone wrong up here because I should have loaded I should have got that file from somewhere oh yeah that's right yeah I reloaded it right at the end of the first one excuse me bear with me just one second here I lost my weights here we go there they are uh so this this was the the piece of code that we download this was the file that was trained by myself over 100 epochs that we're now going to use um to hot start our model and unfortunately that means we're not going to get to see the warm but you should still see the warm if you go through the exercise I hope that's clear to everybody let me know if that's not okay so we're now we're we've now initially initialized and compiled our hot start model and we've given it this is the transfer learning part we've given it those weights and this time we're going to just train on top of that okay so while that's training um let's go back to this idea of residual uh skip connections all right so what you're what we're looking at here is a figure that originally came from a paper by drosdilla et al um which introduced this idea of the of using um residual connections with units and it's now turned into uh sort of an alternative version of the unit and it works better for a lot of cases um than the original version which is why we're using it um the idea behind this is that it's the same essentially the same structure you see the units you see the the dimensions are approximately the same in this version of the figure um and you can see that it has every it has these long range skip connections that we talked about earlier but it also has these short range skip connections and what and they're called they're otherwise known as residual connections and there's a video that um we're not going to look at now but I do encourage you to watch it it is a you know technical it's computer science video essentially but hopefully it gives you an intuition of what this really is but the essentially the idea is that as well so every time that the these convolution blocks um sorry every time the image features get passed into a new block the skip connection will take a version of the feature that comes out of a particular layer and it will just combine it it will just sort of add it and concatenate it with um with the with uh sorry I'm getting lost the it will take the that version of the feature map and it will just skip the next layer and just add it to the output of the next feature map so in this case you have three layers you have your image um comes into here goes through these three gets pushed through a pooling layer into this layer which is now smaller and then it gets pushed into um the next layer but before it does it gets sort of skipped over and added back in and the intuition behind that is that the each layer um now sees not only the output of the previous layer uh but the output of the previous layer so it saw the information that the previous layer saw when it made when it tried to make its uh adjustment to the weights as well as the output of that layer if that makes sense um it wants to know that the reason why this allows for faster convergence and for deeper models um and more powerful models is because it sees not only the feature representation that's the response of a particular filter but also the information that went into that filter so it gets to learn twice essentially we're using the same version of that feature representation um and I strongly encourage you to watch that video which probably is a lot more eloquent explaining that same principle so um the residual unit is essentially the version of the unit that we're using and that's the one that I found to be more powerful in general for these types of these these types of problems that we're looking at here at least this type of scale um we're on to the uh training epoch four out of five um go through I wanted to just reiterate some of the scores that you're seeing just to sort of recap from yesterday and we're about an hour into this um into this clinic now and we've got an hour one hour left so we we've still got quite a lot to cover but um the the what you're the the the numbers that you're looking at here since the first one's the loss that's the training loss that's the current state of the model and where it's out in its lost space um the loss is going to be going down from a large number and converging towards zero and as soon as it gets to zero obviously it's a perfect model so you don't ever get zero you get a small number that's close to zero if you've done well your dice coefficient um conversely that's a number that scale between zero and one one closer to one is is optimal but the things um generally neural networks are very good at fitting to data right the the universal function approximators they're very very prone to overfitting to the data and so a lot of the challenge a lot of the art and the science that goes behind deep learning is to regularize regularize the model which means preventing it from overfitting the data learning the general trend the general patterns in the data so it's able to perform on on things that it's never seen before but it's seen things like that so the numbers that you really do need to pay attention to are the validation losses and the validation dices there's no point having a model that has a dice coefficient of 99 if its validation dice coefficient is down in the 70s or the 60s or whatever um that's the number that we're trying to maximize really which we try to maximize both of them we're trying to we're trying to make sure that those two curves um over over time sort of converge to the same place they may not get there at the same rate uh but they ideally they want to get to the same place and that's when you know that you have a good model and enough data so this so the model now has stopped the hot model is now stop training um so i'm able to make that comparison but because i didn't have the warm model i wasn't able to train the warm model right now i'm just going to go ahead and uncom i'm sorry and comment those out but you should see on if you've gone through this exercise more diligently than i have then you should see those curves and come out i'm just going to have two curves to look at on every plot you should have three so interesting the um every time you get a different result that you can see because you have different training batches and very few training epochs so you'll everyone will see a slightly different version of the graph but most people will see in the end the red line has got a larger uh uh dice coefficient than the blue on average and a smaller loss this is actually a pretty interesting case this is one of those cases where there's not a massive difference between them in the training data set but there's obviously a huge difference between them in the in the in the validation set um it's pretty interesting that oh all right that's right so the hot model here is better for the validation set than the cold model on average and maximum minimum and mean and the um the hot model here has a smaller loss which is what we want and what we expect what's odd is that the loss is going up but i think that's just by virtue of the fact that we're just looking at a very small um number of training uh epochs and that we would expect that to come back down again and i imagine you're all looking at very different things um in general i've i've run through this a few times in general the red line is is always better than the blue but whether or not the green line is better than the red line is it's it's completely dependent on the specific images that you saw over this number of training epochs so don't worry if you see a green your green line at the top that's fine um it's also an indication i think that this particular model for this particular data set and this particular class doesn't need a hell of a lot of um transfer learning to occur for it to sort of get a kick start on what it would otherwise achieve um what i mean by that is in this case we trained a model for five epochs and we trained a model for a hundred epochs and there wasn't a huge difference in the ability for those two scenarios to uh to to create an improvement in the model they both worked about the same way um but what we'd expect in this in this case is is then to just train us for a small number of training epochs on one data set and then you're ready to start training uh for real on your data set and in and so in in an ordinary scenario you would keep that that training going for much longer than five training epochs you would probably choose something a number between maybe 50 and 150 in order to see how well that model gets to and you'll see over time the dice coefficients will go up and up and up and they'll start to asymptotically pat plateau and then the loss function will do the same it will hover hopefully just above zero um and stay there if you ever sort of encounter situations where your loss function just keep your loss value just keeps on going up and up and up and up obviously there's something wrong you need to stop the training and fix the problem it's not normal for the for the for the loss for the loss value to keep rising it should just keep going down um for the validation set it may jump around quite significantly but the general trend over time is that it's going down and down and that the variance of its jumps is getting smaller and smaller so before I go on to the oh and I guess for the completeness here let's um let's get the test scores for each model this is the same piece of code that we saw earlier on with the other set it's it's just calling this image batch generator function one more time to grab a generator to describe a generator function based on these test files not the training or the validation files so we have our 25 test files there and um we're going to evaluate each one of our models but I'm going to comment out my middle panel here because I don't have that model and what it's doing is just evaluating how well the model did um based on the entire test set and we should we should see that the dice coefficients and and mean IOU scores for this one are going to be lower than they are for this one that's the hope anyway and that's what you should all see um there's going to be a lot of variation in the values that you see um because you trained on different subsets I keep saying that but it is important to understand that this is a random you know this is a stochastic process that we train models with we give models random batches of imagery and we do things to them that that you know that doesn't does not enforce consistency but in the end you should have a model that if it's trained over sufficiently long period you should have a model that does converge towards towards its its ability and the consistency that you see should should increase and so we have a situation here where we've demonstrated that a model that has been hot or hot started essentially by by transferring the weights from another training exercise that has significantly it's sort of given us a 10 increase in our dice coefficient straight off the bat so it's a good starting point for you to do your model training with and it might you might even get to a situation just in a few or maybe a few more training epochs where uh that's good enough and that you you're done um so this last piece of code here before I take some more questions and move on to the next tutorial that's just taking that that hot model um and just um and doing something where it's making a making a prediction and you can see that you know that is not perfect but it does a reasonable job of of finding the vegetation uh if anything it's under predicting where the vegetation is because it hasn't found in these particular examples it's done a much better job at finding the trees than it has the grass and I I offer the reason for that is because we didn't see grass necessarily in our previous data set maybe that wasn't the best class to use but but you also see that if you train these these models for longer then it will start to pick up the grass so um the grass is basically not detected yet but we would expect that the model would learn how to detect the grass later on it's going it's it's learned the trees because the trees have a much more distinctive distribution of sizes compared to lawns which you know have a bigger distribution of sizes I would imagine but also you know the specific structure of of these leafy vegetation is very different from the grasses which are obviously kept very short so they just have a very different texture to them even if they have even if they occupy um a similar color space so um that's the transfer learning uh learning transfer learning sort of in a nutshell for this particular workflow um I wanted to spend a lot of time on it because I do I do think that this will be a very common um situation that you might involve that you might encounter when transferring these ideas to your own data and another couple of I a couple of sort of general points that I wanted to make before I answer questions is that you know typically you are using this sort of size imagery this is probably smaller tiles of images than you're used to using and again it's because of gpu memory limitations we use a gpu not a cpu because the training times are in order to shorter but the the downside is obviously we're um we have a lot of memory issues when we're using very large images and so we typically either chop them up into smaller bits or as here downsize them so with that I'm going to have a quick look at the questions and if anyone wants to ask me a question now a bigger time per pixel confidence probability scores all right that's a good question the um not easily it you're not it's not a trivial thing to look at the per pixel probabilities um we all we have is a sort of a pseudo probability that comes out of our um of our of our classifying layer so if I go back to that um let me go back to that if we go all the way back to our model actually we'll see in this line of our model um the classifying layer or what's what's called the classifying head of the model is um is using an activation function called sigmoid and so what that does is it gives you the a measure of the probability of it being every class or sorry of it being that class because we only have two classes in this example um so that is a value between zero and one like a probability but it's not really like a it's not really a probability because it doesn't come from a joint probability distribution the what we have is a conditional probability distribution of it of the label given the image but we don't have so that's you know that's the p p is y bar x but we don't have p is x comma y which is the joint distribution of of the of the image features and the labels and that's not possible because we that would have to be specified over the entire data set whereas we only have um that we have we only have matches to work with at any one time but it's also a direct result of the the model itself being a discriminant function it's not a function that can generate probability distributions it is not capable of specifying or evaluating or solving or inferring the joint distribution of the two so what we have is essentially a discriminant function with a sigmoid activation function which does give us a value between zero and one with zero being there's no chance that it is that thing and one being that there is that that it definitely is that thing and all the numbers in between so that is also why when it came to let's scroll down all the way down to our batch function our image generating function here oh no i didn't put it in there um somewhere in here i have this yeah so in here for example this is also why we use a threshold of 0.5 so here i've made a note so we use a sigmoid function a classifying layer that are like probability scores but they're not true probabilities if you said that to a statistician she would say no that's not a probability um but a machine a data scientist considers it a probability because it's an entirely data-driven exercise it has no theory behind it so what we're doing here is all i'm saying here is i'm going to call it one if it's greater than 0.5 so in that case that's a decision that i made and there might be more uh there might be better ways to arrive at whether or not that really is the thing um but that would be that would be on you to try and figure that one out um but what i'm gonna go what i'm gonna i'm gonna touch upon this issue again in the next lesson as well so hopefully if that wasn't entirely clear maybe the next time i say it it's a little bit more clear can we instead of a binary results actually get that continuous data between zero and one and look at that in a map uh yeah so okay we can do that right now so if i did um if i just took i gave this a color bar i'm gonna make it smaller because it'll look better and then here i'll just get rid of that line and i'll run this again and this time you'll see a color bar next to every subplot and that's going to show you where you are within that space the real the color map i've used here doesn't necessarily give you a good indication of that so let me use something that's a little bit better what would be a good one that one maybe um this one's gonna show reds and blues so you can see that in this in this case it looks like a lot of the values ended up being one but there are many values on the edges that would be less than one oh in this case actually you've got more than there's some values that are more than one which doesn't seem right but maybe that's maybe that's not right but um yeah that's how you would essentially look at those those sigmoid scores um so it wasn't it's not it's not too much of a leap actually to just always do that threshold but because a lot of the values end up sort of bunching up in the in the point nines um up to up to one but um as i say you would you would probably figure that aspect out for your own particular problem it would also change obviously if you had many classes and you could use the relative scores or the overlapping scores in different ways and that's sort of that's a good lead-in to what we're going to cover next does that answer your question yes thank you so much you're welcome um let me just kind of one more look at the chat um leslie wants to see everyone's models i'll let you guys confer over that so oh one person got a hot of uh i'm not too sure what that means um so did the loss function and metrics you specify influence over under prediction jason could you explain what you mean by that question can you hear me yes so you noted you noted in this model that there was some potential under prediction in the vegetation and i'm curious you know if we specify different metrics in the loss function if uh if if use of different metrics might push the model to predict more vegetation be more aggressive or be less aggressive and how it's classifying things i was just curious if your if your specification of the metrics you're using and the loss function you use kind of can influence how the model is predicting yes it does and that's a good question so um let me go back to the loss function that we're actually using um the dice fun the dice loss function in our case um is appropriate for the class imbalance issue but because we're using um three categories i think you're right in that the if perhaps a better way to do this or a better way to actually tease out whether or not the either any one of these three classes is badly influencing the model would be to separate them out and give them their own loss functions or to weight the loss function so you'll see examples of people using weighted loss functions if they have class imbalance even among the classes that they're looking at so in this case is we have class imbalance among grass and trees and i don't know too much about the vegetation but we certainly have fewer tree pixels than we have grass pixels so that would be a case of a more much more advanced case of using your own your custom loss function that would be a modification of an existing loss function that would weight those classes differently and that's another way that you can get around class imbalance does that does that address the question yeah it helps thank you okay and um yes so there's stochasticity in it this is why you're seeing different scores and um a couple of people are seeing their cold model better than their hot model perhaps that is just a function of the us training over very few numbers of training epochs i would say that if you trained over many many more you would always see that hot one better it's just it just by virtue of the fact that one one small one batch over over these number of over these number of training epochs they could just throw the model into a different trajectory that it takes a while for it to come back so as as you know a way to visualize what this model is really doing i find it intuitive and helpful to think of a lost landscape right so you have this and that's a lot of people call it you have this this landscape that's full of hills and valleys and one of those one of those valleys contains the optimal solution right your learning rate sort of specifies how far you can jump across that landscape and you may start you know at some point you may be on the top of a hill which is where you really don't want to be but you may sort of descend to training you sort of descend into maybe the first valley and you have a look around there to see whether the optimal solution is in here and then you might have to sort of climb back out of that valley into a deeper valley and it's the deeper valley that's going to find that you're going to find them the the actual minimum of the function so the the stochasticity is because you will start with random numbers when you're cold starting you're you're introduced so that's one you you're introducing random batches at different times and different um chronologies so that's another one and um it takes a long time for the model to figure out it you know the one of the reasons why we do train these models over hundreds typically hundreds of epochs is because it takes a long time for the model to figure out that it's in the right valley and that it needs to stay in that valley and just figure out you know where in the valley it needs to be in order to find the minimum so hopefully that analogy is useful but that's the reason why you're seeing this this variability but if you all train that model for a hundred epochs and send me the results I'd be really interested to see them because that would be a good way to actually examine how much variability that does introduce okay so um I'm gonna address there's one more question here about subdividing the classes lumping or splitting I'm going to address that in the next in the next uh tutorial just for the purposes of completeness I would like to move on from this one into the next one if anyone doesn't have any more urgent questions hopefully that was convincing that the that the transfer learning is useful even though some of you are seeing a variation okay actually before I move on to this but just I wanted to remember to say that so this is going back to this is our this is our course website the deep learning landscape classification that that you all started from and you probably clicked that link in order to get here we're now moving on to um to this one we're using a the the the same model now but for a multiclass segmentation problem so we're going to be sort of estimating a few different features at once um but I wanted to just go to the end here because I did manage to finish this additional course so this one um this one's a little bit more complicated but I wanted to just sort of introduce it to you folks to those folks who really do want to take these methods and and try and run with them um this is a bit this is a more complete course because it shows you how to organize your data how to generate the labels and and how to deal with different label formats it's also using a data set um this is not my data set it's fully acknowledged here but um we're using a data set that comes from Duke University it was used to train a model called model called Oysternet and and it was it's basically for detecting intertidal oyster reefs and it's a much more difficult data set um so there's a few different strategies that I've gone through here um but it goes through from scratch like how to prepare the data set um different you know getting ready to get everything ready doing augmentation things like that um and then it steps through three different training strategies essentially and converges on a good solution in the end and so it's a bit more of a complete uh case study for specifically you know remote sensors on the core who are interested in these very sort of difficult problems it's it turns out it's quite difficult to segment some of these reefs because the roughnesses are very similar to other roughnesses in the landscape um so enough of that but the um but that's that's available it's going to stay up there and you're quite welcome to to look at that and if you had feedback for me that would be awesome because you know I'm trying to make these these tools as accessible as I can um so without much more of a do then so you've got these two sites now so without much more of a do let's move on to the to the to the second and the last part we've only got half an hour left so this is going to we may not get to the end but this is hopefully or we may get there fast but um hopefully this gives you uh a few ideas some codes and a starting point to you know a specific workflow that you might combine some of these models that you train for binary segmentations to reiterate what I said right at the start of yesterday um I I find it valuable to treat segmentation problems with real world data these messy data that we have um oh sorry I'm paying too much attention to the chat here I'm going to minimize that just for a second um treating this as a series of binary decisions um there's a couple of reasons for that partly is because you don't always know what labels you need um in order to make the entire segmentation work of your entire landscape for a multi-class problem if you're just interested in one class that's one thing but if you're interested in many classes what I'm talking about here is a workflow that's going to combine models that were trained on individual classes and then use a secondary workflow to then sort of deal with any um any mismatches or or weirdness that gets um that's the result of that and this again this goes as a Zoltan's question about you know if you're in a situation where you've got a very large imagery that you need to chunk up into smaller tiles you know it's conceivable that you could have uh that the model working in prediction mode would segment that entire scene in such a way that you know one tile and the adjacent tile might have classes that don't overlap at the edges and so this would be a way this this workflow that I've got in this particular workbook here is this it would be a way to deal with that situation in a in a in an objective way um but there would be many ways that you could deal with that situation using geospatial tools or or or some form of logic that you came up with in order to assign uh those those those values if you had a windowing approach that is a good one I mean if you if you had a very large let's say you had a satellite image and you windowed it and then you had um you had predictions for every window but there was a 50 overlap between the windows for example then you could use like a an average or a mode or whatever but but this this workflow that I'm going to show you right at the end here is you know a way to to deal with that using a a more powerful model called a conditional random field okay so I'm going to start by what we're going to do here we're going to load the same libraries and we're going to use the same functions as we did previously we're going to use the same metrics and the same models so I've skipped through all of that stuff because you've seen it all before and there's not there's there's not much to talk about here so I really want to talk about and we've talked a little bit about how I program it as well so really I'm going to skip here down we're going to again we're going to use a batch size of 8 and we're going to use an image size of 512 it does it 512 is just a number it doesn't have to be um on the log 2 scale it it it could be any number um oh is that true maybe that's not true for now let's just use 512 so um we're going to download the air escapes data again because we're in a different workbook now so we need to go through that um we're all we're being backended by a different kernel different computer so we need to redownload the data hopefully that doesn't take too much time the um again you're we're using these air escapes data so that's the same data that we used at the start of yesterday so that's uh we're going back now to um you know our vegetation our sky roads um and things like that this should be familiar to you because it's the same code that we used uh yesterday in order to ascribe colors to labels I'm using a combination here of html color codes and in built colors um it doesn't matter it knows how to deal with both and here I'm making my model again I'm using the same uh optimizer as yesterday I'm using the same loss function the same metric I'm not this time keeping track of the uh of the mean IOU score that was just for the purposes of illustration the metric that I really care about is the dice coefficient for the reasons that I've already explained and again this is a the the the files that we're going to use to enforce some level of consistency among our respective outputs but that's not to say that we're actually going to get exactly the same um outputs and I think you've already learned that lesson the number of training files again is double the number of testing and validation files that's not for any particular reason it's just that um it's just a rule of thumb that I happen to have applied here okay so what we're going to do instead of waiting for all of these models to train um I've got all of these models already trained for a subset of these categories I ended up choosing just the categories I was really interested in um I'm segmenting the landscape in this sort of pretend example here so I don't need I don't really want to know where the boats or where the people or the animals are I'm more interested in where the buildings are which is what construction is where the obstacles are which is just things like fences and posts and stuff like that um I I assume it's obstacles because this is for some sort of robotics application but it's sort of obstacles to some sort of device or vehicle um my background class it's sort of my catchall class uh that contains lots of landscape elements that don't quite fit into construction vegetation roads and obstacles so all of these files again they're all hosted on uh google colab I've just downloaded them they're all h5 files um that contain model weights the model is the same model that we used before we just I just basically pointed um the model to to these uh six classes and had it go through the process of of of training on those classes now you probably have realized that that is not the most efficient computationally efficient way to do things but and and of course there are models out there of course and including the unit that will will um that you can adapt to multiclass segmentation problems you just set things up in the same way but it keeps track of all of the different classes all at once and sets up a different loss function of each and sort of averages them all combines them in the end um but what we're doing is is is the is a is a I'm proposing a different way which is to essentially treat the exploratory in an exploratory fashion uh look at these individual classes and we're in we're sort of simulating a situation here where you may have classes you may have inherited some initial classes and you want to see how well they work before you added more or you may be on the fence about whether or not a certain class might actually work and you've gone down the the you know gone down the road of of creating a few of those labels from imagery and now you're in a position where you're like okay I want to see how well this this this label is going to work um and I'm going to sort of do that for a few of my labels um and so what I'm the workflow that I'm showing you here is a sort of a very specific one that is not particularly common even but I think is uh one that's worth exploring for those reasons um it also allows you to examine the errors in um in isolation oh and of course I don't I don't have that file why do I not have that file um bear with me just a second here I don't know why that has given me an error now um what this piece of code is doing so this is just a very efficient way of loading all of my models in but really it's just uh replicating something like model vegetation equals res unit uh size this is where I'm adding my third dimension if I had a single band image I would just leave leave that a size but because I've got multi-bill bands I'm just sort of using some tuple math there to add them together um and then my batch size right and then I'm using I'm saying okay model vegetation compile is the next line of each one and then uh load weights would be a third and all I'm doing here is is making that more efficient in terms of coding because I have a list of these classes that I'm interested in so and because the the way that I construct each model is going to be the same um I'm you know I'm essentially just making uh using the exec command which is similar to the eval command in matlab where you can just evaluate a statement and it will just execute that statement and so it's a really useful uh if a little advanced way to sort of make that more efficient but essentially all it's doing is this but it's just doing it in a in a for loop so I just need to spend just a second here trying to figure out why um what's going on here and I imagine everyone else is getting the same issue it's telling me that that file is not found air escape weights vegetation you resonate so I'm I'm seeing that file I don't know about you guys interesting um if anyone wants to chime in here they could give me a hand I don't understand is anyone else seeing that same thing uh I got the same error dad hmm well I'll be darned and I didn't but I but I was having problems downloading one of the zip files for the images in the other script oh well there's some weird thing going on here so I just for the purposes of moving on I wonder if it's it's being I imagine this is just caused by the no I'm I'm looking at this I think and I just can't see it what the issue is it's telling me that those files don't exist gonna get it it didn't take very long to download these files so I'm gonna just nuke them all get rid of them refresh they're all gone and then I'm gonna redownload them and hopefully this time they work so okay they're too small so something is happening here which means that for whatever reason these files aren't getting downloaded um and that's not something that's that's not part of the script that's something that I don't have control over unfortunately it looks like unless anyone has any smart suggestions those files um they exist on my google drive so I can just I'll tell you what I'll do I don't know why I think it's just some glitch with either google colab or with google drive at this moment in time so what I'm going to do is just load them in from my computer because you can upload files using this so I'm just going to go and find those files on my computer real quick and upload them um so I'm able to finish the rest of this class you only got 20 minutes left um while I do that is anyone got any questions for me apologies for this I was not anticipating this those files should be five megabytes and um they're not they're only a few kilobytes so something happened um and I don't know where this stuff is okay there it is sorry Dan quick question yes I'm not typing in the chat because I'm actually watching on my phone and it's too awkward for me oh okay go ahead just how long are these are these notebooks going to be available for us to use and play with oh they'll be available they'll be available until um they become obsolete I'll keep them up there all right perfect yeah if we make a copy yeah we can play with yes please yes make a copy I strongly encourage you to you know there's a lot of difficult material in here so you know this isn't the sort of class that you're going to walk away from feeling like an expert but I think my my my goal here is to expose you to these workflows really to enable you to replicate them yourself but also to you know to sort of give you an indication of what sorts of things are possible and and the types of workflows that that are common with this type of work that doesn't mean that you know these there's not some you know idiosyncrasies in the in the way that I have done this but um oh but um but but yeah as I said right at the start you know it's it's going to take you probably it might take most people a few run-throughs so to really understand a lot of these difficult concepts um oh I don't know why this is happening so these files should just load in but they're not for whatever reason does everyone having the same problem as me so it's telling me that they're five megabytes let me see if I can just do this real quick apologies for this it was going so smoothly too um let me just see here if I can just read one of these files in because otherwise we're not going to be able to do the rest of the lesson live coding works for me compiler and then I give it a loss so you have to in order to work with the model I think I mentioned this yesterday but in order to work with the model you have to compile it even if you are using it in prediction mode or transfer learning mode you you still have to compile it it still needs to know what metrics to pay attention to okay so oh so um um metrics oh what am I doing optimal oh okay so then the next thing I should be able to do is model load weights and then error escapes you can see how bad I am at typing now okay so I'm sorry guys but I'm I cannot see what the problem is here I see that file it has the it doesn't have the right size something weird something weird's going on here um all right well just for the purposes of getting through this apologies I I honestly have no idea what happened there those files yeah I could hear my screen because they ran and you could walk through it um well it's okay Chris because I've already got it I can do that too here because I've already got the outputs but but thanks anyway I think I'll just keep going here and plow through um I will of course I will troubleshoot this immediately after I get off the call so you should be it shouldn't impact you too much but um what I'm what I'm doing here is um I I set that up in such a way that it was going to load uh six different different models for six different classes it was going to compile them and load the weights that were downloaded but what's happened is that I can't seem to load a version of those weights that seem like they're fully loaded because they're not the file sizes and it's complaining at me because it can't find the file signature um but let's assume that those models were uploaded with the right weights then this is just an exercise here of just really just reiterating what we did at the start of yesterday's lesson just sort of seeing what that looks like how to plot it with labels we have um but this part here this takes a very long time which is why I commented it out because I didn't anticipate having enough time in class turns out I was right um but what this does is essentially it just cycles through each of those models that have had the weights loaded in so they're now useful for prediction and it's just using that same uh that same piece of code that we used uh previously uh to create a test generator to evaluate the number of steps and then just to evaluate the model based on the entire data set and so what I got when I did that these are this is the output that I got when I did that so you sort of see these the output for six of those six different classes in this order um so vegetation uh was about 97 percent uh correct uh background 94 the um these two classes here oh sorry this class here and this class here was worse so that was um obstacle and sky and then um road did pretty well so in general there's there's a bit of a mixed bag here and that's good because we want to uh demonstrate a specific uh principle at the end but we've got very strong classifiers uh something you know a few of these I sort of anything above 80 percent I'm going to call good and um you know so four out of the six are pretty good in that regard but um we're going to combine those two together and then use another machine learning model to actually try and figure out um whether or not there's some room for improvement on every specific image and the algorithm we're going to use for that is called a conditional random field all right and I can't I can't run these these these um these cells but I can tell you that this is this is one where I've just loaded in a sample image and uh I want its original size and I'm just circling and just cycling through those models and making prediction for for each model and I'm storing the prediction and the ground truth resized version downsized version in uh oh sorry um the actual size version into two separate lists into the M list which is uh for the for the model and the T list which is for the ground truth so M stands for model in this case and T stands for truth um and um it's it's using each model to predict each uh to each pretty each class and creating a separate binary mask so what we have is then a stack of these binary masks that we need to somehow combine into our multi class segmentation which is going to be the end result um and in order to to flatten these these uh these this stack of uh masks we just simply sum over the first axis and that's just going to essentially uh take all of these binary masks which is just zero as a one and it's just going to sort of condense them down but that's it's going to give them different values because I use this counter up here so I'm basically cycling through this I've initialized a counter of one and so one is now vegetation that next time it comes back through the slope counters on two and that's background and that gets assigned two then to those pixels that are background three to construction four to obstacle five to road and six to sky and so in the end then you have this sort of this this array that's just values between one and six but it might not be integer values because um there might be um the same class may have been predicted over the same pixel so that's the situation that we need to deal with we don't want there to be two classes per pixel so we need there to be a secondary process to sort of go through each image on a task by task basis in order to evaluate how likely either of those two classes were so if you have a certain pixel that's um called one thing then that's not a problem it is consistently called that thing but if you had another pixel that's called two or three different things by the by two or three different models then you need a sort of a different process to sort of tie the room together at the end and that's what I'm demonstrating here and that's what that's what the conditional random field is for so um and I'm doing I'm I'm I'm deliberately resizing stuff because well partly because you know the model was as we know it was trained on 512 512 and it would be upsized into its original size and that's common but the the artifact of the the implication of resizing the labels is that you might get some interpolation uh strangeness uh in those areas that have been resized near the boundaries of the labels so um this this basically goes through a process of seeing okay okay what does that look like in its sort of raw form and you can see that there's a lot of sort of high frequency noise in these labels near the boundaries because of that resizing that's gone on it got resized down to make a prediction and then we had to resize it back up again in order to make the um to to get it to the same size that we want so one strategy a simple strategy and a common strategy that you're probably familiar with is to use a filter you would just use a filter to pass over that and sort of smooth out some of that higher frequency noise and so what we're really interested in is then preserving the actual boundaries of those labels as best we can but it's a fairly naive way uh approach because we don't know what size to use and that size might actually be different optimally for different images you know especially if you've got a different perspective um but also because of the just the very specific nature of how boundaries might uh interact at their sorry how classes might interact at their boundaries so that you know that the the median filter that I've implemented here I just took a number 15 and it seemed to work okay for that image but I guarantee that that number wouldn't necessarily work uh equally well for all images and really it's not a very sophisticated way of dealing with the problem because it's not learning anything it's just smoothing over where the labels are but if so a better way um to deal with this problem in my opinion is is this is the use of this condition around the field and that sounds like a very complicated thing and it is but um it is a very powerful algorithm um that gets you um much uh that it's very useful for post-processing um your um your model outputs your segmentations for both binary and multi-class segmentation problems and for models other than unit and for model approaches other than merging binary classes like what we're doing here so for example many many of the state-of-the-art neural networks out there for multi-class segmentation they have some form of a condition around the field in there in there in there in there processing often and that's often this the very last thing that it does um in order to just tidy up that segmentation on a task by task or task specific basis and what I mean by task is a single image so what it takes is it takes your your bad labels which could be your ground truth if they've been resized or it could be a model output if it has some error and it and it tries to figure that out it tries to it tries to relabel those portions of the image that have been labeled incorrectly and the way it does that is by basically producing a it uses a machine learning algorithm to figure out what the likelihood of every class is given the information that has been presented to it um there's many many details about this algorithm that I don't now have time to explain but I will encourage you to read the paper that I wrote on this topic um a couple of years ago um that really demonstrates how powerful a conditional random field approach is um for cleaning up the these these types of data and and these types of segmentations and the reason why it's so powerful is because it uses both it takes your your label and your image it it creates a probability distribution that is a generating probability distribution so it's able to evaluate px given sorry px y px comma y the joint distribution of the image and the labels together and it does so by essentially passing a kernel across the image that evaluates the spatial structure of the image features with respect to each class and also the color space structure associated with uh with of every class and and their features so it's a very very powerful model and the math behind it is very complicated but that doesn't mean that we can't use it if we don't understand what it's what it's doing it is controlled by a few parameters that end up being hyper parameters that we have to decide and I've given you some uh numbers uh in here that seem to work pretty well for most cases so I would encourage you if you are interested in this to start with that paper and other papers that have used the conditional random field and then start with this function to explore some of these ideas but but also be prepared that uh these parameters might actually change depending on your specific uh needs and your specific case um and apologies for having to do this quite fast at the end but I did want to get this uh get to the end here so what we're looking at um above here this was just uh a few examples using the median filter and I wanted to to sort of contrast that against the uh the same the same thing but using the condition random field instead and so what it's doing is essentially passing these passing these kernels over the image uh essentially generating probability distributions that um that convert features into classes and then uses that to sort of evaluate how likely every label that we have given it is and it may make a decision to unlabel and relabel it um if it feels like that's a mislabel and then so that's a very powerful thing for both um for cleaning up the segmentations that might be the output of a model but it also conceivably is a good way to clean up your label data that you give it in the first place if you're going through the process of manually annotating imagery with polygons uh you might consider using a model like this in order to actually generate fine scale labels from the image but that's that's going to be something that we can talk about um after the class so here I'm running through and I'm essentially running this uh this function which is implementing this conditional random field on every predicted label and also every true label and I'm doing on the true labels because there could be some error in those but also for the most part because they're being resized and so there's some interpolation artifacts there and you can see that in the end it doesn't it doesn't a reasonable job um at that sort of dealing with some of these um some of these labels and combining um the labels in such a way that you don't have to get involved or set thresholds or or do anything uh super specific um in the decisions that go behind combining labels where they overlap so that was a super super super fast run through of that tutorial because um we spent more time on the previous one but hopefully I may have lost a few folks on the way there but hopefully this is something you can circle back to you'll have my email address I'm very I'm very happy to answer emails um on these topics I am going to keep these uh these two um these two websites going as I said um I'm going to return to the questions now on the chat just for the last few minutes here before we have to wrap up in it um and but if you have any questions about anything at this point then um please let me know oh actually I should mention so you can um if you don't know this already you can download these as python files or as um ipython notebooks or jupiter notebooks that you can run uh you can run you don't have to run these on colab you can download them and you can run them on your own machines um if you need help with you know setting up the the the the environment then I can sort of email that around um after the fact or I can even update the presentation to sort of explain that um there's a lot of detail in here this is necessarily something you're going to have uh you're going to have fully full grasp of right now but I was more concerned about giving you something that really worked and something that you could build off of uh so even if your python skills aren't sufficient yet to to understand everything that's going on here and if you're understanding of machine learning it's insufficient to understand everything hopefully if I've at least demonstrated that this that these sorts of things are possible and hopefully you're able to take these workflows and and use them for your own purposes okay so I'm going to go over to the the chat real quickly and see if anyone's got any more questions yes restart my kernel was a good idea that I didn't think of until too late there is a recording um I've linked to some resources in the website that you can start from and they have they have links and textbooks from them uh the website I'm talking about is is of these websites there's links in there um let me see what else we've got here um we have to downsize the images because they have to fit on our hardware so a hardware in this case is gpu but if you had a limited ram then your cpu would struggle too so that's the main reason it's really just dependent on on the uh the specific computing power that you have um the github yes there is a github repository um I can send that around that's so these websites actually they are github repositories they're they're sorry they get lab repositories because github does a better job but or it's traditionally done a better job at continuous integration so I could just push changes to this and it just uh automatically um updates that but I'll send that around um and chris asks is there's if there's an environment.yaml file there isn't but I can certainly make one and share that around and I'll actually I'll end up probably uh just adding a last page to this website that has that you could download or something or has as an explanation um but we're at we're now at two o'clock so I think we're gonna have to call it hopefully you got a lot out of that um please stay in touch thank you so much for for joining me today and yesterday it's been a blast um I'll stay on the line for a few minutes as long as csdms allow me to and I'll continue to ask answer questions that you have um but I imagine at some point Chad is going to just drop he's going to uh kick me off at which point we're done but uh thanks so much everyone thank you Dan thank you Dan thanks really nice yeah good really welcome thanks bye bye hey Dan hey um I'm just curious if there's any way to look into any kind of inference that the model is doing and how it's making some of the classification decisions because it seems like we're losing some inference yeah absolutely I mean what what we've done here is um a a four hour deep learning class that could easily be four days or even four weeks and and so some of these things I've obviously brushed over but that's a great question so I'm gonna go to my my um browser here oh you can't see my screen anymore the um the the technology that you're after is called class activation mapping and what it does is it shows you a heat map of where in the image is important for every prediction so if you're looking at trees and it's classifying as trees then your heat map should show up as relatively high values over those trees for example uh we didn't have time to cover it in this class but there's a number of um keros implementations that you could use uh that you could find online that you you could easily build from if you if you had the if you had the patience okay cool yeah I can do that googling yeah so look look up cam class activation mapping and grad cam gradient class activation mapping and it sounds really complicated but it's not really that it's not super complicated it's at least not it's not very complicated to implement with code okay awesome thank you so much again you're welcome hi Dan uh thanks for the lecture we really enjoyed it so question can we can we actually apply these units to classify difference uh vegetation types um like with multispectral images so not only rgb uh yeah you could adapt this to multispectral images um so in in the you'll see in all of the lines of code where we actually initiate a model we've got the size right and the size we used was 512 by 512 and then I added three to the end right and I added just those I added three bands to my model input because I had an rgb image and so it's three bands but if you had a four band image you could just add four and it would do the same or if you had a five band image for example you could add five and it would it would do the same the I'm not going to say that it will work the same or function the same or have the same accuracy or anything like that because that's going to depend on your data but um and your and your classes and various other factors but in theory and in practice yes you can for sure um there's there's so many examples of that if you if if you look inside the remote sensing literature um search for hyperspectral units or something you you'll already see that a number of people have implemented things like that cool those verifies thanks you're welcome alli under has two questions and they're in the chat oh okay let me see those oh sorry I didn't see those um different sizes right yeah so if you've got these big format images um then for sure you have to chop them up into this instead of the same size um subdividing into smaller windows yeah so I think um that's definitely how you deal that um oh right and the apologies yeah I was meant to get back to the lumping and splitting but obviously my technology failed right in the middle right in the middle of it um lump yeah that's a that is the golden question right there um that is very difficult to answer so the question is do you lump if you have classes that are the same uh do you try and lump them together or do you try and maintain the splits as best you can and unfortunately that really does depend but that's gonna that's that really does need experimentations trying to figure that out that one out I've I've gone through this many times um in my own research where the initial set of categories that I thought were going to work well ended up having to be modified based on initial trial so I'd go through the process of labeling using those images using those classes I generate a small set of labels and I would train a model just to see how well I think it might do and at that point you know it becomes clear that you maybe have to combine similar similar classes if they're being mispredicted um in general I think you're combining more than you're splitting but it really does depend I think you know it really does depend on on your data and how resolved those features are within the data and how different those features are among classes if you had two different vegetation types that are basically exactly the same then you know if for all intents and purposes you know they have the same range of colors they have the same uh general texture or whatever then it's going to be difficult if there's no if there's no really strong spatial zonation signal in there or any really other sort of uh covariate that it can sort of use in order to make that call a lot of this stuff as well like I should say is you know it's as much than art as it is science you have it's it's numerical modeling but it's it's a very different way than you may be used to and you do you do have to develop an intuition over some of these things and that really it does come through experience but especially when it comes to to whether or not you you lump or split classes I hope that I hope that answers the question that's the best I can do I I think it does um and I see Will has another question also in the chat um the variation of your size or perimeter complexity matter with your with your labels um yes it does uh the perimeter is is obviously a big one because it needs it's it that's the hardest place to actually label because right of the boundary of two things um because the you know that that boundary the the location of that boundary and the crispness of that boundary gets modified by the model so your features have to be spatially strong enough to deal with the fact that you're sort of like you know you have these two adjacent things and the size matters because of the the class imbalance problem thanks so I don't know if my mic is on but um so with the size then is it good to sample across a range of sizes like let's say you've got sediment different patches of sediment of the same composition that um are different sizes um you presumably then you want to capture all of that that size range as part of your labeling process um so when you so what do you mean exactly so the size the size of the image or the size of the oh sorry the size of the sediment patch um so if um you know like I say I work with gravel bedded rivers but sometimes you get certain environments you get um the femoral deposits of fines and okay you know they might be two square meters or they might be 50 square meters um so as far as defining your labels then do you want to try and capture for that size variation with your labeling um I guess I still don't quite follow I mean I definitely see I see a situation where your gravels have in field but I don't understand what you mean by the size of the label like that what you're labeling is what whatever size it is you're labeling it right so do you mean the image resolution oh the polygon yeah you'd want your polygon to be the same you'd want your polygon to completely cover the entire thing that you're interested in so in your case it would be gravel you know you'd want your your polygon to to cover the entire gravel patch but your case though it's going to be more you know distinguishing between different grain sizes is going to be tough because the um because of all of the downsizing that happens in the imagery potentially so you'd want to keep you'd want to split your image up if you had a big large image you just I think you should just split your image up into smaller images but you you preserve the size of the the actual size of the pixel um and then when you're labeling you're labeling on those tiles and you can either decide if the tiles are really small you can either decide to just give it a single number value like that's gravel that sand that's silt that's gravel that sand that's silt or you could actually go through and you know say do the polygonal labeling on those smaller tiles but if you want well send me an email and we can pick this up over over an email conversation yeah yeah sorry to tie it up oh that's okay I just I just don't know how long Chad and Albert here have got uh that's all I I got one question but I see Alejandro has uh one more question about splitting images and if you need to have an overlap on those tiles right right so yeah so as I said um probably you probably missed it because I was saying it quite fast but but a good strategy is exactly what you're saying to to split the late the images up and the labels up uh such that there's a big amount of overlap because then you've got more to deal with you've got more computation that you need to do because you know if you've got 50 overlap you're essentially doubling the amount of work that you need to do uh but right as you say the edges used with the labeling if you have multiple versions of that label for every pixel then you've got more information to make a decision about that decision with and I don't want to hold you are too long then and that's okay this was awesome and I've always always want to you know get more into machine learning so so maybe this this will push me over the edge in doing so um I'm interested in uh detecting water which is pretty easy I think only for satellite imagery so we've done some simple stuff on uh uh modus but I want to use fears for example now I was wondering if if there are some good labeled datasets out there already for satellite data that you can point me for water yeah there is there's a lot um let me have a look here uh I can't remember exactly what it's called but um I can tell you this real quick so I can tell you that the these models work well for water I've I've put together a class um for detecting lakes and that worked pretty well from Sentinel to imagery um I also know that I've used this model pretty it's quite effective of pulling out um for my personal research I have to pull out the land from the surf zone from the ocean sea so it's pretty good at pulling those out as well so that's a form of water detection yeah um there I'm just trying to look up here a I'm trying to find a link to something that I found that had it was essentially a global water polygonal dataset um I'm just struggling to find it because in the end what we would like to do is you know every day there is somewhere on earth flooding and right now there's yeah because there was a dam bridge right and there's this this cyclone hitting Bangladesh um so we want to capture those events and and provide flooded area or flood extents to the various relief agencies and we do that right now already but I think we can optimize this so I want to do it do it for optical data but in the end I would love to do it also for radar data as well but that's a whole different category I think yeah I mean I I think um I'm sorry I can't find this this dataset I do have a copy of it so if you send me an email maybe I'll just send an email later on um but that's what you need but then going back going back to what you're saying yeah I think um I'm fairly confident that this model would get you at least most of the way there um for detecting water I'm not too sure how well it would transfer you know between different types of satellite imagery for example like you know there's like you know different bands um or different sensors yeah but I would I imagine that it would it would work pretty well for the case that you're describing I don't know about you know really super specific cases like really optically clear water or very shallow water or you know but I certainly imagine deep pools of water that are that are optically opaque I imagine this is going to work pretty well with okay okay that's cool yeah and definitely water you know there are certain areas in the world where the water is so sediment rich that it's yeah its color is you know too much brownish instead of dark black and that it might yeah yeah but even but even in that case so when when I put together a course that looked at lake detection there was a lot of muddy lakes and a lot of really dry lakes a lot drying up lakes um you know the aral sea and places like that which which you know very the the color signatures of all of these different lakes across the world were very different but the models still managed to be able to reliably tell where the where the water was so cool and I don't know if that would like necessarily transfer to like the real extreme members of the of like water bodies like hyper saline lakes or hyper turbid lakes or hyper turbid rivers but I think for the most part I think it would it would work okay cool cool okay I will follow up with an email and okay cool that sounds good this was awesome then thank you very