 Hello everyone and welcome to Introduction to Nuclear Segmentation with Stardust, which is a deep learning based method for nuclear segmentation in 2D and 3D images. And before we go into the details of everything, I just want to point out, we provided a webinar material page, which can be found in this URL at github.com or by a new bias Academy Stardust. So please everyone just click on that where you will find the slides of that, the whole webinar. We will find Google to call up Lang's, Jupyter notebooks and everything that we do in this webinar, you can follow along or do afterwards. So for instance, if you have a specific question to a certain slide, you can find the slides there and point us to the slide number, for instance. The second part is, as Julian already mentioned, we're four people, it's me, Uwe, Oli and Sean. We're all very versed in Stardust and for instance, Uwe Schmidt is the co-developer and maintainer of the project. So I think we can hopefully will answer all your questions. We have two extra additions to the team, two late additions of Raghulani and Daniel Saats. So thanks very much for joining us and helping us with answering questions. OK, so the outline of the webinar is followed. So first, I'm going to give you a short introduction to nuclear segmentation in general and the method Stardust in particular, and this will be a little bit more theoretical. So it provides you some of the basics of deep learning base segmentation and detection and specific focus on the Stardust method. It's going to be a little bit more theoretical. Then we'll have a question and answer session. So in the meantime, all the questions that you're going to answer in the Q&A panel will be created by the panelists and we're going to try, we're all going to try to answer them in this section jointly. In the third part, this will be a little bit more practical part. There I will speak about how to use Stardust for your 2D or 3D segmentation problems. We're going to see how we use pre-trained networks and I'll give you a quick introduction to how you train your own model. The fourth part will be done by Oli who is in an image facility, core facility at EPFL and we'll speak about how to use Stardust in the core facility. So even more practical experiences and I hope everybody will enjoy that. At the end, there will be a final second question and answer part where questions that arise until then will be then jointly answered. Okay, so let's jump right in. So what kind of problem we're dealing with? The problem of nuclear segmentation in microscopy. And the problem is typically that you're given a microscopic image where you have stained or labelled nuclei and this can be 2D or 3D images and the typical problem that arises in microscopy or microscopy of biological samples is that you want to create what is called an instant segmentation of the nucleus. So you want, given that image, you want a label map or a segmentation where each of the cell has a unique identity. So you want a segmentation of the outline of the shape and you want to separate nearby closed cells and you want to identify each cell as individual. This is of course very important for things for biological questions such as cell phenotyping for asking questions about the shapes, the sizes of individual cells, for mapping back gene expression differences or mapping back gene expression per cell and get a feeling for the expression differences between cells and for things like lineage tracing or tracking development organisms where you first have to identify individual cells to answer questions about where they will go in the time when they develop or when they re-organize their shape. So this is a very fundamental problem in microscopy or in biological applications that rely on microscopy images. And to give you a short overview of the typical data that we can find in microscopy, I'll just give you two examples from fluorescence microscopy. So these are two of the examples of human stem cells and of lymphomatic cells. So where the nuclei is labeled and then which is acquired thereof and you can see typically these images are very, very crowded. This is more pronounced even in this data set. So each individual little thing that you can see here, little blob, is an individual cell nuclei and of course segmenting them correctly such that neighboring cells are distinctly segmented. So segmented is very challenging in these cases. But it's not only confined to the modality of fluorescence microscopy. The same applies for other types of microscopy. For instance, histopathology, these slides where we typically have H&E staining and RGB images. So images like this where you want to segment the individual nuclei, get the shape, get the color for diagnostic purposes or for bright field images like this one that's stem cells, mouse stem cells that are actually evolving over time. So you see there in the petri dish and of course, like just knowing where each cell is and moving over time is a very interesting biological question. This only was 2D images, but the same goes for 3D images. So most modern microscopy modalities allow for 3D imaging of samples over time. So you have images that are like this. So this is evolving in parihala. So a crustacean that was acquired spinning this confocal microscope. Or this is warm as the elegans a part of that. And you see each individual little thing is again the cell nuclei in 3D. And segment them correctly in 3D is an extreme challenging task, but very important for biological purposes. To give you a short background, stepping a little bit away for a moment for microscopy, I want to give you a short, very short introduction into what are common problems relating to segmentation and detection in computer vision. So not confined to microscopy, but computer vision in general. So where you are concerned with natural images of cats or dogs or something like this. And there are two main categories of things that you can discern. So first, you're concerned just with a single object. And there are three subtasks that can be mainly categorized into image classification, object localization, and semantic segmentation. The first one, image classification is probably the most easiest. So just given an image and you just want to classify that in a certain category. For instance, this is a cat or if it would be a dog, it would be a dog. So you have a different discrete set of labels and each individual image just gets one specific label. So the whole image just gets one individual label assigned to. The next little bit more involved thing that you or tasks that you can feel yourself concerned with is object localization. So you not only want to know that there's a cat in this picture, but you want to have a rough outline. And this is typically done in the form of a bounding box that kind of tightly packs or tightly squeezes around to the object of interest. So you not only want to know that it is the cat, but roughly where this cat is. And the third and even more complicated task that you could think of is something called dense or semantic segmentation. In there, you actually not only want to have a rough outline of the cat, but you want a very precise delimitation of the cat's shape. So every single pixel will receive a label. Is it belonging to a cat or not? So you end up with a very precise mass. So in this case, every single pixel gets a label. So it's kind of, that's why it's called dense segmentation, because every pixel votes for a certain class. So you end up with an image. Whereas the output of this and this would be either just a discrete label or a bounding box. The output of that would be a complete image of labels. And this same can be done not only for single objects, but as well if you have multiple objects. And in the case of object detection, this would be concerning yourself with finding all the objects in the image and putting a bounding box around that. So a very coarse grain detection of objects. And something what is called an instant segmentation, which is the generalization of that. Is not only find all objects, but delineating them in very precise manner. Such that you get all the instances in a very precise way. And this is called instant segmentation in this case. And all these methods in recent years were the most successful error times to solve these kind of tests are typically can be categorized in something what is called supervised deep learning. And I think everybody probably already heard of that or is aware that this exists. But just to give you a very, very rough reminder of that. So typically you formalize your task as a prediction test with a deep neural network, so the input would be a large set of input images. Or the input would be an image. And the output would be whatever you're interested in. This relation would be formalized or would be, is given by something that's called a deep neural network, multi-layered network. And you typically optimize the parameters of this network by training on a large set of annotated ground truth or training data. So this is called ground truth. And this works like you assemble a lot of inputs, you assemble a lot of outputs. And then you optimize the parameters of this neural network in something what is called network training, such that you give the best results on your training data. And then you apply that to unseen inputs where there is no existing training data. So it's a very general paradigm, but it's the most successful for these kind of tasks. And these are just two or three instances just to let you know that these names when you see them, they are instances of difference of these subsets. So there's UNED, which is for dense semantic segmentation. There's something called YOLO, which is for object detection. And something called mask RCNN, which deals with instant segmentation. They follow more or less this paradigm. And for all of them, you need a lot of training data to train your network, but it's then very efficient data predictions that are very, very, very good accuracy. In our case, now backtracking a bit, we can now formulate our problem a little bit in the categories we just seen. So we have typically an input image and we can now ask ourselves two questions. First would be just delineating the foreground and the background of the new client. And that is actually then just a problem of dense segmentation. Or you can not only do a foreground background classification of each individual tixxel, but you can then as well aggregate or get every pixel's identity assigned to a certain cell identity. So every pixel here votes for a certain cell, it belongs to, and you end up with what's called an instant segmentation. And this is actually the problem we're interested in. So we're interested in the instant segmentation of individual cells in images where the new client is marked. And of course, the challenges are in microscopy specifically. And this is a bit a little bit different than a computer vision where natural photographies are typically very high SNR and not have a lot of crowded objects that this is the opposite. The opposite is true in microscopy. So you have many crowded objects and noisy images. And to just give you a short example of what are current methods doing in our case for nuclear segmentation. So I imagine you have an image and this is the ground truth, so the annotation. So what could be done in normal machine learning pipelines is either one of two things. First, it's called the bottom up approach where you first segment each cell. So you're first to a foreground background for localization or a foreground background prediction of each pixel. And then you try to separate those into different cells. So you first to a very like foreground background and then you kind of try to discern that into individual cells. So one example is a two class unit and then you can add a boundary class. It would be a three class unit. So these are kind of examples for the bottom up approach. And then there's the top down approach where you first try to localize individual cells. So you first kind of try to separate those cells and then you refine those very core separations into refined masks. And an example of that would be what is called mask horizon. And we've just seen that as one of the architectures used for instant segmentation. And the problem with, and there are, sorry, there are other methods that are in recent years or actually in the last years, in the last two years appeared that are specific for microscopy images. For instance, affinity based methods or embedding based methods. For instance, just recently in cell posts, which is a very cool method. They work a little bit differently. So, but I just mentioned them here because I think they're very promising, very cool. The problem with the bottom up and the top down approach. However, in crowded and very noisy images is the following. So imagine you have three cells that are very crowded. And now imagine what would happen in a typical bottom up approach. So you would first get a foreground background segmentation of that. And then the problem with these kind of crowded scenes is that touching and very crowded object would get merged. And this applies to these classes. So it's very hard to discern very crowded and very touching objects. So you get mislocalizations and the segmentation map would actually fuse. If you now do the top down approach, so you first try to find a bounding box with that try to separate cells. You have the following problem that these bounding boxes are just poor approximations of the actual nuclear shape. So it could be that the bounding box of the central one would suppress the bounding box or overlap the bounding boxes of the neighboring too much. And typically things that overlap too much in a bounding box sense will be suppressed by these methods. So you get only the single cell in the middle and leave out these, which is of course very bad. You get again a mislocalization. And what we then try to think of is a method that kind of elevates these problems and is adapted to the shapes that are typically found in nuclear images. And these shapes are typically round shapes that can be approximated by some round object shape descriptors. And that's what we did. So we did something that is well adapted to these nuclear shapes. And instead of a foreground background mask, we predict two things. So for each of those single pixels, we predict two things. First, it's an object probability that we calculate from the ground truth, which is just the distance of the center of each object to the next nearest boundary. So this kind of gives you a probability of being the center of the object. And radial distances, which is for every point, we pre-define, for instance, 32 different radial directions and ask, what is the distance to the closest boundary? So every pixel has this notion of how far it is away from the boundary along a predefined ray. From the ground truth, we can compute these two things. And the radial distances, I just show three examples of them would be 32 distance maps that can be calculated from this ground truth. So here you see the distance along the right direction. So you would see pixels that are on the left border have a very high value because in this direction, the next nearest boundary is very far away. And then we can just train in your network to predict these aggregated object probabilities and distance maps. This is essentially the first step that Stardust does. Just given an image, we try to predict these sets of object probabilities and distance maps. And we typically use a CNN, a convolutional neural network, such as a DUNET, or ResNet for that. But that's just a detail of the method you could use a lot of different network architectures. That's not important. Of course, at the end, you want individual instances of this nuclear and not just distance map and probabilities. So there is a second step that gives you from the object probabilities and the distance maps to the final results. And this is called a subpolygon selection and a non-maximum suppression. So we construct polygons out of these distance maps and object probabilities for every point and then ask which points overlapped or which polygons that are assigned to a specific point have the highest probability and which points should be suppressed because the corresponding polygon overlaps too much with an already existing polygon. And this is called non-maximum suppression. And this is just how quickly how it works. So from these representations, you would get normally a lot of multiple potential conflicting objects like this. What we do is we keep the most probable object, which is according to the predicted probabilities. And we just set that as a predicted polygon. And then we ask for all the polygons that are predicted but overlap above a certain threshold, we just remove them. And this threshold, this is called the intersection of the union threshold. And depending on the threshold, we get a different set of retained polygons. So if the threshold is very low, a lot of those regional surrounding polygons could be suppressed. If the threshold is very high, just fewer of them would be suppressed and we get a lot more overlapping, potentially overlapping objects, which sometimes is what you want if very crowded, for instance, cell division cases where objects could potentially in a platonic world could overlap a little bit at the beginning and then they separate out. So this can be, this is a parameter that you can select and sometimes this is better than this for a selected task. Of course, not all shapes can be represented by the stardust or the star convex approach. And essentially what we are, the shape representation that we predict is something called a shape representation of a star convex object and only those shapes can be well approximated. And what means star convex? So star convex shape is simply a shape where we can find a point in the center from which we see all the boundary. So essentially we can find points within the shape where we can reconstruct the original shape by just giving those distances. So this would be an example of a star convex. This would be an example of a star convex shape. Star convexity is in general a little bit of a weaker notion than convexity. So convexity means every line segment would be included in the whole cell, which is clearly not the case here. So it's a little bit weaker, but typically it applies to mostly roundish shapes. So these are just shapes that are clearly non-star convex. So here, for instance, you cannot find the point from which you can see without crossing all the boundaries. So here, for instance, there would be a crossing point and the same goes for shapes like this. And this applies, of course, to sort of plasmic markers or cells that have protrusions. And that's why our method is mostly well adapted to nuclei markers because they typically have a round shape. And to just give you a quick example, so this is how it would look like. You get an input, you have predict the object, probably different distance maps. From these two things, you would get a lot of convex polygon candidates, which then would be suppressed with the steps that are outlined before to give the final results. And this is just showing you how the training process looks like. So this is the input, this is the ground truth. These would be predicted instances. So just two of the distance map, it would be theoretically 16 or 32, but I just showed two here. This is the training loss that you typically have in the machine learning pipeline. So you always optimize the training loss, which in our case is a combination of the probability and the distance loss, and that would be defined prediction over time. So at the beginning, you see, it's a very badly segmented, very coarse segmentation of the cells, but we're gonna see this is gonna be better and better during training. So if you look at these cells, so here the ground truth, so this is one of the challenging cases. You see that at the beginning, if I go back at the beginning, this is not disentangled, but more and more later, the more you train this model, the more these jobs get delineated. Just to give you a short synthetic example where it would matter in our case, where our method is really, really good, we created kind of a coffee bean synthetic data set. So we have shapes that are very tightly packed and the bounding boxes of them would almost perfectly overlap. So this is a very challenging case for typical detection pipelines. And if you go through the different methods that I earlier mentioned, the foreground background aggregations, bottom up approaches, they do very well, at the top down, which rely on bounding boxes to vary poorly. So this is just an example of why a shape or approximation like we use is very adapted to this kind of data. And this is another example from the DSP Kaggle challenge of real images. And you see here, due to the crowding of the objects, the bottom up approaches, they fail at a lot of those instances where a stardust is doing pretty good. Not perfect, but it's doing pretty good. Just to give you a short introduction about a currency measure. So how do we find out if our segmentation result is actually good, even if we have ground truth? So how to compare actually predictions and ground truth? So assume you have a ground truth, you have ground truth objects that are here delineated in these blue little shapes and you have a prediction. So how to compare and find an accuracy measure for these two? So what you typically do is you first try to match corresponding prediction in ground truth objects in a called IOU match predictions. So for each of the predictions, you ask what is the closest in terms of overlap with the ground truth and then you just color it in the same color. So this one would match to this or kind of do the same way. This one would match to this. So it would color be in the same way. So we have kind of a match prediction. Some of them won't be matched. So for instance here, this one has no corresponding part. But after this matching, you then can compute for each of the match cell. You can compute what is the overlap to the ground truth and then decide if this overlap is certain above a certain threshold, I count it as a true positive. So this is a correct detection. Of course, if something counts as a true detection or a true positive, this of course depends on the threshold. But we will see later how this affects the accuracy measure. Once you have that, you can color every of those instances if they either got matched correctly and they're threshold and the intersection over union is higher a certain threshold as a true positive. You can count predictions that for instance, this predictions that still overlaps with ground truth predictions, but not substantially enough, you count as false positives or missed detections. And then you count ground truth segments that were not found in your predictions, you count as false negatives. And from these three values, you can then define accuracy measures that are classically even like classical machine learning or classifier metrics such as precision, which penalizes false positives. So a hundred percent or one position of one means there are no false positives. So all of your ground truth segments got matched and know all of your predictions are matched to a ground truth, but it could be that they're missing. The other one is recall. So it penalizes false negative. So it's penalized as missing ground truth. And then there's something called the average precision, which is kind of an average of those two, not really, but it penalizes both of them in the same way. And as I mentioned this, this matching, what counts as a true positive kind of depends on the threshold. So for a very low threshold, it's very conservative. So almost all of those predictions will match some ground truth label if they're just slightly overlap. So your precision metrics and your average position accuracy will be very high. And for a very high threshold, almost all complete only complete overlaps are counted as true positive. So for a very high threshold, you're penalizing almost everything that doesn't overlap with the ground truth object. So typically most methods, if you plot them the average position over the threshold, we'll see a decline. And for higher thresholds, this decline is actually due to that your shapes that you predict are not pixel wise perfect. And if you look at this corp here, you see stardust in red, though it's doing really well at the beginning. So when the threshold is low, so it's really good in delineating objects, but then it suffers a little bit more at the higher end, which means that the predictions that we get are not pixel wise perfect, which is expected because we only predict a finite set of rays as the representation of the shape. And for instance, in this case, 32 rays cannot fully capture the composition. This is an example of crowded lymphoma cells. And so just to see, so a lot of those cells, there are 20,000 cells from collaborators and you see that most of those cells get delineated in the correct way. This is an example for misopathology, so H and E stain. Here the same, just a different modality. We want to find a nuclei for each of those within the whole slide. And even on such large images, it works pretty well. So you can see that it is able to segment most of the nuclei quite correctly. So in this example, there were 350 nuclei in total found. And I just write here, six million candidates, these are the two different candidates that were in the first step predicted and then get more and more suppressed in the second steps. So these six million candidates, only 315 nuclei will survive. And this is just an example from the biofield images here. These are stem cells and of course you can segment them at every time point and then just plot them over time. And you see it does a pretty good job. There's one little thing in the middle that is classified as a cell, but it's hard to see whether that would be not the case. So these are just examples from 2D. Of course you want, sometimes you have 3D images. So how to extend that for 3D images? And you can do roughly a similar approach. So instead of a 2D image, now the input is 3D and the segmentation is a 3D mask. You can do, you mostly can follow the same recipe. You define object probabilities that are just the distance of each of those center to the boundary. And you can find the distance maps or start and mix distances just that now they're not in 2D but they delineate a shape in 3D. So they are raised in 3D that point towards the closest boundary or that's point towards a specific direction and give you the distance along this specific direction in 3D to the closest boundary. And again, you just densely predict these things and then do a non-maximum suppression at the end which is roughly more or less the same conceptual stuff as before and 2D is just computationally a little bit more involved. Additionally, considerations that you have to, that we had to solve was to make a ray choice. So in 2D it's quite obvious how to distribute the rays that is not as obvious in 3D and we do that with something which is called a fever natural lattice and you need a little bit more points or 96 or 72 in this order. And typically you have data on isotropy so in microscopy the z dimension or the pixel space in z is typically different than from x, y. So you have to take care of that and then on maximum suppression of large sets of polydries typically computationally way more involved than it is for 2D. So these are examples in 3D. Obviously elegance data set and this is the same for a data set of a crustacean that we have seen at the beginning. Okay, so this was the more theoretical part in the introduction to Stardis and yes, if there are questions, please let me know. Okay, thank you very much, Martin. So we've been curating all your questions as fast as we could. Thank you very much to all the panelists for all the work that you've already been doing. So yeah, we've liked a couple of questions, Martin, for you to answer in the QA. The first one has been asked by Xareny Kalindo and asks, which kind of data is the ideal for Stardis? For example, what is the bit depth size? Is there a particular multiple of some number as is the case for unit input images? Do they need to be normalized, et cetera? So in terms of the bit depth and the actually shape of the data, it doesn't really matter because we make sure that this is kind of done correctly. And I will mention that later probably, I think everything of that will be answered later in the technical questions or in the more practical examples but mostly when it comes to the shape and the bit depth of the data normalization, there is currently no limitation to that. The only limitation is more how, what kind of sharps morphology you're interested in and they have to be more roundish objects. That's the only limitations. Thank you. Another question is asked about the Colab notebook. Is the version available in the Google Colab capable of performing transfer learning or does it need a heavy training process? So currently it only demonstrate the Colab notebook on the train demonstrate what would happen if you train from scratch. So we haven't implemented transfer learning yet mostly because it's not as easy to give precise and good defaults for transfer learning. And I think you will probably speak about that in your sector a little bit more, but it's typically not something that can be done on the click solutions. That's why we didn't provide it yet. So we would have to know a little bit more than that. So currently it's only training from scratch. All right. Another question from Stefania Tavano asks, how sensitive is Stardust to big differences in intensity and or size between neighboring nuclei? So the first one, it's if you train it correctly and I'll speak about that later too. So you can the intensity change because you do some sort of normalization. Anyhow, it's pretty robust to intensity changes. What the size to of nuclei is concerned. It's typically a little bit more sensitive if your training data only X or if in your training data only cells of a certain size exist and you suddenly give size cells of double twice the size. It will probably be performed badly, but you can always reshape or rescale your images such that more or less are similar to your training data. So but it's a good question. All right. Thank you very much. Panelists, correct me if I'm wrong. I see no other questions that will be curated during this QA session. A lot of questions have already been answered directly online. So feel free to scroll through those and hopefully maybe some of your questions have been answered at the same time. Then Martin, the floor is yours for the demo section. Perfect. Okay. So now we come to a little bit more practical session which is how to use Stardust. And so the main entry point is our Python library which is called Stardust and can be pip installed like this. And so this is the link. You'll find that in the overview GitHub section too. There's a link of that. And what it provides is a Python library for training and prediction of 2D 3D images. It has a neural network back and via Keras and TensorFlow via CSB Deep which is our package that we kind of maintain too. It has a lot of sensible training defaults. It has multicore and non-maximum suppression type prediction. So a lot of these things that you would have to do otherwise by yourself is already included. It has these segmentation and detections measures that I've just mentioned and it supports model export to Fitchy for later for prediction. So this code is the main entry point for training and prediction and it would look like this. So it's very succinct. You can construct models quite easily and train them and predict instances with that. So the second one is the Fitchy plugin that is only for prediction. So at currently we don't support training yet but sometimes it's easier and already to apply an already trained model by the software Fitchy which is a very, very cool software that is used throughout life sciences quite heavily. And there you can, it's a GUI based solution we provide a plugin that allows for the prediction of 2D images so far, so far on the 2D by already trained models and we provides pre-trained models too. So you can quickly try it out if it would work if you have 2D images on your own on data that's probably the best for people that don't want to set up Python. So you can just try it out on your 2D images with the pre-trained models that we provide. And it even runs on the CPU fast enough typically because prediction on the CPU is for 2D images fast enough. So these are the practical aspects of the software, practical aspects of what you should do with your own data. So there are two questions that you have to answer or there are two use cases definitely. First, if you have 2D data and data that is similar to one of the pre-trained models that we are similar to the pre-trained model images that we provide. And this is HME and fluorescent nuclei. If you have that, you can just try the pre-trained models on your data. If that's good enough you can just directly predict with either Python or Fitchy even on the CPU typically fast enough in 2D. So this is the easiest workout that you could always try. Just try it on your 2D data with the models maybe easiest is even from Fitchy and just try if that works. If that doesn't work or if your data is 3D. So if your 2D data is very dissimilar to the pre-trained data and I will show examples of them later or you have 3D data because we currently provide don't provide 3D pre-trained networks you have to go through a little bit more painless painful approach or aspect of it to generate your own annotations and train your own model. And this typically has to be done on the GPU to make it fast enough on a workstation. So it's a little bit more involved but it's doable. And then after that you have trained your model you can then again predict it on your data and this again can be done from Python or from Fitchy on the CPU or the GPU. So I talked about the pre-trained models and these are the 2D pre-trained model we currently ship within the package. So the first one is trained on the license microscopy images of nuclei one we already seen. So it's trained on this cool curated data sets by the Carpenter Lab and they really took a lot of effort into annotating images. A lot of images, a lot of annotations and it's a very, very good starting point if you have nuclei images with fluorescence modality. The other one is histopathology RGB H and E slides. So these are colored three channel images of histopathology slides and there again we provide a pre-trained model again relying on the annotations done by amazing work by other groups. How would you call them? So from Python you would call them just by these simple steps. So you just import a 2D model and then you just call it from pre-trained and you get a list and this and this will be automatically updated and you see the available models. So we have a Fluo versatile fluo model, fluorescence modality of the versatile H and E model and then we have some paper demon models from the paper just for purposes of reproducing our sense. So these are the two ones that would be probably the most interesting for you and you can just load them with the string and just predict instances on the image, that's all. And the same goes for Fiji. It's even more simple. You have a quick button down menu where you can find all the different models and when you try it on your own data, there are a few tricks. If it doesn't work, what you can try. For instance, in a nuclei example, try different scaling. So sometimes if you see your nuclei are too large try to scale it down to make it a little bit smaller and make it bigger depending on if you see over or under segmentation. You can do some tricks. If you have an RGB image and you see if you convert it to grayscale, it looks like a nuclei marker. Then you can just apply a nuclei model on the converted RGB image and you can play around with these parameters that we provide that give you a little bit more flexibility in terms of how many objects to retain and how big the overlap should be. Okay, and I just want to give you a quick demo how this would look like in Fiji. And with for that, I closed it. Finally, some magic Fiji is already open. So, okay. So you have Fiji and you can, I already installed Stardust, but if you wanted to, it's rather easy. Again, you find the links on the GitHub page for how to install it Fiji. It's essentially to update sites that you have to activate and this, of course, is now happening. Someone updated Fiji in between the last 10 minutes. Oh, my Fiji is up to date. So if you go to update site, you find two packages or update sites that you have to activate. The first one is CSB Deep that provides a network backbone that you would have to activate. And the second one is Stardust proper. And that's the package. Again, just an update site, click these two packages or these two update sites and it should be automatically installed. What you can do then, and I just demoed it with some simple data. So here, for instance, I have some very noisy 2D images of Nuclei. And what you would then could do is just open the two plugins and your Stardust, the open Stardust. And you get here a list of the different available models. In this case, we're probably gonna try the versatile fluorescent Nuclei model. So here, these are typical values that you don't have to adjust because they're typically good for default. You can even load the default thresholds from the selected model by clicking this. So you see this changed these two things. But typically the values that are here are typically good enough. And then you just hit okay. And this is running on the CPU, but it still should pop up quite soon. You see it's running. The Roy manager will open and this is the result. So you get, this is the label mask of that input image. All these labels are in the Roy manager. So they're available as regions and you can do the measurements. You can overlay them with the image. So you see it's doing a pretty good job on this very noisy images. Okay. So the same goes for other data. So for instance, pathology slides. So again, open Stardust plugin. Now you probably select a different model, not the fluorescent model, but the H&E nuclear stained model. Again, you probably want to set the thresholds according to the model and then hit okay. And it should segment the RGB image. Hopefully just fine. So again, this is running on the CPU. If you have a GPU then this would automatically use the GPU and everything would be a little bit more faster. But you see it's, it was fast enough and it does a pretty good job on this, on this very tightly packed images in the middle. So it works on time lapses too. So I'm not going to show that now, but if you have a time lapse to the image, it will do every time lapse individual. You still would have to do a tracking, but I'm just going to show you quickly what would happen. So if you see, if you would run it on this type of images and you would use the fluorescent nuclear model, what you would see is that the nuclear here are relatively large, they're pretty big. And this might be that the pre-trained model doesn't work as expected. And hopefully we will see that. Yes, so you see it's doing a lot of, a lot of small things that are not overlaying perfectly with the, not at all with the nuclear. And this is an indication that your cells might be way too larger than it was seen in the training data. What you can do just as a quick, a little quick fix, you just try to scale it down. So for instance, let's do it this way. And just recall, recall the plugin. And voila, you get a pretty good segmentation of those nuclear, that's pretty perfect. Of course, this won't work always. And if it doesn't work, you have to do then other things. And I'll just going to show you what you would have to go through if that wouldn't work. So if your pre-trained model does not work on your 2D images, as I just shown you not, but if it wouldn't work, or your data is 3D, you have to train your own model. And this typically involves generating annotations and to train your model on those own annotations and then predict on your data. And I just shown you some example images from the EPFL, from the Monari, from the EPFL, where the built-in model is doing not really a good job. So it finds some of those nuclear, but in this region is most of the nuclear mist. So it would make sense to retrain a model or train a model with your own data. And what you need is then you need annotated images, label pairs, you need a GPU workstation, Python. So you need our Python library and you need some time. So training from scratch, typically in 2D, it takes up to two hours. On 3D, it's a little bit more involved, it takes 12 hours. And that's what you have to go through. The biggest part is probably the training data generation. So you have to actually assemble corresponding image mass pairs. So in 2D, that would be these kind of images and the mass could be a densely annotated image of the same size, where each of the different cells would have a unique label that is an integer. So you would have to produce these kind of images. It's even more complicated in 3D because you have to do that for the full volume, such that different C planes where the cells, pixel belong to the cell instance would have the same, so it has to be consistent across C planes. Typically you have to do some crops from different regions and time points of if you have a developmental time lapse, you would have to do that for different regions where there are different morphologies and different time points where the morphology might change and the size should be at least 128 cubed squared or 96 cubed in 3D and roughly at least 10 images. That's our guess for 2D and for at least four stacks in 3D. So that's quite some annotation work, but if you have a project that is long running, it's typically worth the effort. So it typically takes maybe one, two, three days to label them and then if you have experiments run with the same specimen for a longer time, then it pays off to do that. We would have to aggregate those images in the following folder structures. You would have training images that are consisting of the training images and test images. So here these are the image and mass pairs that you would train on. To validate your model, you would then apply that to images that you haven't seen during training. So just to validate that you don't overfit. And again, you need dense labeling these kind of images and non-overliving tests and train sets. How to do this annotation? So in 2D, I can recommend these three things to do it either in 3D alone, so you can just draw on the Roy's. You can draw on images and get these regions of interest in the Royal Manager and then convert that to a label mask. You can directly create label mask with a tool, a very cool tool called NetKit done by Matthias Arts from the CDG. Or you use Qpath, which is a super cool bureau for 2D images and by Pete Bankers. And you can draw Roy's and the image and again have to convert it to label mask. So these three tools are available for 3D or at least I am aware of or I'm using and I can recommend. In 3D, this iteration is a little bit more dire so can still use LabKit, but you have to go through each of those slices. So it's very, very cumbersome. And there is another tool from the Salford Lab called Pantera, which is for very large volumes but has a slightly steeper learning curve, but it's a very cool tool that does interpolation for instance of segments. So you don't have to segment every single slide. So I recommend checking those two out. And if you have done that, what you would have to do is you would have to save those data somewhere and use our Jupyter Notebooks that you can find under this link. Again, this is again in the GitHub repo and you would then use one of the 2D or 3D example workflows and adapt it to your needs. Essentially, typically for a first try just changing where is the data directory and everything should run just fine. We try to split the notebooks in a sense of the format such that first we have a data preparation step then the actual training and then the prediction of new images. And I suggest following these steps and they should be quite self-explanatory. One little thing, if you're just a few data and generally I would always recommend is do something called data augmentation and thereby you do during the model training you kind of try to geometrically distort the input images that you already have such to create artificially more trained data by transforming existing into different but plausible versions. And this can be done automatically. We support at least a function that can be used for that and I will show it later. So for instance here, the original images and layman mask would be this, this would be the input that would be masking from that. You can just rotate those images and you don't have to do that on the hard drive but it can be done of course in software in this we support doing this. One other thing is elastic information. So you kind of wiggle a little bit with your existing annotations or you add some noise or shift the intensity. And again, we're gonna demo that again, typically it should be done on your workstation with your own data, but we created a Golab notebook that hopefully just runs out of the box. Again, the link is on the webinar page and it would look like this and we cannot go into all of the details but I just want to show you and you don't have to understand everything of those code lines. I just want to show you how it would roughly look like. So you would have to install some of some packages that typically should already be installed as your image facility or your IT department how to do that and install Stardust. In this case in Google Golab, TensorFlow is automatically already installed. Then you would install Stardust with pip. This might take some time but it should be rather painlessly. Again, if you have troubles, you can go to the GitHub repo and you see these are all the dependencies. So we rely on a lot of packages but they should be automatically fetched. For instance, Keras and CSV deep. And at the end, there should be successfully built Stardust and everything should be installed. And here we just import some things like label image reader and some utility functions. And here we import Stardust 2D and a config 2D object and this is the important line. But again, you don't have to change anything. And for this demo notebook, we just download some existing data which are images from this Kaggle challenge actually that we've seen earlier. So these are 2D images of new client. We just download them. It should work for you painlessly. If you then look in your download folder, it should have the structure that I've shown you. So it's called DSP, that's our folder name and it's split into train and test. And each of those train and test subfolders will have images and masks. And that's the format generally that we expect. And as long as your images are categorized like this, everything should work even for you. So here in this case, we have foreign images. It's quite a large subset of the whole Kaggle challenge and we're just gonna display some of those here just to give you a short feeling. So these are the raw images and these are the ground truth labels. So these are potentially the annotations that you would have to create it and they would look like this. So again, every cell has a distinct label. So the first thing that you can do is you can try to see how well you actually shapes our star convex or how well they can be represented with a star convex polygon. And I always recommend doing this first because it saves you the trouble of at the end realizing that your shapes are probably not as good approximated by our whole approach. And for that, we're just gonna test out some of those rays. So remember the more rays you use, the more finer can you represent those shapes. But of course, the longer training will take and the more involved and complex the model will be. So you have to find a sweet spot, typically 32 is fine. And we then gonna reconstruct the ground truth images with different rays and just see how well the result looked like. So for four rays, I mean, of course, it's probably a silly idea to try to approximate your cloud with four rays, but you see it's not a good representation, obviously, but the more rays you use and this is just gonna demonstrate that that's the case, the more rays you use, the better this approximation become. And we can actually quantitatively calculate that by calculating those accuracy metrics that I mentioned earlier for each of the different approximations with different rays. If you look at them, they should approach one, if your nuclei or the shapes that you try to segment are actually representable by star convex. If this is not the case, you probably should not even try. If that's really bad, you should save yourself the hassle of trying to train one. So here you see, this is a reconstruction score or accuracy and the more rays we use, the better we can reconstruct actually the ground truth. So this indicates that the images we use are actually well adapted to starters. So it works pretty well even for 30, yeah, 40, 30 ish rays, we already get more than 95% overlap. Okay, having checked that, you actually can start with the training. And again, we're just gonna read all the images that we have from the training dataset. So not from the test, that's very important. So train subset. In this case, we have 407 image pairs and we just gonna normalize them and we provide a function to do that in a rather robust way. So your central normalize and then we split them into train and validation. So remember the train and the test are distinct such that at the end, after our model is trained, we're gonna evaluate it on the test set and the test set are images that the model has never seen but to evaluate the model training while it's training, we can kind of artificially create a test dataset or called validation dataset from the already having trained dataset by just splitting them. So we can artificially probe what would happen if the network sees data it wasn't trained on and that's what we're gonna do. So we split our 4,400, 4,700 images into 380. Actually training images and 67 validation images, just gonna plot some of those. And then we can actually progress with the configuration of the model which typically should not be very involved. So it's done via a conflict 2D object which has a lot of parameters that you probably should never change because there are sensible defaults but you could if you wanted to. So this include how deep is your network? What are the patch size that you extract crops from? What are the training losses? How long you train and these, et cetera, these kind of things. Again, probably you don't want to change it, just leave it by default. The only sensible things that you probably want to change is how many rays in 2D you want to use and something what is called a grid parameter and the grid parameter just tells you do you want to have a dense output of every of the whole size of the whole image or is it enough if you just sample every second? So if images or nuclides are bigger than three, four, five, six pixels then this always makes sense and it makes everything faster. So we're just gonna use that. Yes. Excuse me, just so you know, it is 1630. So I think it takes five more minutes. Perfect, just letting you know. Thank you. Perfect. Okay, once we have the configuration we just can actually construct a model and this model will live in a folder called startist slash models or models slash startist and it will use some default values and everything is fine. We're just gonna do another check that is embedded in the notebooks to see if your field of views. So every neural network has a receptive field. So that's roughly, every pixel sees that much of context around each nuclei. And of course this context has to cover a complete nuclei and this is just to check if that is the case. So if your shapes are too large the network would not see beyond the boundary of each nuclei and you want to avoid that. If that's the case, you probably have to downscale the images or make the field of view of the network larger. It's automatically checked. Again, if it says all good, it's all good. Again, we can check what kind of pre-trained models we have. We're not gonna use that now but just as a quick example of how we would check that. Then again, we would do some augmentation and in this case we just do random flips and rotations and some intensity change and we define a function that takes in an image and a mask and just produces a randomly augmented flipped and intensity changed mask. Again, the details are not that important. Just that is something that you can adjust through your likings. If you see that your model is very, very sensitive to certain parameters. Just for instance, if you have artifacts in your image sometimes they're stripes or hot pixels or something you could add that and in that way you could generate artificially artifactual images that still should be segmented. For instance, if a plane is missing or if other imaging artifacts appear in your data and you see your network would be tripped off like this. We just plot some of those augmented images. So this is the same image just different augmentations applied use. So you see the intensity changes, the orientation changes and this is just a sanity check that your augmentation function does the correct thing. And then finally you can actually go to the model training. So having set that all up and to monitor the model training you can use something which is called tensorboard that gives you the loss curves over time and intermediate results. And here you see quick demo is true. I just don't want the model to be run in this demo for two hours. So we just artificially use slow number of epochs and we just do this quick demo. But if you do that it should start the training on your data. And this is hopefully the case. Okay, you see. So an epoch is just the number of steps over the whole data set that will be taken. And we chose 40 epochs. Typically you want to choose way more than this. So 200, 400. And again, the steps per epoch should be a little bit larger but we just do that here for the demo. You would have set quick demo to false if you do it on your own data. And you see your model is training. So your loss hopefully goes down because it's a bit hard to see that in this kind of representation. We go back to the tensorboard in there. You should have a graphical depictation. We'll just make it a little bit smaller of what happens. So if you reload. So here is loss function over time. And that hopefully should. So you see the loss function is the loss or the value of the loss is decreasing over time as it should be. So this is always worth to check if this does not happen, something is wrong. Always check there should be a nice, almost exponential decay. And you can check intermediate results. So for instance here, we would see some inputs. So this is the input the network would get. And you can check what are the intermediate results as you progress with the training. So for instance, again, if you remember, we predict two things, the probability and the distance map. And just for instance, it's the probability of being a center. If you can scroll through the different epochs you see at the beginning, does a pretty poor job. But if you go as training progresses, it suddenly realizes what is the center of nuclei. The same goes here. And the same should apply to the distances. So you see, and this is just three distances. So we color code RGB for three of the 32 distances we choose. And we'll just have an image. And it should approximate what is the ground to that you can see here too. Okay, this is the training workflow. At the end, you can do a little bit more things like threshold optimization and a final prediction on your own test images. I will skip that in the interest of time. And what you can of course do is at the end, what you always want to do is, you want to export your model to Fiji. And this is the function that we provide. So it's just model.exportTF. If you've done that, you will get in the model folder, you get something that you then can easily import into Fiji and then use as I've shown you before with the pre-trained model. So in this case, it's identically then to a pre-trained model. And that's something that is probably very cool for image facility to train their models and then give user only the exported Fiji model to be applied on your own images. And with that, I want to conclude the section of the practical with that slide. If you have questions after that webinar, if they're more technical, go to our GitHub page and open an issue. If you have a bug or unexpected behavior or something, if there are more general questions of what kind of data, what are best practices, what are your problems that you encounter that are not clearly bugs or technical questions of the package, then go to the image AC forum and please use the tag stardust and then we'll get notified and at least feel bad if we don't respond. And with that, I'll hand over to Ali that is going to give you some insight into how to use stardust in the image core facility. Great, thank you very much Martin. So yeah, I'm going to take over the screen sharing now so that I can then run my part of the presentation. So my goal here is to present to you a little bit the work that we've done with stardust at our core facility, the Bioimaging and Optics platform at the EPFL in Los Angeles, Switzerland. Ideas to tell you a little bit about, well, basically why we decided to pick stardust, why is it that we like it? That's the most important question we asked ourselves. Why not another deep learning framework? What justifies us spending time as core facility members to learn how to use, you know, yet a new tool in order to provide assistance to the users? I'll hopefully I'll be able to explain that to you. Then I'll show you our initial ideas that we've had on how to implement this within the core facility. And sort of the more fun stuff, I'll show you a couple of user projects who a couple of users who agreed to have their data being shown for this webinar. Finally, I'll show you, I'll tell you about some of the things that basically we found out as we struggled through all of this new field of deep learning and tell you a bit about what we're hoping to do and hoping to see in the coming months. So I am definitely not going to talk better than Martin explaining what stardust 2D does. So yes, I should specify that my examples here will be only based on stardust 2D. We have not had projects that needed stardust in 3D. So for us, what's really powerful about this is that stardust 2D is based on deep learning, but it is tunable, right? So even if Martin said that these non-maximum suppression thresholds or intersectional reunion thresholds that you can set up in the plugin, you have good estimations already there, but you can actually tune those. And it's a parameter like, you know, an automatic threshold in Fiji or something like this. And you're perfectly allowed to do that. So that's already pretty interesting for us. There's a lot of other stuff that happened in the background and U and Martin have done an incredible job at making sure that basically this was easy to install. I mean, you've seen it just now on the example from Martin. There was this huge development of the CSD Deep Library, which is actually used by other projects, not just stardust, but also care and also noise to avoid, that tries to sort of bring in a lot of smart things to do about how you should train your data, how you should load it up. That really makes our job also a lot easier, not least of which all the Jupyter Notebook tutorials that have been released since the beginning really let us dive into it, see exactly the examples that work. We're also extremely happy that the processing is actually extremely fast. So we don't need GPU when we're going to be doing inference. So where we're going to be predicting. And we actually found out that the training for the 2D part is actually quite fast or definitely fast enough for us to be able to offer this very easily. And something that maybe turns out to be a little more psychological than explainable in any other way is that when you show a biologist such promising results with already the pre-trained networks and you tell them, if you would annotate a certain number of images from your own dataset, we could get such a better result. This really motivates people and it makes them sort of pass the threshold into accepting to annotate all this ground truth data because we agree it is a very, very tedious process. And finally, since the release of the Fiji plugin, this actually solved a huge problem for us which was how do we distribute like this segmentation pipeline to users who are not burst in Python. So, you know, oh God, are we going to have to train them into Python so that they can run the Jupyter Notebooks? How are we going to get this back into a software that they like, like Fiji or ImageJ? And basically this problem stopped existing the moment they released the Fiji plugin. So, what is it that we are doing in the core facility? Like I said, we want to distribute trained models. Ideally, we want to use the versatile models but when they don't, then we have to train the model ourselves. We want to set it up as a zip file that people can use in Fiji. So, Romain Guyet, my colleague, really started this whole project with a simple three-step thing. So, the user will now annotate ground truth data. We'll talk about how much. I saw there were a lot of questions about that and usually we use QPath. We found out that this works really well. This data then gets brought to us. We will actually be the only ones who see Python during this entire workflow. We will take the data, train the model, make sure that the data is enough, maybe go back to the user, ask him to annotate more data, make sure it's fine. User then gets a zip file and he can use the model in Fiji as though it were any other plug-in without the need of knowing anything particular about the deep learning part. So, our current protocol looks like this. We tend for 2D data to tell people to annotate about 15 images. The size that I'm giving you is, yeah, 256 by 256 sizes. And from the video here, you see we use QPath. It's a really convenient tool for annotation because it allows you, as you can see here, to correct for mistakes very easily. And this was done by me. I'm not used to annotating things a lot with QPath, but some of our users are incredibly fast at doing this. And it's extremely powerful. It's extremely fast. It's very responsive. And you'll see also why we decided to use QPath in the following slides. And then this is kind of the interesting thing that we found out works really well. We realize that 15 images to train a deep net is definitely not going to be enough. But it turns out, I mean, we are talking about cell nuclei and a lot of features our data has, as long as their nuclei, let's say, are still perfectly similar to the Stardust DSB data, which has a lot more heterogeneity, which has a lot of fringe cases that we might not have in those 15 user images. So what we do is we bundle them up together. So we put the 15 images along with the Stardust data science ball data that was curated in the Stardust article. Then when we are going to split for the validation and training, we simply make sure that we have some validation images from the user as well. In this case too, this is quite arbitrary, but it's worked with us so far. Then here, this is really cool. We don't need to worry too much about the hyperparameters of Stardust. We train using the default and it works really, really well. This is all done now on an analyst platform that's running on Windows 10. We have a mini-condi installed and we're using a high-end gamer GPU to do this. So I checked the price recently on Amazon. It would set you back about 700 euros for an eight gigabyte card. And at the end, we do the threshold optimization that Martin mentioned and get a zip file. And for us, this entire pipeline takes us about two hours to work on this dataset. So it's actually really short to do the training compared to a lot of stories I've heard about how deep learning can take weeks or months to train on the data. And then basically, thanks to all the work that was done by Debra Schmidt and of course, the Stardust 2D plugin that Uwe and Martin made, we have now TensorFlow libraries available in Fiji. That's also being used by Deep Image J, for example. And the user simply puts in their new data, enters the thresholds that were suggested by the threshold optimization pipeline and then gets their segmentation in the Roy Manager. And then these are just Image J, one Roy's, so we can go and query all sorts of statistics that we would want, area, intensity, across other channels, et cetera, et cetera. So this is all very nice. Let's talk a little bit about how we are using the patches. So like I said, we wanna use KUPA because it's easy to use as you saw in the previous video. And what we usually do is we break it down into small patches. We don't want the users to annotate entire images, but we want them to annotate enough images that would represent the heterogeneity of the data they have. So in the example here, you have some very dense areas and some much sparser areas, and we've tried to annotate those. So in this case, in total, the user would have annotated three images and five fields for each image. Now, with this example, it might seem a little strange. Why would we bother only annotating half of the image? I mean, basically, why not do one full image and have less of them? This is true. But in a lot of cases, user data tends to be a lot larger than that, right? So in this example, we have a confocal tile scan of about 30 by 40 fields of view that was stitched together and is visualized in KUPA. And then thanks to KUPA, we can actually drop a few rectangles there for the user to annotate these very heterogeneous areas, but only do those annotations there. In this example, the user annotated three different brain slices and annotated 10 regions at their convenience or let's say at their judgment for judging the heterogeneity. And from KUPA, for us, it's very easy. We have a little script that will simply export two TIFF files, one with the labels and one with the small patch. All right, so that's pretty much how we've done it. And maybe some of you still don't believe that maybe this is a good idea, but it turns out we actually have a few cool projects that came up with this. So I'm going to show you three of these. So first one, you've seen the example data here. We had Celine Constantin, Fabio Di Martino from the Brisken group at EPFL. They were interested in memory gland ducts and they had three channel standard wide field data acquired with a 20 X objective. And we asked them to annotate 21 of these images by hand and we trained the data from scratch. We added the DSB data challenge with, sorry, the data science ball data. And after, yeah, about 45 minutes to one hour of trainings, we were really able to get a very clean segmentation out of that. I will have to point out that this project was done quite a few months ago and now that the versatile model for Stardust is out, this project in particular can actually be dealt with very well with the versatile model. So now this one, it's actually the larger subset of the example that Martin also showed you. So this user has confocal extremely noisy data that have this very particular shape of the nucleus. So as you see, it's not filled. The chromatin tends to be around the periphery of the nuclei. So much more complicated, definitely something that the Stardust pre-trained models hadn't seen. And despite the fact that this is confocal, we still appended the whitefield DSB data and it really gave a very, very good result here. And what you see here actually is a small part of the segmentation but we actually managed to run the segmentation for the entire brain. Now for something a little bit more different. So obviously Stardust isn't limited to fluorescence and it's not limited to the H&E model. This was a really serendipitous case that happened recently with Professor Malin Knoblo from the UNIL who's collaborating with Professor Matthias Lutolf at the DFL. And they were interested in cell counts and cell areas of hematopoietic stem cells that were acquired only in bright field on a multi-well plates. And they asked us, well, can we do something to ease the segmentation of this because we're drawing all of these things manually. And the data was very varied, was very different from something, let's say, that I would have been used to dealing with with classical image processing, if you wanna call it that. And we got really lucky in a way because the user had already suffered through annotating 51 of these images by hand because they really needed the data, right? So we said, fine, give us the data. We'll try to run this through the Stardust pipeline and we got a really, really excellent result out of that. The model still makes some mistakes. There really wasn't enough data to cover all the strange things that are happening. But for the user, clearly annotating, re-annotating two or three cells every once in a while in an image was a godsend compared to having to annotate all of these things manually. So what did we learn from doing all these things, basically? So the most important one and something that actually came up during the Q&A parts here is people asking, yes, do we really need to annotate all this data? So yes, we found out you don't need that much, right? About 15 images 256 by 256, that's a day's work in QPAT, if it's not too complicated. But what we found out is we have to go through validation around with our users to make sure that the data that is being annotated actually makes sense because sometimes we have assumptions about how we want to annotate the data or where a cell should be, but we have to place ourselves in the exact same situation as what the neural network is going to see. So if all we have is DAPI, but there's another signal that's membrane and it really looks like because the membrane is there, the nucleus should also be there and maybe for some reason DAPI doesn't show up. Well, if we're not feeding that extra channel to Stardust, then we shouldn't annotate a nucleus there. It's just not there from the raw data. We tested a couple of things and we really found out that QPAT was a great tool for annotating the 2D data. So we're definitely sticking to that. No answer to give you about the 3D parts. On a more technical note, Stardust uses internally something called GPU tools, which is also something that was written by Martin that accelerates some of the parts that are not deep learning concerned directly. So sort of the post-processing steps after the prediction from the UNET or the ResNet. And we found out that if we went through the tedium of installing this because it needs some compilation, we actually got a faster training. So it's really worth for us to go through this like extra step to activate that and we will offer a tutorial to those who are interested online. Something we learned, let's say, we suffered through and learned a little bit was, we started reading a lot about deep learning and finding out all the different vocabularies and the things we're supposed to know, like what's the batch size? What is the number of steps for epoch, et cetera, et cetera. And at some point we thought we were gonna be a lot smarter than the Stardust default parameters and we started tweaking around these things and where we're getting worse results and we were a bit frustrated because we thought, hey, I took a two-hour course on deep learning. I should be an expert on this about now, but it wasn't the case. So if you wanna play around with the hypergrammeters, feel free, like Martin said, there are a lot of them, but read through them and make sure that it makes sense for you to tweak them because Stardust does a lot of really smart things internally already to help you out. And for us, this has been really, really great. So for example, it does patch extraction on the fly. We can give it data of arbitrary sizes as long as they're bigger than the patch size and it'll worry itself about how it's gonna extract those patches and that's really a lot of work less for us. And maybe a last note, we said it already a couple of times, but really don't worry too much about having Fiji running with the GPU because it's really fast enough without, for us, this was convenient because that way we can train a model using the GPU on a Jupyter notebook and on the side, we can still test different things on Fiji as the model is training and we don't have any strange things going on that the same GPU is trying to be accessed by two things that want to run a deep model pipeline. And last but not least, when we give this to our users, we hope that they can use their laptop for Stardust 2D and most people will definitely have more RAM attached to their CPU than on their GPU. So keep that in mind if you want to work this out. A little bit about the future. So we've done this, we have another project on going to really establish, let's say prove that what we've done is doing a good job. What we're hoping to have next for our users or at least as members of a core facility is how do we provide the pipeline as materials and methods for the user? How, what kind of, you know, let's say material text is important, what sort of validation data should we provide? Like here, for example, you have the loss and the validation loss curves. After 400 epochs, you also have the mean square errors that are being shown. What do we show? How do we explain this to our users to make sure that they can justify what was done with Stardust? Some other nice things to have for sure. I mean, Stardust is really cool. We have it in Python, we have it in Fiji. We also use QPAD cell profiler at a lesser extent, IC. So if at some point somebody is developing Stardust for some of these platforms and will be more than happy to play with them on these different ones. Finally, as a proof of concept, just to tell you that it is possible. So we do have Stardust's example script that is running on QPAD. This is also a shameless ad so that if you want to register for the QPAD seminar that's gonna happen tomorrow, where Pete Bankhead is going to show some of the more advanced features, you will also most likely be able to see a neater version of this. But basically, we have something that QPAD can take care of breaking up the data into small chunks. Then we save these simply as separate TIFF files that are calling the typical Stardust pipeline. So this is actually Java calling Python to do the work. When it's done, it'll basically output this back again as what's called detections within QPAD. And here you see, I'm gonna zoom in a little bit. And so the outline that you see is simply an expansion of the nucleus. So we've only detected the nuclei of the cytoplasm. And then we can extract all sorts of statistics out of these very directly. And you could pretty much do the same thing in image J. It's just less pretty. Sorry. On that note, I'd really like to thank all the users that let me show their data for today's seminar. I'd like to thank my boss, Arne Zeitz, and all my colleagues at the Biomagic and Optics Platform for all the discussions and all the help that we've had and done into sort of putting this Stardust as a service thing going on at the Biop. And with that, I am done. And so thank you very much, Ollie. So there were a couple of questions about how much you can share from your presentation. For example, can you make available the slides and the script to export this from QPAD once you annotated it? So, yeah, sorry. So if I can share the slide, wait, wait a minute. Oh, it's already there. So the slides are already on the initial GitHub link. So I mean, I can just, if I share my screen, you can find them. So here, slides. No, no, this was about Ollie's slides. So for my slides, I'm going to double check with the users themselves. That's for sure. It will be a Google slide. So at some point I have it published. The link will be made available. And maybe I might have to remove a couple of them. But for sure, we will make this available on the NBS website. Sure, but yeah. Regarding the QPAD script, this is not a problem. The export for the annotations is very straightforward and it simply uses image J1. I will put it up on GitHub and provide a link. Okay, but I'm assuming you cannot share any of the actual data as data sets for people to use. I mean, that's probably not possible. Expect our users to publish soon in very high impact journals. And then we will make sure that they can, that everyone can download the ground truth data as well. Yeah, but this was a question that was asked a couple of times by people is about 3D models and the availability of pre-trained models. And this is kind of a problem right now that there are no data sets to use to really train a versatile pre-trained model in 3D that works on a variety of images. And also the variety of 3D data is much higher, especially the anisotropy of data is really so different in many different imaging projects. So that's why currently there is no 3D pre-trained model and we would love to have one, but we don't have the data at the moment. And there was another question about the training data being a multiple of 16, for example, some person probably did use the patch size, that was not a multiple of 16. And this is actually true, the training data should be a power of two, ideally, so 128, 256 and so on. It's just when you predict later on your test images, they don't have to be a multiple of two, there can be any size it would be internally handled. During training, it's assumed that the patches, the patch size in the configuration of the model is compatible with the architecture of the neural network. So making it a multiple of 16, for example, if you just do that, then you don't have problems. And so as already mentioned, you can put in bigger images into, you can annotate bigger images, but then use a patch size that is smaller, and then the training will automatically crop patches from that bigger image that you feed it during training. Okay, any other questions that Martin would like to answer? Yes, so I marked some of the questions that I think make sense. So there were some questions that said or that were asking if they have a membrane marker, can they use those starters? Typically it's nothing about the marker, so it's nothing per se about nuclei. It's about the shape of roundness. So whatever you are interested in segmenting, it has to be a roundish star convex shape. If your membrane marker like labels the membrane or stains the membrane such that there are a lot of protrusions, it's not possible if it's still round membranes or roundish kind of membranes, then it doesn't matter what kind of stain you have. Then another, a lot of people asked if you have to label each in the training data, if you have to label each nuclei, yes, unfortunately, yes, but you don't have to label complete images if the images are very large. It's sufficient typically to label crops of images, for instance, in 2D or 3D that contain roughly like 20, 30, 40 individual cell instances. So you don't have to label complete images if we really have large images, but only crops of them would suffice typically. And typically you would use crops of different regions of the image that contain 20 to 50 individual cell, instances in that should typically be enough. It's still specifically in 3D, it's very time consuming, like one, two days. But again, if you have something that runs for a lot, typically it's a good thing to do. And the one thing about this, when you annotate crops of images, make sure they have a certain minimum size. So when you, the patch size, for example, and started that is extracted. So it will automatically extract patches of a certain size from your annotated images. So make sure all the crops that you annotate from your own data have a certain minimum size. Yeah, that's one thing to keep in mind. There was one question about big data. So if you have a really big stack or really large image, can you handle this? So in general, yes. Our implementation does, for example, predict on a GPU and several overlapping tiles and will stitch the result back together and do all the processing for you. There's a parameter called underscore and tiles in Python. There's also an option in the Fiji plugin. And also it's possible in 3D. So we're even currently working on something that will enable us to work on really arbitrary sized images. So this will not be an issue. So if you're concerned about your data being too big, then you don't really need to be when it comes to prediction data. We would still kind of qualify that statement a little bit that in 3D everything slower. So expected for, just to give you a number, like for a one gigabyte image, it's roughly 10 to 15 minutes total. But again, it's limited by both the prediction step as well that you would have for any neural network predicting on a large stack as well as for the second step. So that's a little bit higher. And without a GPU, it's almost I would not recommend predicting in 3D. Yeah, that question I've seen several times. So training without a GPU, I would never recommend, but especially not in 3D. It's really would probably take days. Also the prediction... Training in 3D is a factor of 50 slower than it's even worse in 3D, even worse than in 2D. It's a factor of 15 slower, which you wouldn't love to pay. Okay, there was another question about the threshold. So already mentioned that we, even though we have this optimized threshold functions at the end of training, and this will pick good thresholds that are good in general, but depending on your application, you might want to set, for example, the overlap threshold differently. For example, if you really want overlapping results, then you can set that threshold really highest. For example, allow an overlap of 70% or something like this. Or if you want to have a dense packing and don't want any overlap whatsoever, you can set it to zero, for example. So it really depends on the application and the probability threshold, you can play with that if you want, for example, if it's okay for you to miss certain objects, but be really sure that the ones you get are really there, then you can also increase that threshold. So it depends on what kind of errors are important for your project, and then you can tweak those parameters to work more in the regime that you're interested in your data. The thresholds that we automatically find are good in general, but they may not be the best for your project. Yes. So another one is, if I say that nuclei are too big or too small to rescale the image. So this was only if you have a pre-trained model, and the pre-trained model has seen certain training data where the pixel size of the shapes or the shape of the nuclei, for instance, has a certain pixel size, for instance, in the DSP model, this is roughly 20 to 40 pixels. If you use a pre-trained model and then want to apply it in your custom images, you have to roughly, and you see a lot of over and under segmentations, you probably can fix it by rescaling images. If you retrain from scratch, anyhow, you would have to provide your images during training as you would have them roughly in the pixel size quality as you would have them in predictions. So this should kind of correspond, but of course you can do that. Another one, if it's working on EM, sure, it's nothing about the modality. So as long as you have roundish objects like granular structures or mitochondria or stuff, it should work in the EM at the same time. Of course, then you would have to use it in 3D. A 3D Fiji plugin, that's a little bit, it's in the work, so we have a prototype, but it's not yet. So I cannot promise this will happen in the next weeks, but at some point, I mean, we have a demo that already provides that, but it's a little bit more involved in terms of deployment, but at some point, this will come again for 3D, you would need a GPU anyhow for segmenting anything in a reasonable amount of time. Yeah, that question was asked a lot about having the 3D plugin in Fiji, and we would totally love to have that, but as Martin mentioned, it's really difficult to install right now, so we can install it ourselves, but we couldn't explain other people to install it easily. And also, right now, it's very nice. As Oli mentioned, you can just run it on the CPU. A Stardust in Fiji for 2D images, it's not too bad, but in 3D without a GPU, it would probably be a bit too slow. So then essentially, we have the 3D plugin for essentially a deployment, and there's more like technical issues why it's difficult to make, not conceptually. Also, the standard Roy manager in Fiji can only handle 2D, but there's also a 3D Roy manager as far as I'm aware, which is good as far as I heard. But it will happen at some point, so that's sort of it. There was one question about asking well how Stardust and Deep ImageJ are related. So I would say Deep ImageJ is more like a general framework or plugin to apply these models in Fiji where Stardust is a very specialized method. So in some sense, they're not that comparable, apart from the need to run a neural network in Fiji, and both of them as far as I'm aware are using a similar library at the moment. I think, Daniel, if you would like to... I mean, we have one of the developers of Deep ImageJ here. Yes, that's exactly the answer of Uwe. I should say that because a Stardust exists in Fiji, you should probably use the Stardust plugin because it's already done, I guess. Yeah, especially the Deep ImageJ will do the neural network prediction but not the post-processing part. There's no maximum compression part. You can call it separately. So if you really want to use Deep ImageJ, you can actually do that. But then there are two steps and you have to manually call the second step. So you have to understand a bit more what to do, but it is possible if you install both the Deep ImageJ and the Stardust plugin, there is a way. If you really, if you have to use Deep ImageJ or it's your workflow anyway, you could use just the post-processing part from the Stardust plugin. It's not documented, but it's possible and we can document it if there is interest in that. So this was one of the two steps. So the first step in the Stardust pipeline was actually prediction, the neural network prediction part and the second step is this non-maximum suppression. There is actually a lot of more logic involved and the second step, that's something specific that as Mover mentioned, you could call it on top of anything that provides you the predictions. So there was a question, if you train all the models on a 1080Ti GPU, no. We train all our models on a five-year-old Titan X. So we don't need something very specific. So any recent GPU would suffice. And so for the Qpath extensions, there are some questions. I would suggest because that's super recent. I think Peter just recently put that in and there was a webinar tomorrow about Qpath. So please, please go there. I think it's interesting anyhow, but I think he will probably mention something along these lines too, at some point, maybe shortly. There was one question about the compatibility between the Python and the Fiji plugin. So the Python, when you train in Python, you have a specific version of TensorFlow and when you export a model, it will require, as a rule of thumb, your version of TensorFlow has to be as high as your version in Python or later, right? So when your Python TensorFlow version is new, then your model may have features that are not available in Fiji. And Fiji has also a way of updating the TensorFlow version that's shipped with Fiji, which is by default 112. So this is a more technical thing, but if you ever encounter a problem that something about your model is not available or so, that could be a mismatch between your version of TensorFlow in Python and in Fiji. And you can either upgrade your TensorFlow version in Python or upgrade the one in Fiji, which I would recommend. It's TensorFlow error messages are not always the best. Like one important part is like our currently, the Python training framework is TensorFlow 1.x only. So TensorFlow 2 is the most recent version, but we still rely on old features. So you would have to install, you would have some variant of TensorFlow 1.14, 1.13, 1.15, or something like this, yeah. There was an interesting question, which was about a multi-channel data. And when does it make sense? So basically the idea when you have multi-channel data is it can handle it. Stardust can take this for training, at least. And you can also, I think, you have to give it a one-channel dataset if you want to use the models that are already pre-chained, right? Well, the one, yes, the other. OK. And then the question is, if you want to add these channels when you're training your data, that really depends on, are they useful in order to do the detection of the shapes that you want? If it's a channel that is going to be varying with respect to all of your different conditions, that is not a nuclear channel but something else, it might not be worth adding it there and maybe add a single-channel image. If you maybe have a membrane marker that you think can help find the nuclei and you've annotated it on these two-channel images with, let's say, DAPI and the membrane marker, then this can help, perhaps, the neural network to predict the cells better. I mean, that's an interesting question for another reason. Even if you, for instance, if you have a membrane marker or some marker of the outline of the membrane and then you have a marker for some gene that you want to get the expression level relative to the cell and you do a study where this varies a lot, then probably you don't even want to use the channel. You want to take measurements on and where you have a higher variability in the training data because then your model could get biased towards something that you really then want to have a quantification of the intensity. So there it would make sense to have a model that only gets or only looks at a channel that actually would give you the outline of the cell and not the one that is maybe correlated with that but might not any more correlated at your test data, which might be then a knockdown or some mutant or whatever. But if, yeah. All right, well, with this, I think it will be time for us to close the session. I'd like to thank again everyone who just short my acknowledgments. So everyone who gave me data and especially their new bias team around Julien, Romain and the others, I think it was, they really did so much work and it was really a joy and pleasure to work with you. That's all. All right, thank you very much, guys. So I've sent everyone here a message. Please fill in, if you are still here, please fill in our survey at the end of the seminar. This will be really good for us to get a feedback of how this went and how we can improve things should they need improving. Thank you very much, everyone. Have a nice end of whatever time it is in your time zone. John. Goodbye, everyone.