 Welcome everyone. I think it looks like participant numbers stabilized. So, yeah, welcome everyone to this webinar. We're very excited to have the U-GLAB today here presenting the care tools. And they promised us a lot of exciting features. And we're here as a, like, of course, Lauren and Yook and then Tim Buchholz, Mangal Prakash and Deborah Schmidt and Alex Skrull from the U-GLAB. And then we have on the managing side from the Bias Academy, we have Nuno is also in the U-GLAB and then Rocco and me. And we have also Kurt Andersen, who is kindly hosting this entire session. And I would like to ask you to ask your questions in the Q&A window and to use the chat only for technical problems that you have. So please use the Q&A for any content related questions. And with that, I stop sharing and pass over to the Yook group. Have fun. Thank you Anna very much for your introduction. I will now share my screen and kick off this webinar about a beginner's guide to quantitative image restoration. The webinar today will show a number of very well known functionalities and also a number of things that we only pushed out in the last couple of days. We use this invitation by Nubias to give this webinar as a motivator for us to kind of release new versions and new features. And maybe run you through a number of those. The first thing that is brand new since tonight is a complete revamp of the CSPD BioImageComputing.com website. And I will start by showing you this website a tiny bit. It doesn't matter if you know the old website or not. It was useful, but it was not so image analyst centric. And the new website really puts the question, what can I do with my data in the center. So CSPD is a deep learning toolbox for microscopists and you can do many things with it. So if you ask the question what can I do with my data then here we have image restoration and multiple examples of what you can do. We have image restoration also if you only have noisy images or only single noisy images and then also some examples for object detection. We will speak about a subset of these things today, but if I show you now one example. You go on to this example you see it before and after image you know who authored the method what is the use method it's actually care. And if you click on it you come to this available tools and you see a bunch of links that point in different places in the internet where this information was already available for many months. The nice thing now is that there is like a center place where you can start your investigations about the methods we offer and we it's not only the uGLAB but a bunch of people that contribute to the CSPD tool set. So on the available tools you see that currently we host for very different types of tools, if you for example click on noise to void which is one of the things we will speak about today. Then you find a gallery and again credits videos tons of things are here, but you will also for at least three of the four tools, you find setup instructions and exercise sheets and they might be interesting for you after the webinar if you're interested if you want to learn more about how to do these things. This might be valuable resources for you to start learning about these tools. Okay. Let me go back to the presentation. Today's speakers are in order of appearance Alex Cole, Tim, Deborah and Manga and the, what we speak about is things that we worked on in the last years, but it would not be possible to speak about all these things if there would not be many other people contributing to. So here we pick six very important ones for the content of today's seminar, but at the very end we will have to give credits to a bunch of more people. The really important is to mention that Martin and over like really of uttermost importance for the entire CSB deep library and Coleman also authored a methodological contribution, we will showcase today. But since we already five speakers and we kind of cut it missed with us five and cover for Coleman. And then the last three names here are research software engineers and their heroic efforts really bring this bleeding edge technology into Fiji or into Jupiter notebooks in a way that it's usable by many more people than just like the researchers around the world. So, enough preamble. What is happening. There is an introduction ongoing by yours truly, and then we will hear two main blocks, where we first will hear about content of an image restoration by Alex Tim and devil. And then the second block will be slightly shorter and will present a new tool called the noisy, which is also a denoising method but at the same time does also segment the data. And with this, I have only one thing more to do. And that is, I want to tell you what you will not learn and what you, and they give you a short introduction to one thing that you will also not hear much about but at least will help you to understand today's presentations a bit better. What you will not learn today is you will not learn how to install Python or Jupiter on your machine. There were other new bias events and the Internet is full of useful resources. We are very happy to help you there too. But this webinar will not cover this. It will also not cover how to install TensorFlow or Keras with GPU support on your machine. And that you either notice already have done this already or will do that in the future on your own account. And if you will also not speak much about Jupiter in general there were also another very good new bias training school events, virtual events happening in the last few months that speak more about this. And we will though very quickly run through as example so I assume that many of you have used Jupiter before, but if you happen to be. If you happen to be one of the people that never used it. At least enable you to understand the rest of this webinar. If you start to put a notebook after installing it. Then in your browser you get this environment is this notebook environment. And what you see here is all the files that are in the folder where you started to put the notebook and I just created a demo notebook. I'm going to run you through the most important things and Jupiter notebook is essentially source code. But the nice thing is the source code is grouped in small cells of code that you can either execute by clicking on this one button, or by hitting shift, holding shift and hitting enter. Asterix here meaning this cell is currently interpreted by Python and and these code cells they can be interleaved with with formatted text cells. They just show you that something here is maybe not so important so shift enter shift enter we can just go over it, shift enter shift enter, and everything works. So what happened here, although it says it's pointless and maybe you want to ignore it, then if you read it, it is just Python. So if that troubles you, it should not matter either, because many of the notebooks you will get in contact in the next hour or so. They are written in such a way that the format the text on top points you to all the things that might be relevant for you and if you just let the rest be. If you use intelligent pattern matching just find the places where you want to change something for example you want really to blur it less than it's kind of obvious that this is a Gaussian filter on the loaded image, and if you change the sigma to something as you see, without really programming just following this was like intelligent pattern matching and just the the willingness to mess around a bit, you will have absolutely no problem. So this is now really the end of what I'm going to show as an introduction. And next up is the first block, it will maybe be roughly 30 minutes. And, and after these 30 minutes, we will have one break for answering a few questions that we think important enough to answer in the planum so that everybody here's the question and the answer, and then go on to the second block. So Alex, please entertain us with the theory part about care and noise to noise and noise to void. Hello everybody. So, I will. I hope you can see my screen. I will tell you today about the different methods for image restorations that have been developed at MPI CVG and in the euclip in particular. So, I know that many of you face the dilemma of having to find the sweet spot between using a lower exposure, which might be gentle for your sample. But we'll also give you a poor signal to noise ratio and using a higher exposure which might give nice images but might endanger yourself. So, luckily, with help from Martin Weigert and Uwe Schmidt, deep learning came along and provided a black box for us called the CNN that takes a lower exposure image and transforms it into something that looks very much like a high exposure image you see here on the right. However, these systems, these neural networks have hundreds of thousands to millions of parameters that have to be tuned in the process called training. So, traditionally, this is what we call care training. This process starts with two stacks of corresponding clean and noisy images. We feed our noisy images into the network. Here this is shown in a 1v fashion. The network outputs some kind of prediction and we can now compare this prediction with the corresponding clean target. We can calculate a discrepancy between the two. We call this discrepancy the loss and we can tune the parameters of our network to minimize this loss. And if you do this for enough iterations and for long enough, the network will actually learn to denoise noisy images. However, there is a slight caveat here. Namely, we need this paired kind of training data. We need pairs of noisy and clean images. And in particular, it can be difficult to acquire clean images for some type of experiments and for some type of data. Sometimes it's impossible. So, a different way to train your networks is called noise to noise. This was introduced by Lehtinen at Alice, not our work. And this is a way that allows you to train your network using pairs of noisy images, which do not require clean data. So, in noise to noise, you assume that your noisy image as it comes from your microscope is just the sum of a clean signal which you want to recover, plus some noise which you would like to remove. Now, in noise to noise training, you start out with two stacks of corresponding noisy images. And what does this mean? These two stacks share the same signal, but have different instantiations of the noise. So, in practice, this would mean, for example, you could acquire such an image pair, but just in quick succession, recording two images with your microscope, hoping that in between, nothing moved in your sample. So, then the underlying signal should be the same while the noise is a different instance. Now, we can do the exact same training procedure. We can feed the noisy image into the network, produce some kind of output, and minimize the discrepancy to the target. And again, surprisingly, maybe, the network will learn to denoise the image. So, why is this? How can this work? So, I can only give you a brief intuition. Since the signal is the same in both images. This noisy input image contains information about the signal, but it does not contain information about the noise that it's supposed to predict. So, the network has to solve an impossible task, and the best compromise it can come up with is actually a good solution for denoise. So, here is still a problem in practice, because we still require training images, and there are situations where it's difficult to acquire even noisy pairs of images, where you only have individual noise images. And this is where the method that we developed in the Euclub 2019 comes in that we call noise to void. Again, we use the same setup, but we make additional assumptions about the nature of your signal. In particular, we assume that your signal is, in a way, predictable in that sense that by looking, you can get a reasonable prediction for the value at a pixel in your signal by looking at its surroundings. This is just a way of saying that your signal is something different from random wide noise, right? There is some structure in the signal you would like to recover. For the noise, we make an equal but opposite assumption. We assume that it is impossible to produce a prediction for the value of the noise at a pixel by looking at its surroundings. So, in noise to void training, we start with a single stack of noisy images. We feed them into the network, and the network will produce some kind of output, but now we want to take a different perspective. We want to look inside this network and we want to see how the prediction for an individual pixel is created by the network. If we do this, we see that the prediction for this pixel here actually depends only on a limited range of input pixels around it. And this is what we call the network's receptive field, the pixel's receptive field. So, what do you think would happen now if I take the same noisy input image and use it as input but also as target? So, if we simply go ahead and we do this, and we train the network to minimize the loss, the network will actually only learn to always forward the central pixel of its receptive field to its output. In effect, this just means that the network has learned the identity. So, this means it will always output exactly what you put in. This is not helpful for denoising. So, in noise to void, we came up with an idea to solve this that we call the blind spot network. And the idea here is to modify the receptive field of the network in such a way that it cannot use the information at the central pixel. So, we introduce a blind spot here. And a network that has this type of receptive field, when we train it in the same fashion, it will actually learn to denoise the image. So, how is this possible? Remember the two assumptions that we made earlier. For the noise, it is impossible to predict what is the value of the noise in the blind spot by looking at the surroundings. But for the signal, we can make a reasonable prediction for the signal at the pixel by looking at the surroundings. So, how can we create a network that has this type of receptive field? Why, you can go ahead and you can engineer a network architecture that achieves this, and people have done that. We chose a different direction. We chose what we call masking. So, here we have the same image as input and as target. But the trick here is to not use all the pixels in the calculation of the loss, but just a randomly selected subset. And we calculate the loss only for those randomly selected pixels. But in the input, exactly these pixels are masked. They're made invisible for the network. Noise-to-void can achieve some quite remarkable results, like you see here. But the most important feature that you should remember about noise-to-void is that in order to produce this type of results, you do not have to go to the microscope to acquire additional training data. You can use the very same data that we want to denoise in the first place. You can serve as training data and then subsequently you can denoise it. So here I want to give a quick overview about the things that the different methods I've talked about can achieve and what they cannot achieve. So, I have already talked about the type of noise that you get when you have low-light imaging. It's for short noise and read out noise that happens per pixel. And the traditional care that uses paired training data can do pretty well here. But you can also do other things with it. If you have the adequate training data, you can try to increase the resolution of your data. If you have pairs of high resolution and low resolution. You can also do deconvolution in the same way. And finally, if you have like some people who are in the unfortunate situation of having to deal with JPEG compression artifacts, problems like this, then in principle you can also use a supervised approach like care to help you out with this. If you have the adequate paired training data. Now, for methods like noise-to-void, it's different. Because for the per pixel noise, we can use the type of data that I've explained to you. And we can do two successive recordings of the same scene under the microscope to produce the training data. But for deconvolution or to increase the resolution, you cannot produce this type of data. The same goes for noise-to-void. You make very specific assumptions about the noise that it has to happen per pixel. And we cannot use this scheme to increase the zero resolution or to do deconvolution. So with this, I want to hand you back over to Tim, who will now tell you something about how you can actually use these methods in practice that I've introduced. Thank you. All right. Yes. Thank you, Alex. I will talk about the practical approaches and how you can get started with CSB Deep and these tools. And as Florian already mentioned, the entry point is the new website and we can go here to what can I do with my data. And you see these scenarios. And for this example, I will choose this denoising in 2D for noisy and clean images so the traditional care approach. Again, here's some information on how to install it. And down here, the link to the Jupyter notebooks, which you can use to reproduce these results. I've already downloaded these Jupyter notebooks and started the Jupyter server. And here we are in CSB Deep, examples denoising 2D. And all care workflows consist of three Jupyter notebooks, the data generation, the data gen, the training notebook, and the prediction notebook. And we start with the data generation and everything we do here is we collect the data on which we want to train our model. So in this example, we want to go from low quality 2D images to high quality 2D images. And we have to organize this data in a way that CSB Deep is able to read it and handle it for the network training. How do we do that? Well, we put all our low quality images into a directory in data slash train slash low. And all our high quality images we put into another directory in data slash train slash GT, GT stands for ground proof. You can see this here, I have data. Then I've trained and in here I have these two sub directories, low and GT. And in here I would have this many, many images. It is important that the corresponding images have exactly the same name, because the name is used internally to match these image pairs together. So, if I open up this Jupyter notebook, the data generation, you can see, yes, this is the corresponding example. We do some import, so I will not go over this and execute the cells and we just talk about it. Here we download the example data. And then the first thing we do, we display two example images just to verify that we are actually using the data we want to denoise later on. After that, we use the functionality from CSB Deep to load our data from this data slash train from the load directory and from the GT directory. Once we have the data loaded, we use this functionality to extract patches of size 128 by 128 pixels. And after they are extracted, they get saved into this file in data slash mytrainingdata.npc. And if we go here, you can see I've executed this, I go here back to data. You see here this file, mytrainingdata.npc, which contains all these extracted patches. The last thing that happens in this notebook, we display a couple of these patches just to verify that the corresponding patch pairs are correct. And this is it for the data generation notebook. And with this, we would come to the next notebook, the training notebook. And the first thing that happens in the training notebook is we load the patches which we generated in the first notebook. As you can see, we have low quality image patches and high quality image patches. And we also call them input or X or target or Y. So we want to train a network from X to Y. The next thing we do in this training notebook is we split the data into two sets, a bigger one, the actual training data, which usually takes up 95% or sometimes 90% or 80% of data we use as validation data. The validation data is used during training as an independent control to measure the accuracy of our model on unseen data. So once we have split up our training data into these two sets, we can define our model. And then we can start training by passing to the train function of the model, the training data X and Y, and the validation data X, Y and Y1. And once training is complete, we will end up with these nice curves. So here we have the lost curves. The blue curve is computed only on the training data and the orange curve is computed only on the validation data. When the curves after training look something like this, everything went well. You can see that the losses are going down and towards the end the validation loss reaches this tattoo and converges nicely. Now sometimes it can happen that your validation loss shoots back up into the sky. And we call this overfitting. So what is happening? The network does a really, really good job on the training data. So you can see here the loss gets smaller and smaller and smaller. But on the validation data it does a really poor job. This is because it never sees the validation data and it's kind of just remembering the training data. This would be a really bad network if we would use this one from the last epochs. In this case, we actually want to use the network which has the best validation loss and best means the lowest validation loss. Luckily CSPD takes care of this and saves not only the weights from the last epoch but also the weights from the best epochs. And by default it actually loads the best network, so corresponding to the lowest validation loss. And with this we are basically done with the training notebook and we come to the last part which is the prediction. I have here the prediction notebook and this is a really short notebook. Again, we have to do some imports. We load some data which we now want to denouce so we forget another noisy image. In this case we also have the ground proof because later on we want to compare how well it actually works on transient data. Then we load our model and you can see here by default it loads the best weights corresponding to the best validation loss. And after this it's just calling model predict on our noisy image X and we get to restore the image back. The last thing in this notebook is a comparison of our low quality input, the restoration with care. And here we have the ground proof image to have a comparison on how well the care job is done. And this is the basic care workflow. So now to summarize and give you some tips. So if it's the first time that you're doing deep learning or image restoration in this setting, we can highly recommend that you just take one of these example notebooks and execute it cell by cell. So you start a notebook, hit shift, enter and go through all the cells and see how everything works. And the first time or the first two or three times, this can take a couple hours and we should just take your time, take an afternoon off and step through all these notebooks and try to read all the comments and understand what is happening in the cells. And maybe the exercise sheets from the website are helpful for you as well. And once you step through these notebooks and feel familiar with what is happening, you can go a step further and start replacing the example data with your own data. You would do this by going here into the train directory and replace the images in these folders low and GT with your own data. Just make sure that the names are actually the same, because this is what is needed for the match. So you have done this and everything seems to work out fine. You can go another step further and experiment with different settings. For example, here in the training notebook. If you go to the network configuration, you see there are many, many hyper parameters and you can play around with these and figure out maybe you find the setting which works better for your data. But in general, these default parameters they work quite well for many different types of data. If you have problems or questions, just go to the new website or ask the question on the phone. And with this I'm at the end of the care workflow and I will now go over to the noise to avoid workflow, which is almost the same, but as a slight difference in the noise to. Sorry, I have to learn quickly. In the noise to avoid workflow. We only have two steps with the training and with the prediction. So this is that noise to avoid. We don't really need pairs of low quality and high quality images. We just can use our noisy images. And this makes the whole data preparation much simpler and we just put it inside of the training. I have here one of the notebooks open. This is the noise to avoid to the example on SEM data. And it follows the same principle. You can step through this notebook by hitting shift and enter. We download some data and then we open these images and the only thing we have to do we have to extract training patches and validation patches. And you can see we don't have any ground proof targets here because they are not needed for noise to avoid. So once we have the patches, we can start creating our network and start the training by placing X and Y to the train function. And after training, you again get these plots of the loss curves. And even though they look really converged you already after a few epochs, you really have to train them for many, many epochs. Because the noise to avoid training gets better and better and better over time, even though it looks like it's already done after five epochs. So if you have such a network for four or six hours or just overnight, it's really the best you can do. And here I would like to talk about the export of a such a train model with the bio image models to format. And you can do this by executing this cell and this will generate a zip file called export.bioimage.io.zip, which you can find. You can find this zip file here at this location. And the cool thing is, I can now take the zip file and send it to one of my collaborators, one of my users, if I'm working in a facility. And this user can just use this train model and apply to her or his own data. Another cool thing is, you can do this training. For example, I did it here in Python. I exported this model and I now send it to Deborah, who will talk next and she will show you how she can use this train model in Fiji to predict on her own data. All right, model gets sent. Okay, hello everyone. I'm going to talk about how our tools are integrated in Fiji and I'm not sure if I'm already sharing my screen. Am I? No. No. So you might have noticed that the file that Tim just showed you ended with bioimage.io.zip. And I just want to show you this website quickly. It's called bioimage.io. The reason for that naming and that is that there's an initiative, the bioimage model zoo, which is built on top of Imjoy and it's a big collaboration between Imjoy, Elastic, ImageJ and Fiji. And this is supposed to be a place where you can store and share your pre-trained models and where you can also find models that fit your data. This is a big task. This is currently under heavy development and we are joining the force and with this model that we can already share now between Python and Fiji, we are taking one step closer to being able to share the models here on this model zoo as well. And if you go to the site, you will already see a few models here and one of this was trained in Fiji. So this is where we most useful, for example, for facilities, as Tim just mentioned, where someone is doing the training and other people just want to do prediction. Okay, everything I'm showing you now, you will also find on our website again. So I will go here to available tools and then to Nice to Void. That's just what we talked about. And there is a link. It's called how to use Nice to Void in Fiji, which will lead us to the ImageJ wiki, which has a whole site about Nice to Void in ImageJ. And the first section here is about how to install it. And the only thing you need now is the CSV deep update site. So that changed in comparison to the first version. The CSV deep update site includes the CSV deep tools, the care tools and also Nice to Void and the Noisek. And that's really all you need. Of course, you need more if you want GPU support. I will not show you this, but I made the page with all the tips and tricks that we somehow collected. I hope that helps. I know it's not that easy for some hardware systems for others. It was really easy. But you keep in mind that these TensorFlow native libraries for Java that we are using are considered experimental and that not many people are doing training of Keras model in Java. But if there are any issues for you, please let us know in the form and we will try to help you here. Okay. If we go now back to the overview. And you will see that there's also a point called export trained models from Python to ImageJ Fiji. And if you click on that, there's a very short list of things to do. The first thing we just saw calling model.export tensorflow in Python, then you locate the exported model and then you can just proceed as described in the prediction section. And I will not go through this with you now, but I will show you one of the ways of running prediction directly in Fiji. I'm using it and I can open a model by pressing file import bio image IO dot zip. And then just choosing the model that I pre trained before in Python or in Fiji and then I open another image, which fits this data and this is also an SEM data set. And then all I have to do is press predict and wait for a couple of moments. This is now real time so you see it takes a bit but not that long. And I will add a progress window soon. So this is the result looks really nice. And that's all you need to do. I was lying before this was not trained in Python. This was actually trained in Fiji. And how was this done. As you heard before, that for no support, you can train on a single image like this one is big enough. But if you have more data, you can also choose this command train on folder, but we will now use the one click solution called train and predict which trains on this one image and also predicts it afterwards. So we click this we can choose the default values and during training, that two windows open, which guide you through the process you don't you can just watch the process here the one in the middle is showing the training and the validation loss is just the same as we saw in the notebooks before. And on the right you see the prediction, what the network currently thinks is the denoist version. During training and then you get to result. This took now in total six hours roughly so this is this version of course. But as you saw before prediction goes really send you take your time you let it train long enough, then you can really benefit for prediction then and I will show you later how to reuse this model. When I show the next tool. And with that, we talked now about care and nice to write and there's one cool new addition to nice to write and this is something that Alex is now going to talk about. Hello again. All right. Yes, so I will talk about what we call structured noise avoid that can help you with to deal with more difficult types of noise. So, this is a paper that was conceived and published by Coleman as a first author. And we noticed that when we want to denoise certain types of images required with some specific microscopes noise to void sometimes does not work as well as be expected to. In this image here, the denoist version does look arguably a lot better but still, especially if you look at the background regions. You see that there's still a lot of structure where there should be a flat surface so why is nice to why not able to remove this. If you take a close look of the type of noise that we have here in the original raw image. And, yeah, you can see this if you look really closely if you if you look horizontally through the pixels and vertically through the pixels, you see that they are actually not the same. So this noise is structured horizontally. And this is unfortunately something that the microscope this particular microscope involuntarily produces. Ideally it should not. But in reality, we have this kind of problems. So this noise does not happen per pixel. It's actually violating the assumption that we make with noise to void. And this means that noise to void will not be able to recognize it as noise and remove it but it will, like, produce this type of result. So what can we do to combat this. So, you remember that we use the masking strategy like the one I show here to produce these blind spots during training. We randomly select pixels and calculate the loss only for those randomly selected pixels, which are masked in the input. Now, if the noise is predictable, which it is in this case by looking at the surroundings of the pixel, we get the bad results that we saw before. So the solution that Coleman can upwards to instead of masking justice pixel. Additionally, masking a bigger area around it. With this, we can get much improved results. So here I show again for comparison. This is the normal noise to void result. And if we use Coleman's method. You see that you get in much nicer result for this type of data. So we have on our CSP deep website, also examples of this, which you can look at. Yes, and now I think we are in for some question time. I'm not 100% sure how we proceed now we are answering questions all the time in the chat and on on the system that is provided by new buyers. And I have multiple of the questions that we also answered in the chat labeled as being interesting for everyone. So if maybe one of the other panelists could read them out loud. I don't know if you feel motivated to do that. Because otherwise I have to read their own question that then I will go to answer which is slightly awkward. But I can do that as well. I see nodding faces. So I guess I read the questions. So, so, dear Lulu, not really I was hoping somebody here might know of a good page. The question here was, can you please on your website also point us at good instructions of how to install files and Jupiter, TensorFlow, Keras and all these things. The reason why we don't have this on our website yet is that, depending if you are on a Windows machine if your Windows machine is new or old, what GPU you have these instructions might need to be different. I'm not aware of a single page that kind of would serve all our users. And so if we instead of having a single beautiful page which would be really, really great. We put that 20 different pages, then we didn't solve the problem. We just shifted the problem from the Internet on to our page in the Internet. So if anybody in the audience thinks, no, but wait, recently I found this page, which in my point of view is just amazing. And it would be extremely grateful if you could send an email to us or contact us in any other way and let us know, we will then check this page and if we think that it is indeed a page that is doing the trick for many users, we would absolutely link at it. And next question. And if my samples are robust enough, is it best to train a care network or go with noise to void. I think, so would it be less work for the experiment how to care and noise to noise and noise to void perform compared to each other on the same data. Alex, maybe you can answer this question. Yes, so I can answer it. And maybe I can show if I share my screen. Oh, you answered super professional. Go ahead. Yeah, always answer super professionally. So, yes, like a comparison of noise to void, and we compare PSNR values so high PSNR values good. And on the x axis you see the amount of noise that we added to the images. And these are different types of denoising methods that we have. And the white one is noise to void. Now, what we call traditional here is care. And you see that noise to it's worse. Right and somehow naturally right because we, we are using less information right we we have this blind spot. And the traditional network does not have it right it can use the full the full range of the receptive field. So, we added disadvantage but you also see that the more noise you add, the smaller, the smaller this gap becomes. And if you have more noise, you will be actually more closely approach the traditional care training. And intuitively this also makes sense because if you have a ton of noise, then it is not so severe to disregard a single pixel right. But if you imagine the other extreme where you have basically no noise, most of the important information is actually in the one pixel that you include. Yeah, so this is basically the gaps that we have. The next question is kind of related I would think, or is there something that you are burning to add. I would just want to add one more thing to it. We have, we are developing newer methods that are also unsupervised like noise to void that managed to narrow this. This is just something I want to say. Yeah, yeah, we will speak about that even later we will hear in about half an hour or so. We will hear about methods that that come that are also in this gap. It's usually not that we can be better than care, but they narrow it. And I think from a user perspective you're the person asking the question is absolutely right. If you, if you do not have to create ground truth images and sometimes this is not even possible. And the propensity of you choosing noise to void is higher right if you cannot get ground truth care does not apply. If it is difficult. There's also the danger of having bad ground truth right so it could be that the only ground type of ground if you can produce is actually not really good for example if your sample moves right. So the noise to void video you showed for example, the one example movie. This process is that the image they are so fast that if you try to make the same image content image in the second time at better quality. The individual punkter being in this that we saw in these images they would already move around, and then the ground truth image even having good quality doesn't match the first acquisition anymore. So all these kinds of problems can go wrong with care. So we have to move on because there are some more questions that you would like to work on the next question is somewhat related. At least the answers are, is there a general guideline on how many sets of images we need to train an accurate to train an accurate CNN 100 pairs 500 pairs. And that's the number of pairs of images depend on the signal and noise. And the answer is hard to say how much data you need and yes it depends on the signal and yes it depends on the noise. So in general, the device it hard to give an answer. Your images look more or less the same. If it is like a simple data set that is not confusing, and the noise is very little, then you need not many examples to see many examples to get a good feeling for how to denoise these images as a network. On the other hand, you image something that like EM images right scanning electron microscopy very many different structures are in your data set. Then you also need to see many more examples, because of the diversity of of the imaging landscape so to say, right. If on top of that you also have huge amounts of noise, then you need to see even more data in order to kind of be more sure what the heck is going on in your data. And this kind of I think it makes it clear why the answer is difficult. And I move on to the next one that says care images often look even better smoother than the ground truth. How does this happen. And here I would like to use this. The answer is giving you only a cliffhanger. Continue after this Q&A. We will hear about noise to noise and stay tuned because that is exactly the answer to your question. The ground truth is not really ground truth. It's just a lesser noisy version of the same image. And so the noise to noise is the answer by the care prediction can be of even higher quality than the ground truth itself, or even smoother. So what is the influence of the patch size on the output, meaning does it affect. Oh, this is, I cannot read the whole thing. Need to enlarge. Can you help me? Yeah, awesome. Does it make a difference if I use a 32 by 32 pixel or 64 by 64 pixel regarding in the construction quality. This is a very detailed question. So it asks, there is a parameter in training our networks that says what size patches do I cut out of the images and then we only during training feed this like smaller patches. And the default settings should be enough. Why would one ever want to cut bigger patches. If you change the network in such a way that the so called receptive field of a node in a network gets larger, then these patches can become too small. And why would you want to increase the receptive field size. You want that if your data do you want to be nice is more complicated. The default settings again for the people that are new to this should, you know, suffice in most cases. If you see that the network cannot come up with a convincing restoration, then it might be that you need to give each pixel a bigger context to see so that the content awareness can kind of refer to a larger portion of the neighboring pixels. So that is an advanced thing and I cannot fully explain it all facets of this question, but it's a very good question. And where have I been question about noise avoid so Alex I think you can already kind of get in the starting position to answer. In the first lifestyle example, the restoration is really impressive, but I also noticed some artifacts in the less bright regions. It looks like the cells are dancing and or reshaping their nuclei all the time, which could lead to false conclusions. I've seen this artifact more clearly on my own data with two second intervals, nuclear should not move. This raises the question, how can you know when to trust the recording images so this is a very loaded question and when to be a bit skeptical. I give a few answers right away then maybe Alex wants to fill in some things. When should be skeptical. Always. You should always be skeptical right you always look at the result as if you want to show that it didn't work well, and then convince yourself that the results are actually amazing. That is what I do every single time and what you should probably do as well. Then you said that there were artifacts in the background. I wouldn't trust a video played in PowerPoint on Alex's computer transmitted via zoom onto your screen right all kinds of things that you have seen might not really be there. But you also said you saw similar things on your own data. I would very much appreciate if you go to the forum, or if you go to the GitHub repository of the tool you used, you open an issue, and you give us a link to your data and to describe these artifacts because we are very keen on learning about these things and many people approach us with such problems. And what typically turns out is that there is a very clear explanation of how how these artifacts came to place and often they're not artifacts. The timing in the movie. It was not a real time movie right the imaging experiment was going much much longer, and we just showed a movie at the two second frame rate. So the reshaping of the nuclear and the positioning of the, it's not MS to but the transcription events. The timing was was very different than what you saw. Maybe I now took all away from you Alex or do you want to add something. I can add the set. I think you produce the data from right so. So, but the data is from the age from Stephanie. And I think you did it. Like taking taking each image from the movie and separately processing it. Correct. So, and this means that, yes, so generally as general consideration, the system the network will try to make the best prediction, giving given that image, right, and the image is very noisy. So it cannot be 100% sure it cannot look into the future into into the past like you can right when you're looking through the movie. So what does help in practice and gives better results in general is if you actually treat this stack of movie as a 3d volume. Right and we have had very good experiences with with doing that so if you have a 2d movie, it can make a lot of sense to pretend it's just a 3d volume and train and network to be nice. Thank you very much, Alex. So this brings us to the last question you have to hurry up a bit to not get into trouble with the one and a half hours but I'm so grateful that you asked all these questions because these are all very important points and the next one is also a very important question. How useful artists. How useful are the model one can find online. For example, from the bio image model so since I thought that each model is supposed to be only used for the type of data and the single microscope that the data was acquired for abuse for training right. I think the question is clear. The answer is, they are only useful in my point of view they are only useful if they were uploaded there for you trained on your data. The data that your colleague acquired but on the same microscope with exactly the same setup and the same sample prep and everything. We do generally not suggest ever to use a model trained by someone on your own data. And there are exceptions to this rule, but the precondition for these exceptions to be allowed is that you are a professional that you are really that you know what you're doing. If you are new to this, or if you are really only a user that is not doing research in this direction, just don't do it. We put so much effort into bringing these technologies to Fiji bringing training to Fiji making the new Jupiter notebooks as simple as possible. It is painful. The first time the first day you will most likely use where words, but in the evening of that day, you know how to train these networks and then doing it again is just boring and just takes time, but it is easy. So I would not do any transfer of restoration models I would not suggest it. It's annoying. Did I forget something is this a good answer. Then let's hurry up as share my screen and we jump into the next deep dive. So here and here and play. So this was the question to a first Q&A session. Now we will speak about one of the latest additions in the CSB deep toolbox. And this edition is called the Noisek and manga will first entertain us with a theory part so that we know what the heck is going on behind the scenes. And then we will also hear from manga how this looks in Jupiter and then devil will take over again and show you how we made the Noisek be part of Fiji as well. So I stopped sharing my screen if I find my mouse and manga please we see your screen. But we don't hear you yet. Because you're muted. Wonderful. Yeah. So thank you for your own. So in this segment, we'll learn how noise to avoid this denoising can be utilized for better image segmentation. So when we go to the microscope. Sorry, my video is having problems. So when we go to the microscope, and we acquire images, often one of the desired readouts is the segmentation of cells and nuclei. And today, the best performing segmentation methods are deep learning based methods. The problem is that deep learning based methods require a lot of ground truth annotations to train the network in the first place. And usually these annotations have to be done manually and hence it's expensive and only available in small amounts. Which means that we'll have to discard most of our raw data for which we do not have the ground truth annotations for this is undesirable because it's a loss of information but that's how it is. Another problem that comes from the fact that the images that we acquire are usually noisy. And hence if you train such a network with raw images, which are noisy and with small amount of data, then it will not generalize so well on unseen test data, and it'll produce many mistakes. So we thought can we do better here by not discarding all the available raw noisy images for which we do not have ground truth information, but can we use them somehow. And hence we came up with the following idea. We remember the noise to avoid network, all it needs for training the denoising task is raw noisy images, which are usually available in abundance. So we thought if you could use all our noisy images to train the denoising network and then could use the features learned via denoising and combine it with a segmentation network, then we may be able to better segment the data, even in the presence of less annotations. And hence we combine these two tasks into one single network, which we call the denoising network, which is trained jointly. And during the prediction, it takes a noisy image and produces not only denoised image but also instant segmentations. But how do you train such a network in order to train it. The first thing you need is a lot of noisy data, which is usually available on which we can compute the denoising or the noise to avoid loss. Remember, we only have few ground truth annotations available, which means we can only compute the segmentation loss for only those images for which we have the ground truth. But instead of discarding all the other raw noisy images, we use them for learning the denoising task. And we combine the denoising task and the segmentation task with this hyper parameter alpha, which tells us how much waiting to give to the denoising task and the segmentation task. And once it has been done in a joint manner, we see some impressive results. So this is a data set from public domain where we have 3,800 raw noisy images, and we only have the ground truth annotations for 10 of those images. And if you train our denoising network in such data limited regime, we see that we do not only do good denoising, but we also perform good segmentations at the same time. So if you compare it to some of the baselines in the literature, we'll find out that denoising outperforms the other segmentation baselines, especially in the data limited regimes when we only use little amount of annotated data for training the segmentation part of the network. But the question now here is that how can you use it to train denoising for your own data to denoise and segment. So we have to go to this CSV deep website which Florian introduced you earlier. And here under this section, what can I do with my data. If you click this, then it will lead you to all the tools that are available here. And at the bottom of the page, you'll find in the object detection category the first two examples of a denoiser. And if you open the page, you'll find that there are many instructions on how to install it for your own computer. And you can also download the code in the form of Jupyter notebooks using the link here. Once you have downloaded the Jupyter notebooks, then we can begin the denoising workflow. The first step is data generation or data preparation. So in this step, we expect you to have three folders, the first one corresponding to training called train, inside which you'll have two different subfolders raw corresponding to all noisy images that you have. And the GT, as you see here, there are many raw noisy images. And if you look inside the GT folder, you'll find that there are only few ground truth annotations. The same holds true for the validation set, which is denoted by the Val folder here. And if you look inside the test folder, then you'll find that it only has raw subdirectory, which means this contains only the raw images that you want to denoise and segment, for which we do not have ground truth. And once the data generation is complete, we can move on to the second step, which is the training of denoising networks. So for denoising network training, we first need to load the data from the training folders and the validation folders, as we saw earlier. And then we can go to our available notebooks. At this point, I'd like to say that when you end this notebook, and if you already have experience with the noise to avoid net notebooks as Tim showed you earlier, then I think that you can easily navigate through these notebooks with some intelligent pattern matching. They're very, very similar. So once you have loaded the data, then we can start the training process. So for the training, all we need to do is just configure some parameters, and we have given some default parameters, which we have seen to work well in most of the settings, but you can feel free to play with these parameters. And after the training has been done, we can visualize the validation and training loss, and if they have converged, this means the training has been done successfully. Next is the prediction part. So in the prediction, we show the denoising network unseen raw test data and it predicts four channels, the denoised channel, then the background probability map, the foreground probability map, and the border probability map. But we need instant segmentations. And in order to get instant segmentations, we take the foreground probability map and threshold it. The threshold is computed on the validation set. The threshold is the one with which we get the best segmentation results on the validation set and we use the same threshold on the foreground channel of our test data predicted by denoising. After that we obtain the binary mass on which we can apply connected component analysis to get instant segmentations. And then we can also visualize our results, both the denoised results as well as the segmentation results. And then we can also export our models to be later used in Fiji, which Deborah will display, show you, show it to you how to use it. And if you have any problems or questions running these notebooks, then you can contact us via the ImageSEE forum or the CSVD webpage. And also if you get some good results with denoising on your data, I would love to hear from you using Twitter or any other medium. Thank you very much. And I'll hand it over to Deborah. Thank you, my guy. Let me show. I need to stop sharing this screen. Let's go back to our website and to available tools and denoisec that we just talked about. Just so that you know again where to get the resources from I'm talking about. So this is the how to use denoisec in Fiji link, which goes back to the Image Day website where you find again installation instructions. It's just the same as what noise-to-volta need the CSVD update site. And once you got that installed, you can access the denoisec plugins via plugins, CSVD denoisec. And there are plugins for prediction and for training. And on the site on the top, there's an overview which shows what I'm also now going to show you. This is about training prediction. And there's also one link which is about creating label links for the training. I will just show you one example of how to do this. And it's also described on this page. Back to Fiji. Let's assume we exported this model in Python. And we are just going again to click File, Import, BioImageIO.zip. Now we choose the denoisec model. And here it is. And now I will open another image of the same data set, which is a big, bigger, yes. And once you first predict, you will receive an image with four channels. And this is exactly the same as Manga just described it in Python. The first channel is the denoised input image. The second channel is the probability for the background. The third channel is the foreground probability and the foreground, the border. Now, one thing to note here is that these are not binary masks. So we don't automatically threshold anything in ImageJ. But you see this network was trained with actually quite a lot of labeling data. And it's really well able to segment these different sets from each other here. So that's why for the probability, which is between zero for no chance that this is border. And one, which is a high chance for a border, that this is very well able to see the difference here. All right. And as I promised you, you can also do the training in Fiji. The difference to noise avoidance that you need the training data to be organized. I have four folders here, two for the training and two for validation. There's one folder with about a hundred thousand images, which are noisy. And six, I'm using here six label links where the names correspond between the labeling and noisy image. And in the labeling, you see that the background is zero. And then each of these objects is labeled with an ID. All right. And now we go to plugins. These be deep. Denoy sec and train and you have to provide now the path to these four folders, which is exactly what I just showed you. About the other advanced options, there was also a question about this. This is described on the website. And we also, I mentioned the batch dimension, for example, or that the patch shape. This, I have to, for example, I'm training here on my notebook, which has an Nvidia graphics card with only two gigabytes of RAM. So I cannot have a huge batch size or patch shape here. But this, of course, will also affect training, but you can also use these parameters if you don't have a high quality. Set up. Yes. Just be aware that this might affect the training. So I'm keeping all of this like it is. And during training, after the first epoch is done, you will also get a preview window, which looks a bit different than the one in noise to void. Here it is. So this is five tiles image where the first one is the noisy input image and the denoised image, the tile in the middle, the background probability, then the foreground probability and blast the border probability. And you can nicely follow how well this network is performing. You see in this case that it has some trouble segmenting these two very closely aligning cells. So in this case, I would probably do a few more labelings and try this again. Now when training is done, I promise you to tell you how to reuse these trained models. So you get two models after training. One is the model which is from the very last training step. This is the one on the left and the one on the right is the model where the validation loss was lost. You can also see this if you go to the training tab of the model. There is a parameter called steps finished and this should differ between the two models. And for noise to void and denoise it really makes sense to always use the one from the last training step because we do not have the full ground truth data available here. And in the preview, this is a whole stack of this is the whole training batch so you can actually go also during the training through the whole batch and look at different scenarios here. You see that the probability maps here are not clipped between zero and one you see much more about what's happening during the training. If any question about this is my sound a bit obscure now the street page or ask me. Now we always get asked how to do labeling I guess many of you know how to do labeling. Um, but since this question comes up I will show you the you clap way of doing it, which is the lab lab kit. And that kid was written by my tears us who's also from our lab. And lab kid can also do automatic segmentation but we are now just going to use it to annotate an image. So you insert a lab kid update site this is available in the default list of update sites. Once you got this, you can open your data. I'm just going to open this one here. Yes. And I'm called plug ins segmentation. And that kid. And black provides you with some drawing tools, which you can now use to annotate your data. So, for example, I now choose this pencil here. And then I paint the outline. And then I fill it with the bucket tool. And it makes sense to be very precise here for the noisy. So rather make a few labelings, which are very precise than doing more which are not precise. You will have a better performance. I'm just storing this labeling, because you might want to continue with this whole process later so here you now see how I'm reopening my data just starting lab kids again and opening a stored labeling so that you don't have to watch me label this whole thing because what you know I think you really have to label the whole image here that's a super important don't leave something out. So bear with me as I'm just doing the very last thing here. And once this is done. We can export this labeling show labeling and image day and this is not just a mask with a background to zero and the foreground is one but this is not enough for the noisy. We still need the labeling with an object ID per cell. You can for example use this video objects counter for that. Just make sure that he and the options you deselect exclude objects from edges so that really the whole mask is used. And once you press okay. You now have an image with the real labeling which you can store to this can use for the noise sake. Now, and there was already a webinar by Martin Weigert and about status where he also showed how to do labeling is there's also Q path I heard in Pantera for example which works really well. Whatever you use we have for our tools we have heard some love it some hate it. If you have any feelings about our tools, please let us know, because that's the only way we can improve it we are not biologists. We are sitting here building the tools and not sitting in the lab actually using it so if you think this should be useful for you and it's not please let us know so that we can make it better. And with that I think we are opening for questions again. Yes. Thank you very much. And I am now going through the questions that are marked for during Q&A. And I will do it in a backwards order. So let's start with the first one. Does the noise sake do better than noise to void, or does the noise sake replace noise to void, when I will use the denoist results for segmentation. Manga, do you mind answering. Yeah, sure. So, a good question. The answer is, since the denoist networks perform two tasks at the same time, the segmentation and the denoising tasks. So the network has to is not doing a dedicated task of denoising alone as it does in noise to void. So it is to be expected that the performance of the network will be slightly worse compared to the noise to void. And this is what we actually see on our data. So on some of the data sets we see that its performance is almost as good as noise to void. But on some others, it's also slightly worse. Yeah, and we provide a detailed comparison of the same in in our paper as well. Just for reference. Yeah, I think it's absolutely right what you say Manga, the idea of the noise sake is the denoising is a byproduct. And it can be useful and so we give it back to you because for many simple data sets the result will still be very good. But really the idea of the noise sake is to give you better segmentation results by letting a network learn how to denoise. Once the network kind of understands where are objects where are not objects where are certain structures and learns how to remove the noise. Then somehow this transfers well also in the ability to segment, because segmentation is somehow a very extreme form of denoising right where you say zero to the whole background and one to everything foreground. So since these two types of of tasks are so related. We figured that doing it them jointly might actually help and it turns out that if you have very little data labeled in this regime where you are kind of bored of making circles around objects. Then doing it together with a lot of noisy data and then profiting from the network learning how to denoise in order to do also better for segmentation makes actually a lot of sense and this is exactly where I would see fit for this method if this is your use case. But is it a very young method so if you happen to try it and you're not satisfied let us know if you happen to try it and you're very satisfied please let us know as well. At the moment we would just be very happy to get feedback on this method and see where is the strengths where should we kind of maybe improve it. Where is it useful for people in the wild. Okay, next Q&A question. What is the segmentation part of the noise that based on stardust. Who would like to give it a go. And anyone could I mean did you. So at the moment it's based on three class unit where the three classes are foreground class background class and the border class. But in principle, actually the if you just look at the theory there's nothing which says that it should be restricted to three class unit, and it could well be applied to stardust or. One of your papers that is pretty last this year is showing in a non joint setup, like the predecessor of D noisack. We also show that if you first train a denoising network and then a stardust on top of it. And that that also actually leads to positive research but D noisack as it is at the moment in CSB deep and as you can download it and use it as one girl said does a three class unit segmentation. Okay. I think I read somewhere that care can be used like the label free prediction Fnet from Ellen Institute to artificially label to the bright field images. Is this correct. I will answer. Yes, it is. And if you give bright field as an input, and then as ground truth image, you give a fluorescence image, and these two images are very well aligned, right. Then a care network can very well train to predict the expression of this fluorescence marker. As I said, we tried that in multiple courses we gave around the world, with, with more than one person having this idea and results were actually really good. Care was never intended to do this. It can do it because for many pairs of of bright field and and fluorescence, there is a statistical correlation that the unit can pick up on. If you want to do that at a professional level. I would go with the methods that are published for this purpose. But nothing stops you from giving it a try in care and if the results are good enough then be our guest. Right. It's actually every time a student tries and we get good results I find it very exciting but it is not the original intention of care. I don't understand why you go to all the trouble to make these models available through bio image IO, but then tell people not to use them on their own data. Am I missing something. Maybe. So the, the, the models to is not intended to be a model to only for care and noise to avoid and the noise like models. It is intended to be a model to for all bio image analysis models that make sense. And for example, the In 2D has trained models that translate very well on a number of different data sets and for segmentation. This is in general maybe better transferable but also the people our colleagues from elastic use the model so and will use the model more and more to distribute some models that you can then use either within elastic or just within pytorch. And in general it is just a marketplace for models that can be used or reuse. So how if you are in a facility and you create a model for someone of course you can go with a memory stick to that person and hand the model over. But if there is an automated like in the cloud solution via a model so that of course adds to the convenience of the users right and so we are trying to build the model so in such a way that all these use cases reuse but also just transfer are actually covered and there was another new buyer seminar by the deep image shape people I think it was actually with the deep image shape people and they have very, very similar mindset here. So to make a long story, even longer, I end this question by saying, we don't know exactly what the model schools will do in five years time, because we are just about to build them, and the world has to wiggle itself in place right but everything I said before, good possibilities and we are eager to find out what are the most convenient ways that people will want to use the model and also we are still figuring out what models can be transferred and how can we ensure that that nothing unexpected happens Tim I think you raised your hand I'm happy to chip in. Yes, one other important use case for models who is reproducibility. If you have an experiment and you train your network and you write the paper, you want to have people be able to look at your data and also get the model you train on this data. And the model to provides this kind of framework that you can publish also your model. You get essentially a unique identifier. And you know if I download this model it is to the last epsilon exactly the model that was used on this data, the models to will also contain all the metadata to the data that it was trained on and to the data it was applied on. So in principle 100% reproduced a result. And that is of course as Tim pointed out, positive as well. Okay, and I think I forgot one question but we will. Gloria, sorry. So, I'm sorry to cut out your enthusiasm, but I think it's time to go for the final outlook and they're all the participants. We will post every answer in the forum. Again, I was just about to say that I forgot one question but I will not answer it now in the interest of time. And we will now move on to the very last part which is just a small look into the future by Alex, because you might have noticed the whole care thing popped up just a few years ago. And then many of these methods kind of came one after another. And it is not that we end now, but in the next year you can expect that many additional tools will be ingested into the CSB deep and brought to you and they already exist in the research realm and Alex will quickly give us a heads up on what you can expect in the next few months to come. So, yeah, this is for me the most exciting part because I get to tell you what we're currently doing research wise. So, yeah, if you're if you're a little bit patient you can experience these tools and in the same way as the ones that we've discussed today in more detail. So, I have to share my screen again. Yeah, what are the next steps, in particular with respect to unsupervised denoising so one thing that I want to quickly talk to you about is probabilistic noise to void. This is something where we try to improve upon the noise to void. I've talked about earlier. We've published this earlier this year, but it's not yet part of this CSB deep collection that we've seen today, but there's code for it and you can find it. So, if we look at the original denoising problem and we have an example image like this, and we apply a traditional care network that we train on this data and then use it to denoise it. You get quite a nice result looking like this. And if you compare this to a noise to void result, you can be a little bit disappointed. Both with respect to Pearson are where this is a measure of the quality higher is better, and also visually. Yeah, and there was previously the question about can we see the Pearson are differences. And yes, so I think we can see it. And unfortunately it looks it doesn't look as nice as the traditional care so why is this mentioned this earlier before. So, we are not making use of the full information, because we are restricting ourselves using this blind spot receptive field, but here I show you already the solution to this, which is probabilistic noise to void. And probabilistic noise to void allows us really to narrow this gap between the supervised care and unsupervised noise to vote methods and I can just give you a brief intuition here on how it really works. So, we still use the blind spot network. But here we don't use it to predict the value for each pixel but instead we predict the probability distribution for each pixel, the distribution of a possible signals at the pixel. We combine this with a probabilistic model for the imaging noise. And by doing so we can reintroduce this missing information that that we are excluding by using the blind spot, and we can produce much improved results. So if you compare this to traditional training in traditional training you have one network which is the black box, and then probabilistic noise to void you have no tool components you have the blind spot network, but you also have this noise model. From the applicability side of things so in care training, when you do new experiment let's say you label your sample differently. You have to retrain your network for this new data, because the statistics have changed. And unfortunately you also have to go back to the microscope and acquire new training data of this type. Now in PN2V, you also have to retrain your blind spot network. You can again use the very same data that you want to denoise to train it again. You don't have to acquire additional training data. But what about this noise model? Well, this is something that really depends only on your detector. So if your detector settings are the same, you have to measure this only once, and then you can reuse it. So the second exciting thing I want to talk to you about is what we call diff-noising, and this is super new. You can find it on archive at the moment. And here we want to denoise in a new different way, providing diverse solutions that account for the uncertainty in our data. So I know that in all likelihood a lot of you have data that looks roughly like this. Which is like just horribly noisy, and you can hardly tell what's inside there, and we have to admit that. So we cannot be sure what is really the true signal that led to this input, even as humans. Now, this is the true ground truth, but we don't know it. So if you train a network, whichever network we use, if you use KER or N2V or PN2V, this network will have to make a compromise in these predictions in the end. And if we produce something like this, what we call an MMSCE estimate, which is essentially a compromise between all the possible different solutions that might explain the noisy input. And you can see this, the network is, for example, here, not sure is this really an N, or is it maybe an H, right, or is this a U, or is it maybe two I's, right, with the dots here. So we don't really know it. And the result is this compromise, which would never occur in reality, right? This is not nice. So in def-noising, we've come up with a way to train a system to provide to you not one estimate, but to allow you to sample a lot of estimates that all could explain this input. We can explain the same image in a lot of different ways. And interestingly, you can combine all these hypotheses by simply averaging, and this will actually produce the same type of estimate that N2V or KER would give you for the same data. But obviously, the most interesting thing is how can we use all these diverse solutions downstream, for example, if you want to do detection or segmentation or whatever we want to measure on data. And this is, I think, a very exciting prospect. So here we've applied this to a bunch of real microscopy data, and I show here only two samples, two sample hypotheses from the system, but it could easily generate 10,000 of those, right? And just, we are only showing two of you today. And I want to illustrate here that starting out with this horribly noisy input, we really don't know what exactly is going on. We get really diverse solutions that might explain the same data if you compare these two. Now, here it's really diverse in this case, because there's really a ton of noise. If you look at this example, there is less uncertainty because there's less noise. And this means that the sampled outputs are also closer together and not as diverse, but still we can see here subtle differences in the sampled results. Yeah, with this, I want to say thank you to all the co-authors who've contributed to the project that we talked about today, and also to the UCLAP in general. And a special thanks here goes out to the software engineers that allow us to really make this available to you as a more usable type of solution instead of like research quality annoying code that you have to fight your way through. So, yeah, lastly also I want to say thank you to the facilities at MPR CBG, the LMF and the computer department, which allow us to do this work. So thanks for your attention. I think we go back to the final round of questions. Yeah, can you stop sharing the screen then I will. And I think Rocco could potentially help us to understand how much time is remaining. But I think I'm sorry to say nothing so we can of course overrun if attendees they are interested and just address few questions and there is a message from Julian about new bias academy. Okay, let me first wrap up and this will only take 30 seconds. So if, if this was all overwhelming for you, the suggested entry points are the website, you will find a lot of things there. We saw it also many of the speakers today went from the website to the examples. So many links should be there. But if this is at some point not enough you have other questions you have problems. Go to the image J forum, so sorry to the forum that image.sc and ask your question there. And this actually is good advice, not only in the context of CSPD, but in general for bio image analysis. It's a very vivid young community, very friendly people, many, many tools are there. So if you have an analysis question it might very well be that some feature guru gives you one possible answer. While somebody from Q path or a Python centric person gives you a different solution or potential solution so I can only if you are not already a member of this if this vibrant community suggests you to to get there and make friends there because it's a really great group of people. And then I want to just also say thanks to all the speakers of today. We put out a lot of new versions of codes website, everything happened overnight in the last few days it was absolutely accelerating which is just like the PI word for exhausting. And we also want to thank the, the, the people that we work very closely with on some of the methods we presented today, and also the labs, with which we work together on the model zoo and similar initiatives at the moment. I could think a thousand people more. But first of all, I would always forget somebody so I cut a draw a line just here and end this presentation and the new bias people will now tell me how we continue. Okay, thanks for and so this is Julian speaking I'll just have a very short message probably 30 seconds and to just inform like the people who are still here that new bias Academy would actually take a break for the summer. So this was the last webinar of a long and very successful series. Thanks to Florian and the team for actually closing it with such a nice webinar. So just to inform you like over the last three months, more than 10,000 people have registered to these webinars so we consider it like a very nice success and actually the feedback from everyone was overwhelming. The only purpose of this message is to actually thank the people who are behind it and name them. So I will just name them bluntly and linearly in alphabetical order of their family name. So, we want to thank Rocco D'Antuono, Olivier Burri, Bram von der Burke, Fabrice Cordelia and Nass Faseli, Alfred Goulany, Roman Guillet, Robert Hazer, Anna Clem, Florian Levé, Mario Nouveau, Kota Mura, Paula Sampaio, Mathilda Souza, and Christian Tisha. All these people have been working hard in the background behind the scenes to actually make it work. So the idea is to be back in September with the new lists and new formats hopefully by mid end of September. So from now if you want and if people want to stay and you can actually address questions and answers further on until you get bored. Okay, so I think if I see it right all the questions in the Q&A window are currently answered or not copied over into the system that we are working on. And if that's the case, I think there is maybe one that I can read out loud by Christian Tisha actually I should not say that but it is him. Looks like you are using different ideas for segmentation than in Stardust. If that's the case, how would the noise compare to Stardust? It's a very good question. I would expect that in many, many cases Stardust would produce better results because Stardust is a very versatile segmentation model. But there is this one window where you have very, very little annotations on a otherwise very large dataset. You have maybe a few gigabytes of noisy images, hopefully very noisy because this is where the noise tag will do best. And you have only a few labors. I think in this regime, the noise tag might be useful. Additionally, it is a bit different in what output it produces. While Stardust gives you stark convex shapes, the noise tag gives you these foreground background and border probabilities. So that might, depending on what you want to do with these outputs downstream also enable you to feed it into different tools and two different things with it. I have little reason to believe that the noise tag will give you vastly better results than Stardust. I think it's more the contrary. So I was waiting until somebody asked another question but I'm also the person having to find out what questions to ask. I'm not sure if there is also a rock or if there's also a possibility now that we are like in overtime, right? That people could also just raise a hand and ask questions live because I would be happy to, you know, like, If you wish so, they can signal it, they can write in the chat and we will do it. So if you if you want to kind of speak up signal it and rock or might give you the microphone, the virtual microphone. Otherwise, let me quickly check. I think there's something else here. That's the noise tag that I know this we had already. Can I use the noise tag for 3D data? Only after coming to the UGLAB and doing an internship or using it as a result. So nothing makes this impossible except the non-existence of the source code that would do it. There is no fundamental problem with it unless you think that not being able to do it in this current real world is a fundamental problem. Okay, I'm not sure if you get me, but it would need us to kind of put this method in a 3D realm and this is manpower, person power that has to put into the project at some point. I really dearly hope that it would happen but it did not happen in the past. Potentially, yes. And if you happen to have time and you want to do it, then I invite you to hook up with us and make it happen. Maybe you are looking for a master's thesis or whatever that might be a good occasion to do such a project, bachelor's thesis. So this could be the first webinar of September if you are able to get this result. Sure. Yeah, and so with that I think that's all and I thank again all these speakers and UGLAB for joining in such a number and with great contribution on all the aspects of this topic. So thanks again, we will stop recording. We will also please ask the participant to go out of the webinar or a set that will exclude them. And thanks again for participating. I think also you and the whole new bias crew for making this wonderful and very valuable events and also including us and giving us a chance to show showcase some of the tools that we are involved in generating. We are very appreciative of not only these events but of new bias in general. I think it's absolutely a great, a great addition to our research fields.