 it's my pleasure to introduce the next speaker of the day, Peter Claes, from the KU Löwen. After career steps in Löwen, Melbourne and Oxford, Peter is now associate professor KU Löwen and he holds a dual appointment at the Department of Electrical Engineering, Processing of Speech and Images and the Department of Human Genetics and that's a particular flavor and a particular expertise in his research that he knows about this boundary between imaging and genetics. He's an honorary fellow at the Murdoch Children's Research Institute in Melbourne and he organized two intentional workshops on facial genetics, so exactly this boundary I described. He's also a member of the Löwen Institute for Human Genomics and Society. So we are very happy to have him here to learn more about medical imaging and analysis in the era of big data, which is a fundamental column and part of our networks topic on machine learning for personalized medicine. So thank you for joining us today and we're looking forward to your talk. Okay, thank you Carson for the kind introduction and thank you also for inviting me here today. As you mentioned, the topic of my talk will be about medical imaging and analysis in the era of big data. Just a brief outline to begin with, I will talk about why I consider imaging to be big data. I will talk about different imaging modalities, the analysis and if time permits I will also dive into some particular applications that I myself have been focused on in the past 15 years or so. So imaging and big data, it's all related to the idea of deep phenotyping, which is quite popular today. And when you look at deep phenotyping, you actually are looking at three distinct aspects. First of all, deep phenotyping involves intensive, so a lot of measurements of the same thing. Extensive, many different things are being measured of a person and large scale. You're trying to measure these on a very large population of individuals. So deep phenotyping definitely arises from intensive, extensive and large scale data collections. If I'm looking at images, images by themselves are big and they're basically underpin the intensive phenotyping of deep phenotyping. So an image is collecting tons of data in a single shot. And in fact, images have been increased in data size over many, many years. As you are all familiar with the cell phone in the early days, you had a one or two megabyte pixel camera. The cell phones of today's, they have 15 megabyte pixels and many more. So the resolution of these images is increasing. So the amount of data that an image is collecting is more and more. So the images are definitely big. They've always been big and they're just getting bigger. That's basically the feedback here. At the same time, I also want to reduce another kind of concept that is related to imaging, which is basically feature extraction and pattern recognition. Often images are used to do some recognition. And this is done by feature extraction here simply as an example. If you want to classify apples from peers, what you try to do is you try to extract some features based on shape and color, for example. And those two features are allowing you to basically recognize apples from peers, apples being red and round and peers being green and a bit of an oval shape. So if you combine feature extraction and pattern recognition with big data, well, basically, those are the two key ingredients for the era of deep learning as we are witnessing today. Now, deep learning itself is not new. It is a subfield of a larger field known as artificial intelligence. And artificial intelligence has been popular since the 80s and the 90s. However, the way intelligence is being approached has changed over the years going from the implementation of knowledge, expert systems to shallow learning, as we say, from imaging descriptors to deep learning from raw images. Nevertheless, image data has been part of AI from the beginning, like image analysis have been using AI since the 50s, 60s and 70s to do the first imaging processing and analysis pipelines. And of course, a lot of deep learning today has become successful, particularly by working on image data. The only distinction of AI in the past and deep learning today is the access to large scale data. Large scale data is a key ingredient for deep learning and machine learning today to be applicable on data. If we simply look at these deep learning networks, I think we're all familiar based on this summer school that these consist of artificial neural networks and in itself is just a connection of very simple units that on themselves are a bit naive. But when combined in multitude, they really can solve complex tasks. And often the comparison is being made to the human brain, which also consists of primary elements of neurons that are being connected into a very intelligent system being ourselves. And artificial neural networks are trying to mimic the same kind of architecture that is observed in the human brain. The key difference of deep learning today in machine learning, let's say 20 years ago, is the following. If I look at an image in the past, we spend a lot of time thinking about what kind of features would I be extracting from the image knowing that these features could potentially solve a problem. So you would do something known as manual or engineered feature extraction. In the past, if you were doing a PhD in engineering, basically if you found a new feature, you finished your PhD. That was the way to get your PhD in the past. Today with deep learning, we have an end-to-end situation. So basically this engineered feature extraction step is skipped and the deep learning network is going to learn automatically to what extent features need to learn to optimally solve a particular task. So now if you want to obtain a PhD in engineering, it's a completely different ballgame and it's all about getting enough data, sensitive data, and to control your data for these deep learning networks to be applicable. And they're very successful. We have seen this especially on image data and has progressed into many other different fields. But I think deep learning has really gained a boost by solving very complex imaging tasks. And the first and the most well-known imaging task is the one of ImageNet where millions of images are being collected and categorized in groups. And basically deep learning has cracked the ability to, given the image, classify the object that is seen in the image as you can see here, the identification of flowers, elephants, boats, and so on. And this is where really the feature learning of deep learning has excelled in solving these tasks. So in other words, if you look at large-scale imaging data sets, these are the new gold in data science and imaging analysis today. And I'm just listing here a few. There are initiatives of collecting a lot of data on individuals, including imaging data. And I think the best known here might be the UK Biobank, which has also been collecting MRI imaging on participants. But essentially these kinds of collections are providing the next generation of machine learning to be applied. If I look at imaging modalities, the second part of the talk, there are quite a few. And if you're interested in more details, I'm just going to go briefly over them because I won't have the time to dive into much detail. But for those who are interested, I do recommend the following book on the fundamentals of medical imaging by Professor Paul Sutons, who also was my supervisor for my PhD. But just to go over a few of these modalities, I think something that is familiar to all of us is simply the two-dimensional photography. Two-dimensional photography is still very used in clinical routines, especially in craniofacial surgery, for example, where images are being taken of the patient before and after treatment. If you move from two-dimensional photography today, we can also do three-dimensional photography. And this is done by three-dimensional scanner. And essentially, it is like multiple cameras are looking at an object from different points of view. And together, they can triangulate the image back into 3D. It's much like how our two eyes are working. We have two eyes of two different cameras, slightly deviating apart in distance. So together, we can basically perceive depth in the surroundings. And these cameras work in the same way. But a 3D image is essentially a point cloud of the collection, which is being connected into a wireframe. And then using computer graphics, you can render this into a surface. You can put some light, and you can really observe a continuous surface. And typically, you can just simply add some texture to make it more realistic and more interesting to look at. If we move from 3D, oh, that was one button too fast. My apologies. So today, we've also moved into the era of four-dimensional imaging. And it's the same again. You have these spots that are looking at the object. But now the object is moving, and the pots are essentially recording movies. So this is a dynamic 3D or sometimes called four-dimensional photography. And this is quite interesting. Again, like in surgery, like in facial palsy, so you have some part of the face that lost its nerve function. And the surgeons try to repair it. They can then reassess the movement in the face or other parts, like in orthopedics, if your knee is bend it properly and stuff like that. So this is kind of an interesting new imaging modality that is finding its way into the clinical practice more slowly. If you look at, well, the more medical modalities, the first one that is best known is X-ray imaging. X-rays are essentially electromagnetic waves that emit a photon-based radiation. And that were discovered by Wilhelm Konrad-Röntgen in 1895. Basically, what happens here is like X-ray vision. And like you see also in airports, like X-rays are scattered, and they move through the object, and they're either absorbed or scattered by the internal structures. So that an X-ray collector behind the object records the intensity of the X-ray. So if you have a substance that absorbs all the X-ray, you will have a darker region. If you have something that scatters the X-ray through, you will have a light. So you have intensity changes like grayscale value. And here, for example, you have an X-ray of the chest. So you can look at ribs and the lungs of the person. So you really see through the body, essentially. A nice innovation into X-ray imaging is X-ray computer tomography. And it's also using X-rays, as it says in the title. But the clever thing is the manner in which the X-rays are being imaged. So there is a whole setup now involved with a patient lying on a table and being pulled into a rotating device. And it's basically accumulating X-ray imaging. And together, thanks to computer algorithms, they are able to reconstruct a three-dimensional image, slice by slice. So you see here a movie where we're actually going slice by slice through the patient's body. But instead of having a planar two-dimensional projection of a three-dimensional structure, thanks to the rotating movement around the patient, you are actually able to reconstruct the full body in three dimensions. And here, depending again on the tissue, like heart tissue, it's absorbent and soft to less. So the intensity changes are going to display different types of soft tissue and heart tissue in the of the internal body. And a similar kind of adaptation in this technology, but more broadly applicable to orthodontics, is known as combing CT. Again, it's an X-ray imaging device. But instead of having a rotating slice by slice, you are now beaming a full cone of X-rays at the same time. And the result is essentially this, that you need less acquisition time and hence less radiation to make a full three-dimensional image. And this is one of the reasons why this is so popular in orthodontics, because they're like the the risk of exposure to treatment gain is not so high. So you don't want to have a system that gives a lot of X-ray radiation to the patient to be imaged. But essentially, these are all X-ray based imaging. But you can do more. You can also say, you can also add some contrast agents to the to the patient's bloodstream, for example. And typically, these are then radio opaque material. So if they get, they really absorb the X-rays. And hence they really, the contrast of these structures is being enhanced in the image. So here, there is a contrast agent being admitted to the blood flow to the bloodstream of the person. And as such, you can really highlight the blood veins in the hands on the left or in the neck and head and brain on the right. This is what is called as CT angiography. So it's by administering an additional contrast agent into the bloodstream. Other types of CT scanning are dynamic, like imaging the heart is not an easy, easy thing to do. Where there the scanner is essentially aligned with the rhythm of the heart. And you kind of take pictures at a certain stage of the heart cyclists. As well as if you're using the profusion, you can continuously image the patient and actually follow the flow of the contrast agent through the veins and the blood vessels. And as such, you can investigate if certain parts in the brain in this case are being under, if they have enough blood, yes or no. And in cases of a stroke, there will be regions in the brain where this imaging is then revealing that there is no blood flow to that part in the brain. And this is then to localize the obstruction and then operate or trying to treat it as such. So it's known as dynamic CT. So basically it's like a movie of CT scans. That's another way of looking at it. And it gives you some dynamic information. Another type of imaging modality that's very popular. And while often using complement to CT scanning is magnetic resonance imaging. It's very different. It's also non-invasive. And it's using not X-ray beams, but it's using a magnetic field. And basically, so the the idea is for when when different tissues are being are being exposed to a certain magnetic phase, they're out there at the magnetic phase of the atoms of the tissue will change under the influence of this external magnetic field. And this change can actually be visualized. And depending on the tissue, there's going to be a different reaction to the magnetic field. And again, you have different intensities. And hence, you can discriminate different tissue types from each other. But what is more interesting as well is that in MRI, you can also play with the magnetic field sequence properties. And hence, you can put more or less emphasis on different tissues. Here are a few different MRI sequences. And I'm just going to highlight the first one is a T1 is known as a T1 MRI sequence. And that is basically used to image the brain anatomy, because in this kind of image, we can really differentiate well between the gray matter, the white matter, and the CSF in the brain. A flare image is another example in which the magnetic field sequence is different. But that's an image that is typically used to emphasize tumors. So different types of tissue of tumor tissue will be emphasized using flare imaging and other pathologies. And that's the nice thing about MRI imaging. You can really play with the sequences and hence emphasize different tissues of interest, either the anatomy or some pathological aspects. Similar to CT, here as well, you can also add contrast agents. But instead of having a radio or park contrast agents, this would be something that is paramagnetic. So something that is highly influenced by the magnetic field and hence lights up in the image of the MRI. Here you can see again, this contrast agent is again administered to the blood flow. And also with this contrast agent, you can highlight the blood vessels and the blood stream as such in certain parts. Another really cool thing for MRI imaging is something known as functional MRI imaging. And both imaging or blood oxygen level dependent imaging fMRI is one example of this. And essentially what it does is it's following the following assumption. So when when certain parts in the brain get activated, they're going to require some oxygen consumption. So there's going to be an increased blood flow to these parts in the brain and it's going to as such going to be an oxygen concentration. Now, what is of interest is that the parts that have a high level of oxygen have a different magnetic property and the parts in the brain have a lower level of oxygen. So hence, you can emphasize the parts in the brain that are being activated during certain tasks. Here in the slides, you basically see the activation of the brain when a person is holding his breath. So basically getting an oxygen deprived situation. On the top, you see like holding your breath after 14 seconds. And at the bottom, you see, well, the blood trying to compensate for the lack of oxygen after 27 seconds. So you can really see that the whole brain is being lit up by this imaging. Now you can use this in the following setups, for example, you can actually ask people to do something or to look at something. And then you can monitor the activation of oxygen and hence the blood flow in the brain as such. On the left, you see an example of the activation of brain regions that are involved in speech and language on the top. But interestingly, it has shown that the same regions in the brain are also activated by gesture language. So not spoken language, but gesture language and spoken language turn out to be processed by the same regions in the brain. And this is visualized thanks to the bold functional MRI imaging. On the right, you can see activations of the brain for the motoric and sensory skills. And so the person is trying to perform a motoric exercise. And by at the same time, imaging the brain, you can actually see which regions in the brain are responsible for your motoric and sensory skills as such. So this is what is known as functional MRI. The next modality down, which is also very interesting, is known as nuclear medicine imaging. And it sometimes sometimes sounds like, like dangerous at the nuclear aspect. And in fact, it's not so dangerous, but it's quite distinct to the other imaging modalities, because here, the imaging source is not being from the outside. But essentially, what happens with nuclear imaging medicine imaging is that there's a tracer molecule that holds an unstable isotope, like a radionucleotide that is being injected again into the body. So it's another form of functional imaging techniques, but it's using radio tracers to visualize and measure changes. And for example, metabolic processes and other physiological activities, including again, blood flow, but also regional chemical composition and absorption. As I mentioned, the tracer module is administered with an unstable isotope. And there are two types of isotopes, positron emission tomography like a PET scan, or a single photon emission computed tomography aspect scan. And basically in the latter, for example, the isotope is emitting gamma rays, and it's the gamma rays that is being that are being emitted. And essentially, if the tracer is moving down the body, and it's being absorbed by certain metabolic processes, then there the tracer will light up. And basically, that's why you see all these, well, apparently little lamps lighting up in the body and exposing, well, the proper functioning of a metabolic process or the malfunctioning of a metabolic process, of course, as well. And this kind of imaging is typically combined with an anatomical imaging. So for example, MRI, you can combine with a PET so that you can also localize the metabolic processes. So this is then known as a PET MRI, but you also have a PET CT or a SPECT CT. In fact, PET CT is a very useful in imaging the location activity of cancer and metastasis of cancer. But it's typically a combined imaging, the PET or SPECT together with an MRI or CT to gain some anatomical localization of the activities. If I look at all the modalities that I've gone through so far, these are all electromagnetic waves. And the main difference between them is where exactly these are situated on the electromagnetic wave spectrum. If I go from the shortest waves with the highest frequency we have, on the left here, the gamma rays used in nuclear medicine imaging. And we have the X-rays used in CT and combim CT. Then there is the aspect of visual light in the 2D, 3D and 4D photography, but also endoscopy, something I haven't discussed here. And then there is thermal imaging, which I also haven't discussed, but at the other end of the spectrum we have radiomagnetic waves as used in MRI. So all of these modalities are basically electromagnetic waves and they're basically different from each other in the wavelength of the imaging modality used. But there is one more imaging modality that is not an electromagnetic wave, and this is known as ultrasound. Ultrasound is based on the time of flight of ultra-high sound waves. So in contrast to the other imaging modality, it's a sound wave and not an electromagnetic wave. Sound waves as well, they undergo reflection and refraction depending on the underlying tissue. And again, for example, if I have a sound wave and it hits on bone, it has a perfect reflection. But if it hits off tissue, it's going to be refracted. And if I basically send down a sound wave and it's being reflected and I measure the time of flight between sending it out and receiving it back, you can then form an image. And I think every one of us or the most of us are most familiar with ultrasound thanks to well, the ultrasound imaginations of fetus imaging during pregnancy and many parents and take the very first image of their child and show it to the rest of their family. And ultrasound as well, much like other modalities, have really known an expansion in technology. And we can go from the traditional two-dimensional ultrasounds here to now today, we can also do a 3D ultrasound. We can also have a 4D ultrasound, so a movie of the baby, and then the high definition ultrasound. So here as well, there is a lot of technical advancements that took place to increase the use and value of these images. Independent of the imaging modality, the complexity of an image data is mainly defined by three factors. And the first factor is the resolution as illustrated on the top. A resolution, this is an aspect also where each manufacturer of an imaging device tries to improve their systems with. As I mentioned in the beginning, the cell phones in the early days, we had a one or two megabyte pixel camera. Today we have cameras with a much, much higher resolution. So with lower resolutions, the problem there, we basically not always see what we want to see. For a, the resolution is too low, like the five by five, we don't see that the letter R is actually displayed. And the advantage of increasing the resolution is that you have a sharper vision of what is, well, for example, the R in this illustration, but essentially also in the medical imaging at the better delineation of the heart or the liver or other organs are becoming possible if the resolution increases. Another aspect that is clearly involved in imaging is the signal-to-noise noise ratio. So any ray or wave detector introduces noise into the signal. And if we look at the imaging modalities we've covered so far, examples of high signal-to-noise imaging, so where there is a lot of signal and not a lot of noise, we talk about photography and CT. They have a very good signal-to-noise ratio. In contrast, when you look at MRI and ultrasound, it's the opposite. The signal-to-noise ratio is low. So there's a lot of noise in these images. And the last aspect that is contributing to the complexity of image data is imaging artifacts. And these are specific for each imaging modality, but they always introduce some artifacts. You can see some strict artifacts here. Essentially, this is a CT scan of a mandibular with teat fillings and the amalgam fillings, the metal fillings in the teat, are basically creating artifacts as they interact with the x-ray. They really scatter the x-ray beams, essentially. This is known as metal streak artifacts. So these are the three aspects to keep in mind when you work on image data. So the higher the resolution, the better. The higher the signal-to-noise ratio, also the higher the quality. And the lower the artifacts, the better as well to do some analysis, which brings me to the next part of this talk. What about the analysis itself? So we've covered some modalities. If I look at imaging, processing, and analysis, I make a distinction between, well, simple operations on imaging, segmentation, registration, representation, recognition, and generation. I will go by these one by one to explain to you what it all means and what the purpose of these can be in the medical setting. First of all, simple imaging operations. A key example here is what is known as histogram transformation. It's a very basic operation. And I always say, lucky, you can even open Adobe Photoshop and apply this kind of transformation to any image you would import there. Histogram transformations are quite popular in Photoshop as well. It basically involves a transformation of the gray value histogram. This is to enhance and emphasize different gray value intervals. In this example, this is the original range of gray values. If you really want to focus, let's say, on the parts of the lungs and have the details of the lungs, you can transform the histogram to emphasize that part. And you will see at the expense of the other surrounding tissue. So you cannot really discriminate anymore here, but you really see the details of the lungs. And the opposite, you can also transform the same histogram differently by, hence, emphasizing the surrounding tissues but having no detail left in the lungs whatsoever. So histogram transformations are often used in radiological examinations. If I want to see lung damage, for example, as a radiographer, then I would emphasize the structures in the lungs much more, especially for COVID investigations today and the damage that they do in the lungs, you would perform such a transformation to emphasize any damage done to the lungs. But they're very simple to do. But there are some other simple things that we can do. And they already generate some, well, very interesting applications and aspects. And the other operations that I'm talking about here are known as linear image filters. So a filter is something like this. It's a small window. And it has some weights. And this filter is being convolved over the image, as you can see here. So the blue one is my image. And the filter is being basically slided up from top to bottom and to below. And essentially, I'm applying this filter to do certain operations. If I'm using, for example, the filter here in the middle, which is basically 111. And essentially what this does is it's counting up the value of nine pixels and taking the average. So this is a smoothing filter. If I apply that filter to this image, I get a smoothed version of that same image. If I apply a filter like this or like that, I will be detecting edges in the one hand vertical edges on the other hand horizontal edges. If I apply multiple of these filters together, I can basically run an edge detection onto the image. And then you can also start playing around with the filtered results. You can add one, the smoothed to the edge detector. And then you have something what is known as image sharpening. So the result here is a sharpening version of the original image. And this is the key of image filters. And in the past, again, this was basically what engineers were looking at, trying to find out the filters to perform certain tasks. And I think you could have guessed why am I saying that? Because today, this task has been taken over by deep learning, but deep learning in essence is again, simply filtering deep learning on imaging using convolutional neural networks are nothing more than filtering images in the same way we have been filtering images for 30 years or more. But the main difference is instead of trying to come up with designed filters, the deep learning is going to learn the ways to extract features from the image that are deemed necessary or useful to solve a certain task. And the other difference is instead of running a single level of filters, you're going to continuously do this operation so that we can extract simple features and make more abstract, more complex features by increasing the levels or the depth, in other words, in the deep learning network. But essentially, this is how deep learning extends simple linear filtering on images, which has been done for many years. And because deep learning is so good as it, it doesn't come at a surprise that deep learning is also overtaking some typical operations on images. Here you see, for example, an operation that is often wanted is super resolution. So you try to increase the resolution artificially of an image from a lower resolution image. This is, of course, very interesting if you're dealing with a low resolution camera, you can still artificially increase the resolution and hence the detail that the image is displayed. And this network is basically trained on a database of images of both low and high resolution. And it really learned to transfer, to upgrade these low resolution images into a higher resolution equivalent. And once learned, once the filters have been learned, you can essentially deploy them on new images and increase the resolution. And this is a Google project, by the way, that is being shown here. That brings me to the first, so operations are one, that brings me to the first more semantic image operation, which is known as segmentation. So image segmentation, if you want to have a definition, is essentially the process of partitioning an image into different meaningful segments. You're trying to delineate that the parts that are of interest for you. Here, for example, is an anatomical segmentation of the brain into gray matter, white matter, and CSF, so the filling, the fluid filling between the matters. Or here, a lesion segmentation of a tumor, that's your segmentation outcome. Or WMH stands for white matter hyperintensities, which is often, which is brain damage inflicted by multiple sclerosis. And by segmenting it and hence measuring the effect of these lesions, you can also make some clinical interpretations of the condition of the patient that you can, for example, measure if the tumor has been growing in the last few weeks, or if you're giving a treatment to a person, you can measure if the tumor hopefully has shrunked in size, so that you can see if the treatment is actually having an effect. And so segmentation is one of the core analysis that is being done in medical image analysis for patient monitoring. Segmentation can be done in different ways, and the first way you can do so is as an object delineation task. And basically, it's trying to find the contour or the surface of the context of the object of interest. It's no more than that. There are quite some algorithms that are basically trying to model this, successful ones as well. But essentially, you're trying to delineate this. And in the past, this was done also manually by the radiographer on the screen, essentially. Segmentation can also be done by partitioning, where you take the image as a whole, and you're trying to define neighborhoods by looking at, for example, similar intensity values. This is a very naive segmentation, as you can see here. But that's the idea. I'm just going to try to group pixels together into coherent regions. And the most promising way of segmentation is segmentation by classification. Here you're going to try to classify a pixel. So an image is a lot of pixels, a lot of voxels. And for each of them, you're going to try to burn with machine learning or any other kind of tool, a classifier that's going to predict the class to which the pixel belongs to. And this is essentially where deep learning has taken over. Deep learning is now doing basically a classification based segmentation. So it takes in the image, like you can see here, even for autonomous driving, the same happens. If the car is looking in front of the street, there's a deep learning network that is trying to classify each pixel into a category. What is the street? What is the moving people? What are the trees? And so on. And in the medical imaging analysis, the exact same thing happens. So the image goes in. And for each pixel, there is a label being put out as a probability of it belonging to, for example, a tumor. And I can see here the unit is one of such AI systems against an expert delineation. It's really getting close to achieving the same results. Another kind of operation on images aside from segmentation is what is known as registrations, the process of spatially aligning multiple images into a single coordinate system. And typically in medical imaging, this comes from different imaging modalities. So for example, a person had undergone a CT scan and an MRI, you can then try to align both to superimpose and fuse the information of both. Or for example, a person has undergone surgery and you basically want to see changes based on aligning two images. I give you a geographical example of image alignment, but here you can see a surface-based image alignment where essentially the alignment consisted of a facial template, a wide mask, that is being deformed and shaped into the face of a person. So basically it's like indicating thousands of points on an individual face. So you can do that for large databases and that's interesting for reasons I will show you later on. But these are other examples of why registration is of interest. So if you have two CT scans, the first one is the original, the second one is an angiograph, so CT with a contrast agent added. If you don't do a non-rigid registration of a normal CT scan and the one with a contrast agent added, you can really segment out everything except for the blood vessels. So that's the kind of application of image registration where you can then emphasize the differences between two images, one with and one without a contrast agent and then you really segment out all the blood vessels and you can really focus on that. Or intra-patient MRI CT scans. So this is the same person scan with a CT and MRI and as you can clearly see here a CT scan harvests or shows different kind of information and actually has a lot of information on the heart tissue. The MRI scan on the other hand has a lot of information on the soft tissue. So if you combine and merge them you have a good vision on the heart tissue as well as the soft tissues in the human head for this example. So that's also done through image registration. And in fact you can also gain segmentation by using a registration for example if you have an atlas or if you have a brain template in which you nicely indicated certain brain regions according to functionality or whatever. You can register the atlas to an unseen image and as such transform or transfer the labeled segments as such. So this is basically another way of getting to a segmentation but now using image registration. And today again deep learning has kind of overtaken the problem of image registration. I will spare you the details but there are some really nice auto encoding systems that are able to given two images to map one image onto the other in a single step. Like most registration algorithms so far up until the point of deep learning were kind of iterative in nature. You had to find a solution and then improve the solution in the next iteration and keep on iterating until the solution didn't improve anymore. One of the benefits of deep learning equivalence to registration is a single step solution so the network gives you a mapped version of one image to the other in an instant. And the nice thing about registration is that you can also then work towards image representation or embedding. An image representation is basically or often a transformation into a low dimensional representation that still retains meaningful properties of the original data. As I mentioned in the beginning of the talk an image is really big in data so there is always an interest to trying to downscale the amount of data you need to retain. For example if I look at the full image here of healthy and diseased people I would like to be able to map them into a same lower dimensional representation and hopefully at those mappings will then also tell me how the diseased images are separable from healthy images. And so I can then use this representation as a diagnostic tool. I now have a new image coming in and I'm mapped that image or I embed the image into the same representation space and depending whether it's closely to the diseased or the healthy individuals I can make a diagnostic outcome and say okay this this configuration is definitely to the diseased or the healthy situation. So a lower dimensional representation is always very interesting for image data and in many for many years the following concept has been quite strong in doing so it's known as a statistical shape model. And I'm going to illustrate this based on facial images also because I've been working a lot with them but essentially in such a representation a single point represents a single facial image. So it's a high dimensional space but of course I'm just illustrating it as a two-dimensional graph here because of well illustration constraints but each point is a full face and if you represent your data as such it's really easy to construct for example the average face with basically the gravity point of your point cloud. Or you can also look for modes of variation in this case it's showing me differences of faces due to age on the one side if I would move along this direction I have very young small faces on this side of the direction I have very older more matured facial shapes so this is the mode of variation that you can then extract from your images. You can extract more like a second mode of variation and for faces this is really the difference between elongated small well tall and narrow faces and small and broad faces on the other hand. And then here you can still see some others like this is a third mode of variation and all these modes of variations combined are known as an active shape model so you can you could really model the variation that is present in images and you can use that model to determine things like for example a boundary within which you say okay if I'm within the boundary I'm definitely dealing with a plausible face outside the boundary I'm going away from the facial configurations and I'm creating anything but a face so it gives you a probability model of what facial probability is given a face. And today as well active shape models have now been replaced by deep learning alternatives and the key network to use here is an auto encoder so here typically you have an encoding stage of your shape and then a decoding shape and at the middle you have your lower dimensional space that is then replacing the active shape model and here essentially what I'm trying to what I show here is I trained an auto encoder on 16,000 faces and then I did some kind of TSNE visualization of the faces and as you can see the auto encoder is able to encode faces that look alike closer to each other so it's clear that the auto encoder is learning what facial variability is all about. So that's the next step of representation learning but now using deep learning. If you do some representation learning it's also very easy to start to do some recognition like image recognition is basically the ability to identify objects places or people writing in actions in images and in medical imaging recognition is often related to finding imaging biomarkers for diagnostics and prognosis for example if you're doing some tumor delineation but you have different tumors in a dataset where patients have been followed up longitudinally and you have some outcome of the prognosis whether the survival yes or no you can then train machine learning networks that given the image trying to predict the prognosis so what is the chance of survival of the patient given the current tumor segmentation here or if I look at faces again if I give at a facial gestalt can I can I diagnose a certain syndrome yes or no for example this is the average face of a control group and this is the average face of the canto syndrome the idea is to look at specific features in the face biomarkers in the face of the image and I can then end concluding a diagnostic outcome yes or no or a prognostic outcome. Essentially recognition on image learning yeah again here deep learning is is is excelling out of there as I mentioned in the beginning of the talk image net is all about recognition from images and like objects but also officially here now I'm showing you an app that is actually in place so it's commercially available for clinical genetics geneticists to just take a picture of the of a patient and to have some kind of facial interpretation of the potential underlying syndrome that is presented and it works really well it works really accurately so recognition again from images is something that is commonly done and very interesting for machine learning to be deployed an interesting network that I always like to use when I try to recognize aspects in this case syndrome classification is a triplet loss network and the reason I'm saying this is because in in medical conditions in contrast to image net which is on mainstream image data which can be gathered by by the millions in medical settings you often don't have very large data sets so you really need to look into mechanisms and learning paradigms that allow you to optimize the use of the data and the triplet loss is one of those instead of trying to classify an image given the image only the triplet loss is going to use triplets of your data so triplets of images together and basically what typically does it gives you an anchor this is the this is the standard image let's say a positive example so this is an image of a person let's say in this case of the same syndromic group and a negative example so it's another image of someone from another group and what the triplet loss is trying to do it's trying to learn a new embedding of the data such that the distance between the anchor and the positive example goes down and the distance with the anchor and the negative example goes up so it's really trying to cluster your data based on these labels but the advantage basically is that you can use triplets of data and and each anchor can be combined with multiple negative or positive examples so you're reusing your data in a very efficient way and such a network is really interesting for medical applications and medical data sets as such and if you do so for facial shape for example I learned a triplet loss to separate males from females here and it's also nice to see that the embedding indeed is putting really extreme males on the one hand and really extreme females on the other hand and there where the two cohorts are touching upon each other indeed the distinction between the male or female face is less clear so here is basically you can have some confusion whether it's a male or female and on the right I've also learned something but not more in terms of age so the yellow are the older faces and the blue ones that are shown here are younger faces so again you can learn to recognize age by just looking at an image or learn to recognize the sex by just looking at an image that brings me to the last typical operation which is generation and generation essentially is a task of generating new images from an existing data set and this is typically done for simulation purposes so if I have a known condition I would like to generate something that I want that I think is going to happen so if I if you look at the animation here oh that was one too fast Peter if I look at the animation so I'm I try to plan an orthognatic surgery by moving the chin forward so I'm asking myself what would the facial envelope look like if I do this and that is basically an application of image generation you're trying to generate something that hasn't been given yet so synthetically generate images well again if you have your models it's quite easy to generate images by simply sampling within the boundary so here you can you have your active shape model if you simply sample within the boundary you can generate new faces that has not have not seen been seen before and I think most of you are familiar with this at least I've seen it in the media a few times now again deep learning is really good at generating images and the well the best network to to start engaging with this kind of work is the generative adversarial network training so you have a generator of images and at the same time you have a discriminating network that tries to say whether the image is correct fake yes or no and then the generator is learning to improve himself to fool the discriminator it's like a police versus thief kind of game where the one tries to outsmart the other and that comes down to the adversarial training but what I what I what I thought was interesting to show is for example at the latest results obtained by Nvidia on the left here they've basically trained a network that is generating faces but all of these faces are synthetic so none of them do actually exist and in my opinion the quality of these images is very realistic but also it has also medical applications and medical applications for example is shown here let's say if you train a network that is able to generate a synthetic CT scan from an original MRI scan so doing intermodality changes so from MRI to CT you could then generate CT scans from all the MRI scans you have and it sounds silly why would we do this well you should know that a CT scan does not come without a cost CT scan is known as an invasive imaging technology with a lot of radiation so if I can generate a CT scan or something that is equivalent to a CT scan by using a non-invasive imaging technique such as MRI of course it's a less burden for the patient to do so and so it has a lot of advantages in trying to solve this question the earlier publications on this topic are promising but I think there's still a long way to go that brings me to the last part but I'm just going to check I think in terms of timing might be more interesting we are a little bit flexible I mean at most you could do another five minutes to leave five minutes for questions but up to I would say up to five minutes of talk yeah okay then then I think it's just interesting to show a few applications and again as I mentioned these are basically where I've been working on so there are many many more of course but I just wanted to show you something I think the third one is extremely exciting for us if I may choose here okay I will I will try to jump but the forensic one is definitely like a virtual autopsy where all of the imaging modalities are being combined and used to really determine the cause of death and to solve criminal cases we've also been working extensively on gunshot trajectory look here also if someone's being shot in the head the purpose of this work is to really highlight the damage done and it sounds a bit cruel but if multiple people shoot at you the outcome of this kind of examination is to determine which bullet caused you to die we said the bullet going through the heart or the bullet going to the liver and that's that's key information to convict one person yes or no another kind of forensic imaging is bloodstain pattern analysis here we basically look at the shape of blood stains and we do a segmentation here and then registration into a statistical model that then allows us to deduct the point of origin where the person basically was hit or cranial facial reconstruction is also a combination of registration and statistical models to learn the relationship between the skull and the face and hence if I give you the skull the question is what does the face look like that's another application in forensics clinically people have a lot of questions like the surgeons for example have what are the changes that I induce on faces before and after what's the asymmetry it's clear an asymmetrical case if I improved it by how much and what's the lack of harmony and again here by registration by taking multiple images and superimposing or registering them to each other you can make these measurements I know I'm going fast but I'll try to make it to the imaging genetics asymmetry is the same and harmony is basically trying to assess the patient in the context of the larger population and assess as such create something like a normal equivalent so this is like the patient without the asymmetrical component in the face and that can be used to plan virtually a complete treatment such that the time in theater itself can be reduced from one day to an hour so that's where imaging comes into play this is diagnostics growth curves syndrome classification triple loss I'm just going to skip okay to the imaging genetics imaging genetics basically is trying to also understand this complex puzzle of going to the genome and to your phenome so hey how does it come that it that certain variants lead to certain facial shapes in this case the difference basically with bioinformatics is that we really focus on the phenotyping aspect so I try to improve the phenotyping through the imaging and I'm basically using tools from bioinformatics and others to hence deduct some information and that generates a lot of kind of investigations that we can start doing like developmental aspects like for example what's the effect of alcohol during pregnancy on facial shape how does a face grow from one to two 25 years of old what is facial heritability so here also just by collecting imaging and by register and images images and by bringing them to lower dimensional representations you can really yield the power to find correlations between in this case heritability genetics and facial shape but also evolutionary questions can be answered like I finished did some investigation of climate adaptation to nose shape and essentially their colder climate in Europe is responsible for smaller noses here but also population genetics can be correlated to facial shape and hence we can generate my brothers from around the world I always say so same person but a different population background and investigate other questions like is there a relationship between heterozygosity and facial masculinity I can answer that the relationship was absent and last but not least I guess yeah we have the whole genotype to phenotype correlation analysis it can also be done we're basically here like the GWAS paradigm and here we basically we generated some kind of imaging technique that was able to divide and conquer the wealth of data in imaging by looking at segments in the face and breaking down the image into correlated segments so it's a data-driven segmentation it's not a manual segmentation that really allowed to optimize the power of a GWAS onto facial shape and hence identify many more low side and will previously identified and then I kind of like to play with the idea of you have genetic correlated aspects then yeah you might try to model even an image or image generation from DNA which is then referred to as DNA phenotyping or the opposite an opposite paradigm where I basically don't try to generate a face from DNA but if I give you a face I try to classify to see if the face fits to a given DNA so it's a it's a rephrasing of the problem but instead of image generation you're doing image recognition and again here in the last part of recognition deep learning and learning machine learning in general it's like a classification really excels at solving this problem so when I contrasted the results of DNA phenotyping to facial recognition from DNA the latter proved to be much stronger in establishing a link between a face and DNA and the former basically I think that's that's the main overall message that I still wanted to get contained here today so I'm happy to hear any questions if there are any if not then thanks again for having me but thank you Peter for this great overview that was very educating and very exciting thank you my pleasure are there are there questions from the audience people that are sending applause to you for this for this talk that was great that's nice thanks so I have one question here in the chat by Leslie for generating CT scans from MRIs do you think the biggest driver for improvement will be due to more or better data or better mod or are better models needed yeah there's certainly the two-fold yes there is more data needed there are only very few databases that have both MRI and CT scans of the same individual so it's really hard to get like paired data so that's definitely an avenue to increase on but it's not easy because CT scanning is not done without any ethical concerns so I strongly believe that the unsupervised paradigms where we see more and more of today like there is this thing of image to image conversion where you basically can simply collect large databases of CT scans and large databases of MRI scans and really try to generate one from the other and this is all GAN based essentially and then you're using the the smaller data sets where you do have paired data to optimize the well the conditional learning input because it's a conditioned network where the conditioning is an MRI image and you want to generate a CT scan that fits to that MRI so it cannot be completely unsupervised but in short it's an increase of data and well we need we need to investigate a new new learning paradigms essentially it's a creation of both yeah then there's another question by Giovanni on ESR in the network talk to a student in the network Giovanni please hello thank you for your talk I have a quick question regarding these very last part that you described I wanted to ask you to expand a little bit on how you use genomics essentially for predicting like generating phenotypes and so on like do you focus on specific loci I suppose you don't use complete genomic sequences because that would be a little bit too much for example in the previous slides before I think this one or the one before with the face but yeah I wanted to to ask in which format you use specifically DNA that's a good question so essentially the information that is of value to creating faces from DNA but before I answer I also have to admit and be upfront that the prediction of a face from DNA is far from accurate and possible and I say this because I know there are some commercial companies that pretend that it is possible there but they're never really validated their work but the information that is of interest is are a few aspects first of all is the sex which is basically determined by the X and Y chromosome and secondly there is the aspect of population or ancestry and so far I've been working mainly with simple principle component analysis of SNP data to generate these as you would correct for the confounding in a GWAS today we are investigating variational autoencoders to do the same thing and to see if they don't code for ancestral variation better than that but that's like the genomic view and then aside from that we're still limited still at individual SNP data so it's definitely not full genome data it's more like the low side that we've identified to be associated to facial shape we don't have something like a polygenetic risk or yet although and I can maybe I can send it around later we do have a paper currently already out as a pre-pint but it's on the review on the review still that is well we call it a polygenic shape score and for the first time try to really condense a GWAS into well some kind of facial phenotype prediction but there I can also tell you that that is again limited to kind of cross-population shape differences we found like it was interesting we did a GWAS on an Asian population and an Asian population only and if we combine certain genetic loci into these shapes scores we were actually able to generate a European face from a from an Asian cohort that's kind of the the thing we saw but aside from that it's not going to expose an individual within Europe or or anything like that I see thank you that's really fascinating thank you Giovanni we have time for one quick question by Lukas so Carson and thank you Peter for the talk I was wondering regarding the imaging modalities that you presented there are some of them that have to my uh small limited understanding a lot of heavy preprocessing attached to them such as functional MRI for example are there any solutions proposed to automate uh this heavy preprocessing using deep learning for example yeah definitely yeah but you're right uh like I just gave a brief overview but a lot of these imaging modalities really come with a lot of noise um and hence the the post-processing is important I can tell you that the manufacturers of these devices already spend a lot of energy in in updating the post-processing of the imaging that their their clientele in the end is is a medical doctor a radiologist who has no knowledge of post-processing so they try to basically relieve the burden and and get the best image out but I always find it a double coin like if you do a lot of post-processing you're not really sure if you're actually not eliminated the information that you were looking for in the first place so post-processing also comes at at a cost I would say but here you're right also a lot of these manufacturers are investing in deep learning to solve many of these tasks and if I can say if I look at my experience there is a lot of traditional algorithms that are able to solve a problem but deep learning is so is to do is able to do the same but it's it's better at dealing with noisy and more differentiating situations so it basically expands the applicability of some previously uh already well-working algorithms thank you very much Lukas and Peter and thank you Peter again for this talk and also for taking the time now to meet our doctoral students in the breakout room we are also grateful for that the general audience the PIs and myself we say goodbye for now and thank you we are grateful that you that you joined the summer school and we invite you to to meet the doctoral students in the breakout room and the general program here continues in half an hour with Jennifer List Garden's talk see you all then at 4 30 Central European time thank you Peter