 pleasure to welcome Francesca Grisoni and she will give our first talk this morning, please. Thank you very much for the introduction and welcome everybody. I'm very happy to be kicking out this day. And so today we'll bring you on a small detour about drug design or actually in the context of drug discovery and how we use a different flavor of deep learning to do so in the chemical language modeling realm. First of all, let's see if it works. Okay, so it's not working. Okay, and just change slides with this doesn't even change slides. Okay, so let's try again. I was saying, we will talk about drug discovery and how deep learning can be used there, especially in the context of designing novel molecules. But first of all why is a machine learning receiving so much attention in the context of drug discovery. Well one of the reasons why is that maybe you already know but the efficiency humans and the efficiency of developing new drugs is slowly decreasing over time. And at each step of the drug development pipeline, the costs and times, time scales involved increase remarkably. So basically the idea here is that we want to use to find a way to fail as early as possible and as cheap as possible so of course we don't want to fail. But surely machine learning is a good tool to help us identify promising molecules. Actually, drug discovery is very challenging. And one of the reasons why it is really challenging is because of the cardinality of the chemical universe. And the chemical universe is that virtual space that is estimated to contain synthesizable molecules up to a certain size. And it is estimated to contain in between 10 to the power of 23 to 10 to the power of 100 molecules. To mitigate this past chemical universe actually we only have a very limited number of samples. In fact, humans have discovered only about 10 to the power of four molecules. And to give you an idea of how different, or how big the chemical universe is, it actually contains way more molecules than there are stars in the Milky Way or through our cells in the human body. And so how can we actually approach drug discovery using machine learning. Ideally, we can we can think of it as a problem of mapping right we have a space of molecular structures and we have a space of molecular properties and we want to find a reasonable mapping between these two. And my group mostly works in the early stages of drug discovery so when we talk about molecular properties we're talking about finding legans that interact or modulate a certain target that is pharmacologically relevant like a protein. We can approach this problem in a, let's say, forward way. So, in what we call virtual screening, and the idea here is quite simple so we start from a library of molecules, we could purchase or synthesize and we use machine learning models to predict their properties, and then simply we rank those molecules or prioritize them and purchase and synthesize the most promising candidates and then we test them. And actually, ideally, what we would want to do is to go the other direction so starting from a set of properties or one property we like to go back and map it back into a structure or a set of structures. And of course this task is much more challenging, and one of the reasons why it is is that there is not a single one to one mapping so if we have a property or a range of this property we like, it's very difficult to know which structures might satisfy this property. For example, classical machine learning models, which are based on human engineered descriptors, are not very useful to do this inverse use are so inverse quantitative structure activity relationship, because most of the descriptors are not immediately relatable or connect cannot be immediately connected to a molecular structure. Nevertheless, even before deep learning became a thing in dry discovery, people were still finding ways to use computers to design new molecules from scratch. And there are a few ways in which you can do it. For example, one to try and enumerate libraries, and the idea of library enumeration is quite easy so you set up some rules, which are used to generate as many structures as possible. One of the most notable efforts is this GDP the references below here, and the idea is that basically the start by enumerating all of the possible graphs up to a certain size, and then progressively start including chemical information and kicking out graphs that will not make sense, chemically. And so they end up with quite a remarkable number of molecules and molecular graphs. But actually what strikes strikes me every time I, this is a very remarkable effort, don't get me wrong, but even with this high like intensive library enumeration they don't even manage to cover all the libraries of known molecules that have been tested. So it's really really rule intensive and still leads to a quite limited chemical space exploration of the chemical space. There are other approaches of course and one way of tackling it could be to go for reaction for even design. And the ideas is quite similar, but instead of enumerating graphs you produce a list of billion blocks you might like, because you have them in the lab or you can purchase them, and a set of reactions, and you try and combine them to, let's say make molecules evolve towards promising regions of the chemical space so for example with the fitness function. And then another very, let's say popular way of dealing with you know design especially machine in medicinal chemistry is to use information from the structure. So usually medicinal chemists start from structural data on the protein of interest, and then progressively combine fragments to try and build molecules that would fit the pocket. So all of these approaches are very interesting, they're all different. But one thing we can notice is that they're very rule intensive. So depending on how many rules we include in our system, then the, let's say, the efforts the computational efforts or human labor required will exponentially increase so it's it's quite limited. All of them are quite limited in the ability of exploring the chemical space efficiently. So of course here I don't need to tell you why deep learning is useful but I think you know what design is one of the fields in which deep learning has played a biggest role here. And I like to always think about some of the breakthroughs of deep learning as something that is very similar to what we want to achieve here. We found that deep learning certain deep learning algorithm can master very complex games, and also are improving a lot in the ability to drive around autonomous cars. And it's basically what we want to do right we want to develop. Ultimately, models that will tell us which moves we have to make for example when we want to optimize molecule, or where do we have to go in this super large chemical space. And there are many ways of leveraging deep learning for the novel design, but today we'll be focusing on a very special type of deep learning models, which fall under the umbrella of chemical language modeling. And they are inspired by language modeling, and the idea of language modeling is quite simple so you start with a corporate of texts usually natural language, and you try and develop models that will capture for example the probability of a word within a sentence or of a sentence within a language and so on and so forth. And basically what happened a few years ago is that people started borrowing these algorithms and apply them to chemistry. But why really can we apply these models to chemistry well we can think of molecules as elements of a chemical language. So if you look at the molecule and you think about starting to randomly move around these items and connect them to each other, you will probably not obtain a chemically valid molecule or something that makes sense. This is because these molecules have or this chemical language has a grammar. So we need to have certain will satisfy to design a molecule that makes sense. But then of course this language also has a semantics. So we want to have the right items in the right place, connected in the right way to have certain higher values like physical chemical properties or biological activity. And basically, this is more or less the philosophy underneath this chemical language models. Again, there are many flavors to it. But basically what we have been doing for the last five years or so, maybe a bit less but yeah four years or so is to basically use textual representations of molecules and use a language modeling approaches to model this information. And basically, what we do is that we start with my strings and for those of you who don't know much my strings to just capture to the information basically. We attached to characters or token that indicates the beginning and the ending of the sequence, and then we feed them into a neural network. There are many networks that we could use but in particular we use these recurring neural networks. The idea is that we use these networks to predict at every at each step of the center of the sequence, which character should follow. In practice how we do it is that we want hot and cold our sequences. And then in particular we use a particular type of recurring neural network. In the API, if you're not familiar with the current neural networks there actually networks developed to deal with sequence data with sequential data. And this long short term memory cells just were developed to improve upon some limitations of vanilla RNNs like exploding or vanishing gradients. And they also have. Yeah they're able to keep track of let's say, long range dependencies within our strings but this is just a minor detail. What we actually want to do is that as a set to basically model the probability of occurrence of each character or the probability to follow a certain portion of the smile strings for each character. Why is this particularly powerful. Well, the idea is that if we train our model sufficiently so for a sufficient number of epochs and a sufficiently high number of data, then we can use them as generative tools. So we can start again by feeding the model with the go character, let the model generate the next character, which will become the input at the next step, and we can actually keep going until the end of the string character will be reached. There are many ways of doing this, but the power is that actually without needing to specify any human engineer rules like in the other approaches we saw, we are able to generate strings just by learning directly from our data. And one of the approaches we have been using to sample from this model model models is what we call temperature sampling and basically temperature sampling is nothing but a weighted random sampling, controlled by a temperature factor. So the higher the temperature, the higher the randomness in our sampling and the lower the lower the randomness. So and again, somewhere between the end of 2017 and the beginning of 2018, there were a few studies that showed that these methods seem to be promising to generate smile strings that made chemical sense and had desirable properties. So actually what we wanted to do was to really try and push this models and see if we could bring them to the lab. So what we actually what we wanted to do was to design novel molecules that were bioactive against certain pharmacologically relevant targets. But of course you are familiar if you're familiar with neural networks and or drug discovery, you will know that one limitation of neural networks is the amount of data and of course in drug discovery you rarely have enough data to train a model from scratch. And it was mentioned already yesterday. And but there are a few tricks to leverage low data regimes, and one of these tricks is the so called transfer learning, and the idea of transfer learning transfer learning is simple. If you do not have enough data to train a model from scratch, then you can use a bigger data set on a task that is somehow related and use that to pre train your model. So what we did was to use Campbell, which is a repository of molecules annotated for their bio activity on several targets. And there we just picked 300,000 molecules that were annotated as bioactive. We didn't really care about the target we just care that they were bioactive. And what we could say could see is that already by pre training the model in this way. We were able to generate more than 90% of smiles or strings that were actually chemically valid. So what this means is that the model by just looking at the bunch of smile strings and trying to predict the next token was able to learn the grammar of our language. Then of course as I said yeah grammar is important but what we really care about here is the semantics of the language. We want to design molecules with certain properties. And in our case what we wanted to do was to design nuclear receptor modulators dual nuclear receptor modulators in particular. And so we fine tuned our model so after pre training we kept training with 25 with the spice of 25 nuclear receptor modulators. So after this we let the model generate the thousands by strings and then we rank them with the usual in house protocols and we will come to that later. And then we we provided the sorted list to our medicinal chemist who was working in the group back then. And then we just asked him to start from from the top and synthesize whatever he could. And these are the five molecules that were picked and successfully synthesize in the lab. And what we were really happy to see is that all these five molecules for actually possess the intended by activity and actually one of them in the same range as the fine tuning molecules. So for us this already told us that by just showing a bunch of data and the right data to the model. You could design molecules that possess both syntactic and semantical properties that were desirable. So since this initial study that was in back in 2018 we really started to work on this model to try and improve their ability to work in small data regimes and to try and automate the whole process. Of course we went back to the concept of smiles and I just told you before that they're just strings that represented to the structure. But basically the idea of a smile string is that you can start from any item you have that is not an hydrogen and proceed in any direction and then basically you will linearize the path and make it into a string. So what this means is that actually from any molecule, we can obtain many many smile strings. So this is a suitable strategy for example to perform data augmentation. And when we look at the performance of our models when we start augmenting smiles. And basically, we will see that we actually can also increase the validity uniqueness and novelty of the molecules which generate. And then of course there are many other things we can play around with so for example did the randomness of the process. And as I told you that the randomness is controlled by temperature and even there by playing around, you can control. Basically how how crazy the model will be in generating smiles or how close the model will be in, let's say to define tuning molecules. So we wondered how well can we exploit this model models in the in the in a low data regime. So we we set up two experiments where we fine tuned our model with natural products. And for those of you who are not familiar with, with drug discovery, the idea of natural products is that they are quite different from synthetic molecules that were synthesized by humans. They tend to be considered as considered as a source of inspiration. So basically here what we did was to pick two fine tuning sets with only five molecules, and then check how well the designs. Let's say the model would move between the chemical space of the pre training set and the chemical space of the fine tuning set. And we could do it on many properties but one of the properties that is particularly interesting for drug discovery is this fraction of sp3 hybridized carbon atoms because this is the say the higher this fraction the higher the chances of clinical success and natural products tend to be rich of this fsp3. So basically what you see here is the fraction for the pre training set in blue and the two finding fine tuning sets in white. And what we could observe is that during fine tuning, even though we had a low number of fine tuning molecules, the properties of the designs would move somehow continuously in between the properties of the pre training set and the properties of the fine tuning set. And this is quite relevant because for example fire natural products have some properties that are desirable, but they're a nightmare to synthesize while synthetically accessible molecules like Campbell might not have necessarily optimal So this means that we can actually decide to stop at any point during this fine during the stress for learning procedure, and so we can mix the properties of the pre training set and the fine tuning set. So this concept in mind we decided to bring this models once again to the lab. And in particular we wanted to use a few natural products for to have some natural product inspired design which is often very cool thing to do. We also wanted to improve upon our ability to choose the molecules that are generated by the model. So as I told you, with this temperature sampling you can generate actually as many molecules as you want, but then how do you prioritize those. So then we started to play around with a different way of sampling from these models, and one of these ways is the beam search algorithm, maybe you are familiar with it. So this is quite simple. Instead of just using this weighted random probability at each step of your sequence, you just extend, you choose a width of K, and you just extend the string with the K most promising tokens based on the probability. So basically you start branching out and only expand the most most. Yeah, using the most likely tokens. So this would work, but actually didn't. So what we notice is that if we pre train and then fine tune as I showed you, and then we start sampling actually we might end up with molecules that are either not valid or predicted as a nightmare to synthesize. And this is probably because you miss this randomness so it's it's it's quite difficult to jump between one space of synthesizable molecule and the natural product space. It's not a trick. And so basically we we took the model by hand, and and had an intermediate fine tuning step. So we first started with ligands on the target of interest as the first fine tuning set, and then we fine tuned again with natural products. So this was the transition help the model actually learn features that were relevant for us. And so we ended up synthesizing the top three molecules and all of them actually were active on the intended targets and as you can see, they look pretty much like synthesize like synthetic molecules but they also have some features of natural products like an increase f sp3 fraction so sorry fraction of sp3 hybridized carbon atoms. Then another thing we were really interested about is how can we really automate the whole process and synthesize way more molecules than just three or five as in the previous study. So in my previous lab when I was at the age, we actually had a microfluidic assisted synthesis platform that could perform 17 one step reactions. And so we really started thinking how can we combine the automation on the design side of things with the automation on the synthesis side of things. And the idea would be that once we are able to do this, then we might probably automate the whole design make tests analyze pipeline and iterate quickly over our design design protocol. But actually integrating these two platforms was not as easy as we thought. And of course one thing you could do is to just develop a filter and say okay I just let my model generate molecules and then I filter out everything that matches the types of one step reactions that can be performed in my micro fluidic assisted platform. But again, we shouldn't forget that these models learn from the data. So what we observed is that if we pre train with the normal Campbell and compare it with a pre training with molecules that actually passed a synthesizability filter that actually we obtain way more molecules that are predicted to be synthesizable in that system platform. This means that actually, depending on what you put in your training data the model will also learn those rules somehow implicitly implicitly. So we proceeded with this idea so fine tuning with molecules that actually passed the filter and use it to generate to use the model to find and fine tuned it with modulators of LXR another nuclear receptors. And here you see an overview of the molecules that were generated with that tool. So of course, most majority majority of the molecules were actually novel, this is what we expect. But actually what we were quite happy to see is that almost around 20% of the molecules we generated were actually patented modulators of that target that we never used for training so for us this already gave us was a good insight that the model was capturing relevant information. So we, among the designs we we filter them based on the predicted synthesis route, and we were left with 41 compounds, and then around 60% of those was were successfully synthesized in our platform so again, the model showed some aspects of the semantics. Then we also tested. So without purification we tested the crude reaction mixtures on the target of interest, and around 70% almost of those showed by activity. So again, we were able to modulate multiple properties at the same time synthesize ability and by activity, just be carefully thinking about what we put in our pre training and finding the data. So with 17 compounds we just picked, I think the most promising 14 synthesize them again purified, and 12 of them showed again the activity we were expecting, also in a quite good range for, let's say the very early phases of drug discovery. There's still a lot to do here. And one of the problems we're always left with is, how do we actually make sense of the amount of chemical entities that can be generated by these models. So we actually started thinking really about what these models produce and actually, of course we like to think about it as a string, but actually underneath these strings are the probabilities that are produced by the model. So what can we do with these probabilities well we can again borrow from the natural language processing domain, and for example use something that will tell us whether the model or how much the model is sure about the string it has produced. So in this measure, one of these measures for example is called perplexity, the lower the perplexity, the more certain the model is of the string it produce so in terms of probabilities of the characters contained there. And actually, we were quite surprised to see to see this. If you remember, at a certain point we replace the beam search, we replace a temperature sampling with beam search thinking it would be a better alternative. But actually if you look at it, the molecules generated by by temperature sampling have very much, much lower perplexity values on average than the beam search. What it means is that actually it's probably smarter to just generate with this weighted random probability, and then just rank them using the perplexity and pick those ones, then trying to build this tree and explore only the most promising tokens. And the reasons why it is, it is, is that, of course, the beam searches agree the algorithm, but it doesn't explore all of the possible combinations. So we're losing something interesting there. So again, we're now using this perplexity measure to try and automate the process of generation and ranking of the designs. I will keep you posted on whether this works in the lab to. Yeah, and since then actually there are many words we have been following and doing to try and improve the learning of the chemical language of these models which are actually developed for natural language processing mostly. So, I also forgot this, I did it the last minute. Another thing we can do actually with these probabilities and this is just an idea is that actually they can help us look into the black box if we can call it this way of these types of deep learning models. So, basically, whenever you generate a molecule you actually have probabilities associated with it. So this could be, I hope so, very good to communicate with medicinal chemists, for example, what are the regions that are highly likely based on the model. So one of the regions where maybe the medicinal chemists could change something here and there to improve the synthesize ability or other properties and, of course, these probabilities could also be connected with known binding modes with the target of interest. But as I was saying, these models are actually borrowed from the natural language processing domain so there is a lot to do for us at working on more chemistry related domain. So for example, with my strings we generate them from left to right because we borrowed this, these strategies, and of course, not everywhere we generate language from left to right but in the Western world we do. But actually, smile strings can be obtained from any items in any position and can go in any direction so does it really make sense to generate them from left to right. So we're digging into literature to see if there would be methods that were proposed to generate language in a bidirectional way, and actually there was so there are approaches that for example generate. So it's two RNNs that generate in different directions so extend the smile strings in both directions, or methods that basically just replace some last tokens here and there but taking by taking account the sequence in both directions. So basically we design another method that we're just trying to combine the advantages of each. And basically what this method does is that it treats the sequence in both direction combines this information and extends the sequence in either on either the left or the right. So we're going to go with the too many details here, but actually despite directional models seem to and especially the third one seems to be quite good in capturing certain semantical properties like the similarity to define tuning molecules a bit more than unidirectional models, and also the scaffold diversity is higher so they seem quite promising actually. What I like about this methods is that actually, we have a new way of augmenting our data. So, if you remember with smiles you can already do it because you can generate as many smiles strings as you want for a given molecule. But actually here with this bidirectional generation, you can place the starting token either in the perfect sentence the center of a string, or in a random place in the string. And actually, since you can do it randomly, then you can also place it randomly as many times as you want in yours my string and in this way you will have augmented your data set. And actually this procedure seems quite beneficial, although it doesn't lead to a big difference compared to the standard way of augmenting smile strings. However, I think this might be promising for the future for example when we deal with proteins proteins are actually non augmentable because it's just a sequence and that's all you get. So with this strategy, we think we might actually be able to augment small data sets that contain proteins or peptides. And yeah, there's still a lot going on we can keep borrowing approaches from from the national language processing domain and try to adapt them to chemistry. So here for example another thing we did was to try and play around with this electric retraining. And the idea here is that instead of pre training the model to predict the next token, you actually train it as a discriminator so you start correcting randomly some some tokens in your stream, and then you let the model identify which ones are not the original characters and again we brought this to the lab and could design kindness inhibitor. So a lot you can do really with those types of models. And for example we're also working with peptides and it's basically the same thing you just replace the alphabet, you're using, and then you're, you're done and you can use those and even in this case we use these models to design and I can surf peptides the novel and, and it worked. So how do we now, well, I think I don't want to say we sold the novel design because of course that's not possible but I see that we are reaching the plateau of this chemical language models when you just want to address one target at a time. What I think we're really lacking is how to have a very fine brain control on the properties of the designs. So you are seeing that the general, let's say the average properties of the whole set. We tend to move in between the pre training sets and the fine tuning set, but how do we actually control the property of a single molecule. And also, how can we actually deal with multi objective the novel design in a completely unsupervised way. So often, we need to design molecules that possess multiple properties at the same time. For example, when we want to cure Alzheimer's there we talk about polypharmacology, instead of taking multiple drugs to cure multiple aspects of the disease why not developing a single molecule that takes care of it. Or sometimes we also want to optimize selectivity we want a molecule that just binds to the targets and not to anything else in the, in the, in the body. And of course, we almost never have data sets that are complete so molecules annotated for all of the properties that we need. So we're actually now working on how to combine very sparse data sets and actually steer directly the properties of the designs by manipulating the latent space without needing any external ranking function or fitness function and so on and so forth but just really working on what the model learns and the promising directions in this latent space. So we are keep going with extending these language modeling approaches to other tasks. So we have an ongoing collaboration with IBM Suric and we're trying to bring this language models to the field of designing new enzymes from scratch. And I hope, maybe the next time I will see some of you that both of these projects will have led to good results and I will be able to share them with you. I'm done. And of course there are many people I need to thank and for the initial LSTM projects I showed you actually most of them, I have to thank all the pro my colleagues in the group of Gisbert Schneider, in particular Mikael and Lucas. And then also of course our medicinal chemistry collaborators in particular Daniel Merck and his group. And of course I I think my current group and my collaborators at IBM and of course the organizers for inviting me and all of you for the attention. Feel free to ask questions. It's a lot Francesca for this great opening talk this morning. So other questions. Okay, do we have a second microphone or otherwise everyone. Great. Thank you very much. I have a question regarding the generative models and more classical user models which predict the properties of the chemicals, can you actually do design by having a generative model which is independent of the properties you want to predict and combine it with a user model that has the properties, because in this way you could do the two tasks separately and then. Yeah, ideally, you could. But the idea here is that really you, you want your, your molecules to be as close as possible to the task you want to perform. So really what you see when you analyze these designs is that basically, for example if you look at the fingerprints of the molecules surrounding the that we're generating generated you see that basically the model explores the neighborhood of your fine tuning set. So ideally if you already know, for example which molecules are the active ones because you train the accuser model on it. Then why not using them directly to focus your library, but then of course for any other generic tasks yeah you can also stop at the pre training and use the whole Campbell. To generate drug like molecules in general, so just a general library and then you can explore it with any tool you would you like, really. What are the advantages and disadvantages of each of the two approaches because obviously they would have the advantages. The advantages of using this data driven approach is that you can really you don't really need many molecules. So for example we used for to design new molecules. The advantage is that if you don't have enough data to train an accuser model, then you can use this purely, let's say end to end approach so you start from smiles and you end the smiles. Thank you very much Francesca for the great talk. I have a question about how one comes up with the vocabulary a grammar, because for the case of smiles, we do know that it contains a lot of intuition and a bit of bias. Sometimes you show some starting grammar about that. So, what is the ideal grammar. That's a tough question. Yeah, so I don't know, of course, the grammar is not is up to whoever is designing your presentation so for example, there is this effort with selfies, and that's yet another grammar. I don't know what the ideal grammar should be, but definitely one of the advantages of the string based representations is that. Yeah, it's it's quite easy to extend a string so it's it's a good way of approaching the problem, but I wouldn't be I never wrote a representation. So, yeah, but thanks. I have a question about this general model that you train on Campbell, and you mentioned the work, this graph enumeration work I think it leads to, was it like 14 million or even more graphs. But but there is kind of a lacking, there are some known drug like molecules that are not part of that graph enumeration work. Do you know like what is the capacity of your generative model when it's training Campbell like how many unique strings can you generate we have any notion of that or is that like infinite. How many I can. Yeah, if you just run and run and run how many it's probably not infinite because of course you have a grammar there. And there's just a certain number of combinations of a given maximum size. I don't have an estimate but I but you can keep going for a while so I yeah yeah definitely you would not struggle to generate millions of molecules. Yeah, first, thank you for the great talk, just a short question concerning your model choice. So you had recurrent neural networks or specifically as lsdms. Is there any reasons why you didn't use transformers or GPT like models. And so we, we didn't try to, sorry, maybe no no that's, I mean that's just a bit. Yeah, so we, we didn't try to transformers, but some other people have tried it and actually the point with this type of strings is that they're not very long. So you don't have. Yeah, a lot of long range dependencies to learn and so actually lsdms or GR use as well seem just a good compromise between the complexity, and what you actually want to achieve so. There are a few studies out there I've seen one couple of days ago that leverage transformers. But yeah, to us it didn't seem like a worthwhile efforts, but maybe maybe we're wrong. I don't know. Yeah. Thank you very much for the great talk. So one of your slides, there were two groups of molecules one level with molecules with low diversity and the other one molecules with diversity. So I was very much interested in, and how do you define this, how do you formalize this concept of diversity, and do you start from smile strings and then the final distance between smile strings. How do you do that. Yeah, so in this case that our definition of diversity was just very functional to what we wanted to do so we just, if I remember correctly we picked, where's the pointer. We picked cherry picked one of molecule one one natural product in this natural product database, and then use the extended connectivity fingerprints so basically they kept the catch the presence of shared radio substructures. And then you just, we just picked the four most similar in one case and I think we did the min max picker in the other case to pick the other four mostly similar, but you could also do it on smiles of course yeah. Thank you. Okay, other questions. I had a while I walk back I can ask a question. So in your automatic synthesis. Did you have a way to also automatically test the properties of the molecules. No this was the bottleneck and probably if I if I had stayed there we could have worked on on this side of things. No we just, yeah, we just texted the crude reaction mixtures as a, as a way of having the minimum labor needed. But then of course, yeah, but it would be cool, definitely cool. Thanks. Okay, good. So I see no further questions I don't know if there's anything on zoom, otherwise. Yeah, will it say has a zoom question. So, let's take one from zoom and then we conclude. Thank you. There was a question on zoom about what are the dependencies of number of epochs to the number of input or output layers for the best accuracy. The number of input and ample layers. I'm not sure I get the question but they don't change throughout the epochs so yeah. Yeah, I, I guess the question about was about how do you choose. How do we choose. Yeah, the epochs in relation to the number of input and later. Yeah, so somebody said it yesterday that it's more of an art than than anything else to train one of these models so we usually start simple a few layers like to to LSEM layers or so. And then we actually work more on the epochs what you see is that the more you fine tuned the more the diversity declines and then the more the molecules become similar to the fine tuning molecule so that's also a bit of an art to decide where to stop. Yeah, every study has been a bit different we have not sold it all yet. Yeah, okay but you start with the architecture and then you do exactly we start with the architecture and then monitor the property change of the designs and then decide where to stop. Okay, perfect. Thank you. Thanks. Okay, then let's thank Francesca again.