 Good afternoon and welcome to today's energy seminar. To introduce our very special speaker today, we have a very special guest, Professor Ishwe, who, as many of you know, is now the director, recently appointed director of the Sustainability Accelerator at the Door School. Previous to that, among his many other jobs, he was the director of the Precord Institute for Energy, which is our kind of guiding light and benefactor. So we literally work for him on this seminar series, but now he's back in a different capacity to introduce our speaker today. He, of course, is a world-class researcher in electrochemistry, nanoscience, and sustainability, actually doing innovations across many different domains, from energy, batteries, solar cells, water purification, air purification, a few other ones I probably forgot. Just to end the introduction of E, he's been awarded 28 prestigious awards and honors, including the Global Energy Award in 2021, and five startups have sprung out from his laboratory. So he's a very busy guy, and we're glad he is able to join us today to introduce our special speaker for today, E. Well, thank you very much, John. Finally, I know how many awards I have won. Thank you for counting that for me. I never know. You said that you probably put it here. OK, I'm glad to know about his number. And thank you. It's my actual pleasure to introduce Cynthia Zhen. Let me tell you a little bit about Cynthia. About two years ago, I met Cynthia for the first time. He is currently still a PhD student at MIT, an operational research center. And she was telling me about the exciting research she has been doing. She entered in BlackRock, SoftBank, on her way in making a lot of money. Then instead of telling her about financing, because I know nothing about, I started to tell her about climate change, sustainability. And presenting to her some of the problems we are thinking about. Cynthia said, well, that's very urgent. That's important. She went back to MIT talking to her PhD advisor, saying she wanted to switch the research topic into climate, climate adaptation, and so on. So in the past two years, she generated very, very interesting results. I said, why don't you come to Stanford to tell us about what you have been doing, what you have learned. Here we go. I think Cynthia now is back, ready to tell us about the results. But let me tell you, she picked a date of today to give the energy seminar. I think she used her AI model to know this extreme weather. In the past few days happening right here, this gives the best background introduction about her research. It's never been even more important time compared to now to work on the climate adaptation. We start short introduction. Cynthia, take it from here. Thank you very much for the introduction. And I am today here to talk to you about weather forecasting and disclaimer. I promise I did not bring the storm. It was not a part of the, it was definitely a coincidence. So as Professor Yi mentioned, I am Cynthia. I am a fifth year PhD candidate from MIT. And since I am very honored here to talk in front of brilliant and aspiring students at Stanford, I thought I might share a little bit about how I got here. So my academic journey has not been so linear. I did my undergrad in mathematics in London. And then I actually worked in financial industry for about two years. The work in finance was very exciting and very interesting. But I felt a drive to do something more impactful and meaningful with my work. So this drive led me back to pursue a PhD. Now at MIT, working with very amazing advisor and aspiring peers on some interesting problems for climate change. So without, so I'm very honored and excited to share with you some of the work that we have been doing so far. Climate change undoubtedly is one of the most pressing problem of our generation. Over the past century, we're seeing tremendous surge in the number of natural disasters worldwide. They're not only deadly, costing a lot of lives. They're also extremely expensive. So in the past 20 years here in the United States, the total loss amounted to more than $2.1 trillion US. Is it better now? OK. It's good? Great. So basically kind of the goal that guided basically my PhD work is that in light of climate change, is there something we can do utilizing our knowledge in machine learning and optimization to facilitate climate change adaptation, to build resilience, and to aid sustainable development. So before I talk about how to do weather forecasting, let's first take a quick tour of history of how we got here today. The art of weather forecasting actually originated, began with early civilization back in 340 BC. Greek philosopher Aristotle wrote Meteorologica, in which he documented his theories about the formation of rain and storm, et cetera. Also around the same time, Chinese astronomers developed this festival calendar in which they divided the whole year into 24 festivals, each of which contains a certain weather pattern. So over the next few centuries, as scientific methods developed and also measuring capabilities improved, that led to our understanding of the physical laws underlying the atmospheric dynamics. So interestingly though, back in early 1900s, when the first dynamical model was introduced, it took Richardson, him working alone by hand back then everything was calculated by hand, it took him several months to produce a wildly inaccurate six hour forecast for the town of Munich. So obviously that was not very helpful if you have to wait for several months to do so. So really, weather forecasting became kind of a more operational field when we started to use modern computers. So it was Professor von Neumann and also Charney, they started to do so and then 50 years later, we finally had the first operational forecast. So kind of since then up until now, weather forecasting has been predominantly made by these dynamical models using a very physics-based approach until AI started to change the game. So we are right now seeing exciting interests growing both in academia and in industry in how to use machine learning and AI to do weather forecasting. And DeepMind is really leading the way in producing operational deployment. They have been working with the UK Meteorology Department and actually just a few weeks ago, well more than a few weeks ago, but last year, end of last year, they released a graphcast. So it is the first time in history where machine learning model outperformed the best dynamical model. So in my lab, we started to work on this topic since the beginning of my PhD. And I'm here today to tell you more about how we take a similar approach to do forecasting of tropical cyclones or hurricanes. So why is AI really taking off now, why not earlier? Like in many other sectors and fields, it is the increase in computing power, data availability, particularly driven by satellite image observations and also processing techniques. And in particular, because of the growth of computer vision, natural language processing and time series techniques, we are able now to do what is called multimodal machine learning, which is really at the frontier of machine learning where you're trying to synthesize data from different modalities and sources. So my PhD work centers around the development of this multimodal machine learning framework. And as mentioned earlier, I have worked on it, so my group at MIT actually focuses primarily on healthcare applications. So I have been working extensively for healthcare applications to develop the kind of the technology to do multimodal machine learning. But my interest is really in weather forecasting and also climate forecasting. So I have adapted kind of leveraged the experience there to predict weather events such as hurricanes and floods. So the technology itself is very cool and exciting, but really what excites me personally is the vast and profound implications this technology enable us to do. Imagine that you can now forecast flooding events for your house say three to five years, then you might do something different to protect your family. And as a society, we might do urban planning, infrastructure planning and insurance very differently if we are able to forecast risks in a very different way. So I have been working on applying it for insurance pricing and also sustainable manufacturing. But the focus of the talk today is kind of how to do weather forecasting using machine learning. So to introduce the concept of multimodal machine learning, I would start with kind of a folk story from India. So this is a group of blind men trying to touch an elephant. So depends on where you touch the component of the elephant, you might infer very, very different outcome. So you would only be able to know it's an elephant if you kind of go around the elephant and take data samples from multiple sources and places. And this is kind of the idea behind multimodal machine learning. So how we started to work in this field is really from healthcare. And it is really a natural way of thinking and inferring. When you go to the doctor, when you have a lung problem, for example, the doctor would prescribe a chest x-ray and the doctor would think about your chief complaints and the doctor would also think about your age, your gender, your past medical history and take all of this into account to make a final decision. But when you look at this, this is across multiple sources and modalities of data, ranging from images to language data to tabular data. So if we can make an algorithm that is able to make decisions and inference by synthesizing data, then it should be better than thinking about each individual modalities. And this is exactly the idea behind multimodal machine learning. And really kind of the framework is very general and the application is also very vast, ranging from self-driving cars to weather forecasting, healthcare, recommender system, et cetera. So when you think about data fusion and synthesizing data, there are generally three strategies to do so. That is early fusion and pipeline. There is the late fusion on the other end, which is basically consensus model. So you would have inch individual models and take an average consensus. Then there is the mid fusion approach, which is basically kind of something in between the two extremes. Based on our experience, we have found mid fusion strategy to be the most performing and also the most versatile. For reasons, I will explain next. So here is kind of an illustration of the overall multimodal framework, taking the mid fusion approach. So we have different sources and types of data. We process them individually and then we concatenate the features from different data sources in the feature space. Then we make the downstream prediction. One of the advantages of this approach is that in the first step, we can leverage some state of the art processing techniques. For example, right now, everyone is talking about the large language models, so we can use them to process language data. And as the models themselves evolve, we are able to update and upgrade our processing techniques as well. And this approach allow us to add data and take away data very easily, making experiments extremely easy to do. And in addition, in the downstream case, we can once again leverage different prediction models based on different needs. So for example, in the healthcare domain, interpretability is very important and figuring out which features contribute to the model is also an important task. So we might take a regression model to do so. And when thinking about kind of the overall performance plus computational cost, maybe tree-based approach are more faster and cheaper. So the first example that I'm showing today is on hurricane forecasting. So the science of hurricane is that they are rapidly rotating storms that originates from the tropical warm ocean waters. So they draw energy from the ocean, from the water, and they release energy through rainfall, especially when they make a landing. So in order to protect our communities, we need to know basically where it's going and also what is the magnitude of this hurricane. And these are track and intensity forecasting tasks separately. When there is the hurricane, typically this is what you see on the news, which is in the field, they call it a spaghetti model because each line represents one model forecast. And overall, we would take an average consensus of each individual model. And broadly speaking, in the field of hurricane forecasting, there are three categories of models to do so. Obviously, the most performing ones are dynamical models. As highlighted here, GFSO and HWRF are kind of the leading two dynamical models. One is run by the North American NOAA, the North American Weather Forecasting Agency. And the second one is run by the European Weather Forecasting Agency, which is typically what we discuss in the news, the North American model and the European model. In addition, there are statistical models and also ensemble models. And typically, if there is an official guidance, it's made of an ensemble model, which is kind of taking a weighted average of each individual operational models, either dynamical or statistical. Statistical models are typically a very simple regression-based model. So it's kind of the precursor of machine learning approach by looking at patent recognition. So in this task, we set ourselves to forecast hurricanes by taking a multimodal machine learning approach. And specifically, we take two modalities of data. The first one is historical storm data, which is typically the data used for statistical models that consist of certain basic features, such as the basin of the hurricane, the time right now, latitude, longitude, et cetera. In addition, we coupled that with reanalysis maps. So reanalysis maps are kind of, they come in a picture format. So basically, they look like a satellite imagery picture with each pixel representing kind of the numeric value of a certain atmospheric character. In this case, we have taken the wind speed of the U, V, and Z direction at three atmospheric levels. So basically, that is kind of nine pictures at each time step. So here I'm presenting the results for hurricane forecasting. Here, this is for intensity task. And I'm showing the mean absolute error and also standard deviation for standalone machine learning. And we are comparing ourselves to operational models. So the first two lines you see here are all of the machine learning models that we have trained. And the last three lines are the most advanced operational models from statistical approach and the last two are more dynamical approach. We compare results in two different basins because the behavior of hurricane is very different in the two basins in Eastern Pacific and North Atlantic. And these are the out-of-sample experiments we have conducted using the last three years of the data set that we had. So that kind of accounts for 2016 and 2019. So here we see that the most, so in Eastern Pacific basin, the most advanced machine learning model shows very competitive results with the most advanced weather forecasting dynamical models. In North Atlantic basin, on the other hand, dynamical models outperform slightly to the machine learning models. However, it is noteworthy that although these machine learning models, most of the training and cost of training and the time of training comes before. So we pre-train the model and the deployment time is basically seconds. And we can run trained model on a personal laptop to make predictions. Whereas those dynamical models, they're typically run at supercomputers in national labs and the runtime is usually four to six hours. So machine learning models are able to actually make real-time forecast using the latest data available. So that's kind of comparing a long machine learning model versus operational model. But what is actually more exciting is that we thought, okay, since official forecasts are typically a consensus model, why don't we try to add a machine learning model to the consensus model and see how they work with other models? And what we have found is that when you add machine learning model into a consensus model, we actually outperform the official forecast in two basins by a little bit. So this is extremely encouraging for the kind of the future of adding machine learning into one of the operational models. This is all great. Obviously, it comes with, it cannot be a fairytale that everything is so great. You might ask, okay, so then why don't we just deploy it already? So there are certain limitations and one of which is that I mentioned that we use reanalysis maps. So reanalysis maps are available retrospectively. So they are obviously unfortunately not available in real time, but it is the best quality of data that we can access to for academic purposes. And the second problem or the second extension is that we started this work at the beginning of my PhD about four years ago. And in between a lot of things have happened. For example, there is the large foundational model, the emergence of large foundational model and also the emergence of kind of natural language processing as a technique. So basically we need to leverage these models and also we need to add additional modalities of data because adding more rich data into a multimodal approach is always going to lead to better results. Kind of these two thinking led to the second project that we did for flood prediction. So basically kind of how I started this flood prediction was at the time I was really brainstorming with my advisor, which natural disaster to pick. So we had a map, we had kind of a chart that says, oh, these are the earthquake, drought, wildfire and obviously flooding. And the reason that we picked flooding was at the time Pakistan was hit terribly by flooding and there were lots of Pakistan students, basically fundraising outside MIT and Harvard. And I personally had friends who are from Pakistan and the economy was just destroyed because the whole country was submerged, a third of the country was submerged underwater for several months. And this is devastating for a developing country. So then we decided, okay, maybe we want to do flood prediction. And kind of the nice thing is that when we use satellite images, we actually don't depend on developing countries such as Pakistan to have high level quality of data because satellite images are available globally and therefore we're able to leverage this available data to do forecasting on a global scale. So kind of a quick overview of what is the existing approach to flood risk mapping. So there are short-term forecasts and long-term forecasts. The short-term forecasts typically depend on the hydrodynamical models, which are once again physics-based modeling. They're rather local scale. And also, when you want to stretch the forecasting horizon, then arrow propagates because you're always kind of feeding the prediction into the model. For longer-term risk maps, for example, if I wanted to know what is the risk of flooding in five years, there are typically kind of historical flood risk maps. And one of the issues is that they're kind of historical looking and therefore unable to deal with the changing climate. So for this work, we decided to take three modalities of data. In addition to the statistical data and also satellite images, we decided to leverage language data. So language is, you know, it's a data that is extremely vast and available on a lot of topics, but the problem is that it's very unstructured. So thanks to, you know, right now the growth of natural language processing and everyone is kind of becoming more and more dependent on large models, such as chat GPT, we're able to better utilize language data. And in this case, we thought about using Wikipedia's geography section to take in consideration of the geography features of a certain place. Because basically at the time I was reading a lot of those hydrodynamical model papers and I realized that a lot of the times you need to know what is the river location and kind of the topological features and if there are mountains and oceans to be able to understand the geography of a certain place. If you search Wikipedia, it actually has a lot of information. For example, there is a lake nearby, there is ocean nearby. So this is the approach that we took. So I have to present kind of not semi-available, semi-ready results because this is still kind of ongoing work that we're trying to do. But those preliminary results show us very promising numbers that basically show us that if we take a machine learning approach, we're able to forecast extreme events such as flooding up to years in advance. And obviously we're still trying to refine and improve the results. This is already extremely promising and that shows the potential of using such an approach and especially taking a multimodal approach can have for tailgate events such as flooding. So kind of combining the two examples, I'm sharing certain lessons that I have learned during this experience. So the first lesson that I have learned is that multimodal approach typically outperforms single-modal approach. And the second lesson that I have learned is that machine learning models are able to produce very competitive results to dynamical models, but in a fraction of simulation time. And the last lesson is that scalable machine learning models can have the potential to produce long-term risk models that would have basically profound implications to many, many fields, infrastructure planning and insurance. So because I honestly think that this is super exciting and that's why I'm trying to pursue an academic career to continue the work and to refine the technology and to broaden the impact. So I have worked on hurricanes and flood, but obviously a similar technology can be applied for droughts, wildfires and earthquakes, etc. And that would really change how the society works in many, many aspects such as agriculture or renewable energy planning, for example, as we shift into more renewable energy supply than if there is, say, no sun, then we cannot afford to have a blackout. So basically the capability to know and to forecast whether in advance will help us make decisions to make our society not only more efficient, but also more safe. So this is all I have prepared for the talk and I'm very happy to take any questions. Thanks, Cynthia. That was a terrific, very exciting research, very exciting research directions. I especially appreciate letting us know a little bit about what's underneath all those storm track, hurricane storm tracks that you see on K-Ball News constantly at certain times of the year, also commented by some political leaders and others. So we do have about 15 minutes for questions in the room. We usually start with student questions. Hi, Cynthia. Thank you so much for the great talk. So the non-linear dynamical systems that govern a lot of these systems, they are known to be chaotic, which means you have to write the simulation for four hours to know exactly what the system is going to be doing. And normally, given a system, you can actually derive mathematically your time horizon beyond which you cannot predict anymore. So I'm wondering how does the machine learning approach address this problem? Yeah, thank you so much for the question. So basically, you're absolutely right. Kind of we picked hurricanes to forecast. And at the time, people thought we were crazy because it's known to be very chaotic and it's one of the most notorious tasks. But basically, what we observe is that the results that I show here are for 24-hour advance forecasts. And basically, both dynamical models and machine learning models are able to make reasonable forecasts, although not very accurate. As you see, standard deviation is pretty large compared to the MAE. But when we actually stretch the testing horizon to 48 hours, we observe that actually both dynamical models and machine learning models stop producing reasonable results. So basically, it is most likely due to the chaotic nature that things evolve drastically as you stretch the forecasting horizon. Other questions? So Fintia, for that, how can you expand the learning? Expand the learning, so you have what, so far as 2016 to 2019, you said that that learning. Expand the learning, would you be able to, based on the past data, increase your, based on the machine learning approach, the accuracy will increase, the error bar will be reduced. So what's your... Yeah, thank you for the question. So actually the testing period is 2016 to 19, so it's three years, but we actually use data from 1980 as the training data up until 2016. However, the issue is that we didn't have access to earlier data because read-in-access maps were not available before then. But generally speaking, as you increase the data set and the availability of data machine learning models tend to increase the performance. Yeah, can I ask one more? No matter it's hurricane or the flooding, I'm also wondering what about people's response to a debt. Let's see if you can predict and say, hey, the flooding is likely coming, it's time window. Then what would otherwise people do? Building infrastructure, of course, is very expensive and you must be thinking about how people can respond. I remember when California having this power consumption this last year, right, John? It's looking, it's going to exist, certain threshold. Then we all get text messages and say, go home and reduce your air-conditioning need or something like that. And suddenly just several gigawatt or power is reduced. I'm interested in the response part based on machine learning and this outcome, because to see what you think. Yeah, I think definitely this is a great point. I think that one of the key advantages that I believe machine learning model has is the capability to use real-time data to make forecasts at the spot. So I think in terms of response and emergency response, especially being able to update forecast models using the latest data at a shorter frequency is able to kind of be very helpful to save many, many lives. So because dynamical model typically take a few hours to run results and at the time things might have evolved drastically. So I think definitely this is a very interesting angle. Thank you. Yeah, actually, if I could intercede, there is a lot of research now on dynamical multi-sector models. And I think your general conclusion regarding the synthesis of the structural physics models and the machine learning models holds over. So I think an exciting thing would be to work with people who do that kind for someone like you to work with those people and they do both the physical and although it's harder, human response, human adaptation options and meld everything together. I think that would probably lead to much better operational results than we now have. We've actually looked pretty carefully at the flood map approach and it's actually frankly not very useful at all for people who are actually trying to make decisions on the ground. I think we have one over here and two back over here. Yeah, thanks for your talk. I wanted to ask about the multimodal ML. I can understand in general why it's useful, but particularly on disasters, where do you think the textual information will help? Like in the example you showed, it didn't seem to be helping that much, but maybe there's things you have in mind where there's information capturing text that actually help pinpoint where things might happen. Yeah, so actually I didn't add the slide, but for example, so for example. You weigh a stupid little bell, who's our expert on food security, so you have at least a little diamond point in this. Yeah, so here is kind of, thank you for the question. So basically here's the backup slide that I didn't include. So basically for hurricanes specifically, there's a lot of experts who describe the behavior of the hurricane in a text style. For example, I give an example here, so there's a lot of kind of the characteristics of a hurricane specifically, and being able to leverage that from a language format probably has some additional insights into different other types of more structured data. So this is one example. Can I read the results for the example that the text didn't add much to what you already have with the statistical? So for the flooding one, the actually adding text helped a little bit, but the overall results are not so great. The ROCA we had the highest was around 0.75, which is why we're still continuing to kind of improve this by adding more satellite imagery data from topological features as well as climate features, but we're still doing experiments, so I unfortunately cannot share more, encouraging results at the moment. Thank you, Cynthia, for such an amazing presentation. You mentioned you're a near-term risk and then long-term risks. I'm wondering could you elaborate a little bit more on how you have envisioned this result to what are the indications for long-term climate adaptation, especially in the exciting field about renewable energy penetration and infrastructure planning, because those designing take a long time. I'm wondering how this model can help us have a better and smarter design in those areas. Thank you. Yeah, thank you for the question. So I kind of come back to this page of results. So basically kind of when I started to do this work, basically I tried one-year flooding prediction two years and five years, and it usually when you think about it, my first instinct would be five years would be harder than one year, at least in the healthcare sector, it's harder to predict the outcome five years than one year. So Mike, I'm kind of hardwired to think that, okay, longer is worse, but actually in this one, results show that we don't see a decline in accuracy over the kind of the horizon, and in some certain cases, it is even easier to forecast the probability of a place having a flooding event in the next 10 years versus one year, because if this place is more prone to flooding, then 10 years is kind of a longer window for it to happen. And in this lens, I think that it's kind of the opposite of normally how predictions work, and this kind of longer time horizon is able to allow decision makers and policy makers to make, have enough time, say to do infrastructure planning, to prepare, for example, better drainage system in certain areas or build better dams, et cetera. Thank you for your question. Thank you. Thank you for your talk. As a future thing, you have suggested renewable energy economy. How would you take it to account geo-political height and height of this, in the model like this? If I'm right. Most people have subsidy policies, right? Are they predictable? How would you, any perspective? I guess you're asking a machine learning engineer from MIT to comment on the geo-political implications. I think I'm right, the idea here would be to do better predictions of wind as it is input to wind machines, so how to configure them. There's lots of research on how to configure them, how they space them out, what locations to put them in, is that correct? So it's more of what you could do with a wind farm, where's the best place to put a wind farm and get the maximum output from it. And then the politicians can, however they do it, they can use that information as some of the input to their decision-making process. So whenever you have an opportunity. So it's so hard to cite these things whenever you get a chance to do one. It's very good to optimize the maximum output, effective output you get at the right time. Yeah, I suppose that kind of, at least personally, I personally haven't thought about this, the geo-political implications. And obviously when you think about energy these days, especially with the war, et cetera, it is an unavoidable topic. But frankly speaking, I am kind of taking a more of a utilitarian approach, thinking that how can I make the overall energy system perhaps on the local scale more efficient and more resilient against climate events, for example. Yes, thanks for your great speech here. And I'm just very curious, because I think there was a tiny hurricane happened in Stanford yesterday. So I was wondering what is the minimum level of the hurricane this model could forecast? Yeah, so, well, I think that, so basically when you think about a hurricane, so a storm, so basically a storm, when it passes a certain threshold, it is categorized as hurricane. So basically at least from our work, we only work predominantly with storms that are already big enough to be categorized as hurricanes. But I think that it really, in terms of what is the minimum threshold of a storm that can be used, can machine learning apply to predict, it really depends on the training of the model. So in our experience, we made the conscious decision of only predicting hurricanes, but if one were to develop a different model by taking data sets from all storms, then the threshold doesn't necessarily need to be there. It's more of a modeler's choice. Does that answer your question? Okay, great. Other questions? I have one last one. I've always been puzzled by a set of results that came out of the National Oceanographic and Atmospheric Administration Colorado Office about 10, 15 years ago. So see if you can help me understand why this might be. That was in many of the climate predictions people have done. This is mostly historical comparisons. The big climate runs, it do take a lot of time to run. If you just take a combination of the existing forecast, you do better, retrospective forecast, you do better than any of the individual models. Any idea where that might be? Yeah, so basically this is kind of the... I guess you're asking why ensemble models, for example, you have many, many spaghetti models, and then you take an ensemble weighted average, it would tend to be better than each individual one. I think that really, I would say part of the reason is due to the chaotic nature of such hurricane evolution. So basically when things are chaotic, the outcome is extremely different based on the very sensitive input that you put. And every model comes with certain errors and kind of wisdom of the crowd when you take more consensus and kind of averaging, cancelling the errors, the results tend to be better. It's kind of like a lot of large numbers or something like that. Yeah. Any other questions? I have lots of more technical questions. Last one, this is the last one, I'm just doing that. Yeah, actually I just realized this. So have you thought about kind of like augment your real data from simulations to your machine learning model? So what I mean is like you can simulate a hurricane that hasn't happened before. People haven't seen this type of hurricane before. And now you simulate it and then add that data as if it's real data to your machine learning model. Yeah, that's a great point because basically one of the critiques of our approach is that most of the data that we use to train, all of them are historical data. And as climate increase and the sea temperature grows, a lot of the meteorologies are suspecting hurricanes to be more stronger, to have more intensity. And that is a major weakness for us to take a machine learning approach using only historical data. And what you mentioned could be a great idea to complement the data set, to basically train our models to be able to learn from more potentially bigger hurricanes that haven't happened yet. So I think it's a great point. Thank you. Definitely food for thought. So with that said, thank you very much Cynthia for a very good talk that opened our eyes to a lot of new horizons that you're working on. We'll look forward to seeing results in the future and thanks to the audience for excellent questions. Thank you very much.