 So everyone should be joining us. I would welcome it's a good afternoon for the ones in Europe and below. And I wish a good morning to the people in the other side of the pump and the Americas. We try to organize the webinar at the time zone that would be convenient as much as possible for all of us and I will give a couple of minutes before I start the formal part of the introduction. So Hans, I mean you're virtually at the institute but practically are you in the Netherlands? Yeah, yeah I am close by. Close by me too. I am around let's say 10 kilometers far from the office physically but virtually I am in. I checked the office today so it is the same as Montpécro. Good. And how is it with the Euro experience? We are not doing that well. I mean we have all the coins now for Belgium. Yes exactly for me I lost from all sides from Netherlands, from France then I am in between now I am in Belgium I think or maybe Switzerland or okay that's fine. But you're still doing better than us but we didn't travel at all for this event. I think they didn't do a good job. Well individual they are supposed to do better but the worst not real team work I think. And that's where we are getting ready and I would officially kick off. I wonder how predictions would work for our soccer games and if these catastrophic results were predicted by artificial intelligence either in short term or long term. I think I mean can give a story about Beijing's network and Barcelona. Let's start with that a story about Beijing networks and Barcelona. This is a good warming up for a webinar as I welcome everyone that is joining. Let's talk soccer then in that sense. I know big teams like Liverpool and Barcelona that are using Beijing network too. They have a special team in artificial intelligence and big data just like any physical preparator, psychological preparator, then they are analyzing the data now even to see who is the key player who gets more balls how to prepare the strategy and the tactics just playing with Beijing network. And there is a couple of I know also even I think IX last year tried to do something similar just to playing with graphs and vision network find probabilities the critical path in the team. It's really now is not just skills or physical preparation or money is also knowledge. Football nowadays is moving to knowledge in that sense also. And there is lots of science behind it and there is lots of data science or computer science. That's what I hear you describing and I think that this is exactly what we want to cover at this webinar. So welcome at the webinar focusing on emerging risks and in particular looking at emerging risks that are coming to affect an overall system. And it's going to be very interesting. Let me briefly say who I am. I am Nikos Manusialis. I'm the founder and CEO of Fagrano. I've been working in the space of food and agriculture data data management and algorithms that are trying to predict or suggest things to users since 2005 which means a lot of years. I'm working on the field. My PhD was on agricultural data management. But it had a very important algorithmic component looking at the interactions of users which were the information sources that were interesting and trying to suggest things that could be of interest to other users. So my heart is really very close to the topic that we will be discussing about today. And as an inspiration for this webinar and for some other webinars that we are organizing with other colleagues is something that about six months ago a colleague from the industry told us in one conversation where he said there are so many models out there. There are so many solutions out there that are trying to predict the future. But what they don't focus on realistically are the actual problems that we are facing. So plenty of solutions but not clearly defining their problem. And this was the inspiration for starting something together with some colleagues from Belfast as an informal interest group but now is growing and is creating interest and is a space in which most people from the industry but also from the academia are sharing experiences and practical applications with the interest group on predictive analytics of food integrity. This is a group where we are sharing knowledge, ideas, models, systems that people can play with or concepts upon which people are requesting people to contribute with requirements or validation or other things. And we have invited all the members of this group to join this webinar. We have plenty of people joining us today. And in this conversation one of my favorite simple diagrams that help us decompose the problems is this one which says that we're not trying to answer a question being it a scientific or a business question by trying to calculate some predicted indicators. I have to select the right variables, the right indicators for which I will calculate their future values and it's an important step selecting these indicators because they will be the signals that I will be using to give answers to my question. And then I have to go and see what kind of data can I feed into my model and which is the right model, the right method that I will select. And this is exactly at the heart of what we will be covering today, asking the right question. And today the question is about long-term risks, not the short-term prediction what will happen next week or this month but looking a little bit ahead. I want to welcome our speakers, Hans, Marvin and Yamin Buzabrak from Vaheningen Food Safety Research. We're working together at the Food Safety Market Initiative. We're looking into ways in which we can take advantage of lots of data so that we can support some important food safety decisions. And they're going to talk about two complementary topics, the scientific approach behind predicting emerging risks and the way that we can use AI technologies to support solving such problems. This is the core of the webinar and we'll have some time for questions and reflection at the end. So Hans, the floor is yours. I will stop sharing my screen and hand over to you. And you can start by also telling us a couple of words about you. Thank you very much. Yeah, my name is Hans Marvin. I hope I think the first slide should be up. Oh, that is the wrong one. Okay, I hope you can see the slides now. Yeah, my name is Hans Marvin. I'm a senior scientist at Vaheningen Food Safety Research. I started working in Food Safety in 1999. Before that, I was working on the whole supply chain, more looking at industrial application of industrial crops. My background is chemist. I've studied chemistry and PhD in it. And now I, since a couple of, let's say seven, eight years, together with Yamin, we started introducing big data and AI in food safety research. And in this presentation, I will show you why we started with it. And with my presentation and the one of Yamin, you can see what you can do with it and how it helps us to be more proactive in measures that we can take to prevent food safety risks. So this presentation is together with the Yamin Busenbraak and it's called System Approach of Predicting Food Safety Risks. And Nikos, thank you very much for this invitation of an opportunity to present this approach. And I hope all the whole audience would like it. And first we'll say why I started doing this with a system approach. And it happened really when I started working at Wageningen Food Safety Research, or at that time it was called RICO, say in the beginning of 2000. And I will explain you how it works and I will give you a demonstration about the methodology. So why did we start? When I arrived in 1999 in our institute, which is dealing with food safety, we are an institute that only does research on food safety, either for the government or international organizations and also certain conditions also for the industry. At that time there were a lot of incidents described in the media. I had a lot of media attention and that caused a lot of concern by the broad public about how the safety European food supply was arranged. So there was a big concern. And of course that also initiated a new revision of the legislation in Europe. But that was not an ideal situation. And when we were in that, and also our institute was very much in the analysis of food safety components or contaminations in all this product. And we had a 24-7, let's say, availability there because when it is a crisis, we are doing the analysis for the authority. So we had to scale up. So that was not an ideal situation. And so we were trying to figure out other ways why, first of all, are we only behind the facts. When we do analyze all the monitoring and the early warning systems that were in place at that time, they were all hazard focused. So they were following a measuring, monitoring a specific hazard, known hazard. And when you find that have measured, basically most of the time then there was a problem. Because then you have to take measure if it was a really serious problem. You also can, there were also monitoring programs and a system available that follows consumer and animal health. The rapid response monitoring code. These systems give you a little bit more time. Because if animal gets sick, you know, okay, that is not necessarily food safety risk, but can turn into food safety risk if, let's say chemicals or treatments are used and you have elevated concentration of residues. We realized that we have to think out of the box and maybe try to develop a holistic approach. There was nothing of this kind of system were available at that time. So we had to develop ourselves. So what we did is, okay, quickly, of course, food supply chains are not completely isolated from the rest of the world. First of all, they are international. So there are a lot of connections throughout the globe. But there is, of course, a lot of direct and indirect interactions with other domains, like the science and technology environment, the climate change, the government, the politics, for example, legislation. These legislation can certainly make that some activities that you did in the past certainly cannot be done anymore. Health and welfare, agriculture. So all of these domains here have an impact direct and indirect on the food supply chain. And when we analyzed all of these past incidents, and we looked at whether there were changes or indicators that you could say in one of these domains around the food supply chain, and if we were able to, if we would have known and would have followed those, would we then have been in time to take measures to prevent the food risk to occur? And we have done that in many of these incidents, dozens of them. And we found that many of these indicators can be identified, either in science and technology, in human behavior, in nature environments, in legislation economy. Some of them are really directly linked, in the Netherlands, for example, in the legislation of the environmental release of components. Many of these indicators and related data that were identified were generic, but also case-specific. The conclusions that we had, because this was done by us in 2008, is that there are signals in domains around the food supply chains that precede the food safety incident. However, we much more research is needed in order to identify these drives of change, how to model them, and also to consider the complexity, because all of these indicators they have, yeah, that's not a direct interaction, but it can occur month or maybe year afterwards, or also there's a web of interactions. A study done at that time, also related to emotional risk and to identify the drivers with the international expert. And this was the result of it. Here, they will ask, what is the most important driver of immersion food risk in your country? And this was a very international, I don't know exactly how many countries, but more than 30 were in full. And you can see it was a scale 1 to 5, and it was done twice, two rounds. And you can see that war and terrorism were identified as potentially drivers of an immersion risk. And the population growth was a migration, increased diseases, economic recession, at that time as well. Also, very high negative impact on occurrence, so that it stimulates impact of immersion risk in the country. However, both rounds, it was seen that the technology, maybe AI, would be a driver that could help prevent food safety risks to occur. So that was a positive expectation. So expert consultation is an important aspect of identified indicators and drivers of change. And we have tested many of those, and also optimized which one you best can use in which circumstances. So we can use interviews, focus groups, descriptions, questionnaires, Delphi method, FMEA, failure mode of effect analysis, where you score the FMP, which is the severity times occurs and detections, just in order to rank the different impacts of indicators. And we analyzed in a big project which one you can use best in which circumstances for immersion risk identification. And that has been published in last year in transit food science and technology, if you have interest. So the troubles that we are facing is that, okay, when you have identified the drivers, there can be many, and you don't know the mode of actions because that's often not known. And also you have identified the drivers. You want to connect them with the expert knowledge about the food supply chain, but also other expert knowledge about trade flows, about prices, and so on. And you want to connect that with the, let's say, the food supply chain. And how to integrate that, we started eight years to use artificial intelligence to use that to predict food safety. I will demonstrate how this out of the box holistic approach works in order to predict the food safety. I will use the dairy supply chain as example. And this was done in an FSA funded project called Demeter, which also has been published. What we did in this project was first started a literature search on drivers and indicators that experts think have a direct or indirect effect on the performance of a milk supply chain. We identified experts in academia, industry in a country of interest in Europe, because we won't focus in Europe. And these are the major milk producing countries in Europe, in the Netherlands, Italy, Greece, France, UK, and Poland. We sent experts, questionnaires for the experts, and we asked them about a lot of questions on emergency risk, drivers, and indicators. And we also asked them to rank. In total, we were managed, four of them were ranked high within total 12 indicators. We searched for data sources of the indicators and we were able to find all of them. So this suggests for you to have an idea about what we are talking in this case. Drivers were in the environmental domain, the social domain, technical and economic. And the indicators, which is said, this has an effect on the development of an emergency risk in the milk supply chain is the use of antibiotic in the dairy sector, the average precipitation in the country, share of land use of pastry, we group them under environmental, then the social toll, inhabitants per country, the average age of the dairy farmers, the population per country, and so on, economic, the raw milk prices, the feed prices, the incomes of the farms. And we have various number of data sources that we identified that will provide information about these indicators. Just to be short, to show you one of these indicators, and this is the milk price as an example, because when you have a data source, you can also, you try to collect as early as possible, and we were able to check back prices from 2008 to 2020. And for all of these countries, we see the fluctuations of the price on a monthly basis. You can do a statistic analysis to see when a price increases or decreases above a seasonal effect or what you can consider as a normal deviation. And that's what we call anomalies. And these are the filled dots within these graphs. You can see for all the countries, there are anomalies found. And our question was, isn't an anomaly in one of the indicators, in this case the price, be a warning of a potentially full safety risk later? Here you see as an example, the number of anomalies that we found in some of these indicators. And for the different countries, so this is the indicator, these are the countries, the number of anomalies records found. So here we found in the period 2008, 2019, we found 2008, 1997 and so on. I will continue making it a little quicker. And indeed, with a statistic, we could find that these anomalies have a direct link. This is an example of Germany and everything above this gray plate is a significant correlation. And you can see the window was preceded about 15 months of the same full safety incident as reported in RASF. Here is a summary of that information. These are the indicators and here are the different countries. And you see for all milk price, there is a lack of, for in Germany, for example, of 10 months, which are positive correlated. In France, it is 21, Italy 22 or positive correlated to the Netherlands 10. So there is a time differences, let's say, more than almost between a year and two months, two years before this. So this is based on data from many different sources all over the world. First, of course, you start with manually, but then we wanted to do it automatically. So we have now a workflows installed in our high performance infrastructure that automatically collect from all of these data sources to data. It cleanses, it checks every month, whether they're on a week, depends on how often it is updated, whether there are new ones, even data from tables in a PDF. And we have made scripts that are able to do that. Then there is the calculation of abnormality. And if there is some abnormality, then this is visualized. And if there is abnormality, automatic color is given. We can also do the same with the basis network for combining that. Let's say a basis network, I mean, we'll tell you more about that. And there you can easily combine these different steps. We also included a basis network in this workflow. That means that also when there's an abnormality with the Bayesian network, it also gives what type of hazards you can find. So here you get a warning a year ahead. And this also says what you can expect, what kind of hazards. Conclusions. Expert elicitation in our experience is a crucial step in the system approach for the development of future prediction. Animalities in many different various different indicators show significant correlation between lag times and rush modifications. And basically, it gives you more than a year. We have created workflows. We use open source software 9 that allows to do it all automatically and also give warnings when a problem is at hand. And this also suggests that early warnings in indicators can precede a real incident. I'm not doing that a lot. We have a team of seven now with two new persons coming in that have worked on this topic. And thank you very much for your attention. And if you want to learn more about what we are doing, you can approach this. Thank you very much. Thank you, Hans. And as we are switching to Yamin, I will make the observation that apart from computer science, there is also a social science component. I saw that is essential in understanding which are these signals that show that have a proven correlation and that we can use as input. Yeah, you always have to use experts. That's a very interesting point. Okay, Yamin, are you ready? Please unmute. You see, that's why it's always nice to give justice in you. Thank you so much, Nikos. Do you see my screen? Yes, you're good to go. Super. Just to put this one on the top. And this one in the corner. Yeah, Nikos and the Bruno guys, thank you so much for organizing this webinar in this very interesting topic for us, which is our business basically. I can introduce myself shortly in that sense. I'm Yomi Buzenbrek. I work in Wagner Food Safety Research with Hans now for almost seven years together, trying to implement AI and the system approach in food safety. This is really our expertise. We use everything that is available in AI and new technologies specifically in food safety. We apply AI and food and machine learning to keep and make food safe. This is really our mission. And thanks for all our participants that joined us from everywhere in the world. I hope people will get really nice questions and discussions. And thank you so much for joining us and giving us this opportunity to share with you our findings. As already introduced by Hans in that direction, I will just show you two cases that we think are worth to share with you, which were we applied system approach and AI. Yes. What I will show you of course is thanks to all these guys that things were really in practice happening. Thanks for Anna, Nijin, Ljunika, Lukas, Rota and Hans. And basically the two stories that I wanted to share with you. The first case is about combining the system approach and the patient network to predict food safety hazards in fruit and vegetables, like a case. Then the second one where we use natural language processing and text mining to identify completely unknown things in food supplements. Because in addition to prediction, like prediction in short term, tactical term or strategic term, which is covered by the first block of the case, we have in food safety questions the things that are unknown for us, really. In the end, I will conclude with what we learned from these two exercises in that sense. Let's start with the first one. I think Hans already explained everything about system approach. We see the problems in food safety. It's like a system where everything is interconnected. That's very important. Where we should look at things at the world, right in the system, the whole food chain, all together, they're either just a group of independent part who is here. We focus on this or just take this like parameters for us. The system is a system, the food chain, the food system that we include in our models. It means looking to the bigger picture first. Then if you have a farther or deeper questions, you can dive in. And here, how we cover the operational level and the tactical level and the strategic level of our decision. We take everything and we take the historical data. Then if we want to predict the long term is already in our system and covered for all parameters, then if you have other scenarios or new questions in operational level or tactical level, they are already in the system. You can go with the system in weekly level, monthly level or yearly level. It depends on the questions that you have. It's built like that. It's the whole system, the whole history, and the daily operations, what is happening daily. For us, it's a full system that you can implement using Bayesian network. I like this first example. This is a few cases that we implemented using Bayesian network. The first one, as you see it here, this was really our first one where we learned from a food fraud case where we were involved in food integrity project, which was an EU project. Can you see here the name of our institute was written in that time. I like it even it was small, but it was the first model that we learned from it. From there, we started implementing the system approach and AI in very, very complex cases. We have now even more than 20 different applications in different domains for different questions also. The one that I would like to share with you today is the BN model for fruit and vegetables. Generally, the BN modeling steps, as you can see on these figures, have four steps. The crucial one that Hans already mentioned is the identification of the parameters that you are going to include in your model. Then you go to target which data to collect. Then collect the data, process the data. Then you need to develop your model like a third step and is the validation. It depends on your result of validation. It continues. It's a circle. It depends on your result. Are you satisfied with the result, like the quality of your model? Then if not, you need to maybe add other parameters or collect more data. It depends really on your, but it continues always. You can do better or improve or go to the next steps. I want to dive into each step of this, of the BN modeling part. Expert consultation for this case of fruit and vegetables. Yes. The first question, which factors are you going to put in your model? Then we check the literature. We ask experts related to food safety hazard and food safety in fruit and vegetables, specifically companies involved in fruit and vegetable chain, and they identified for us the main parameters are relevant to predict food safety hazards, like climate parameters, rain, temperature in the origin country, the use of pesticide, and all kinds of chemicals that you can use in the farm level, agriculture index stress, the prices and the important export data parameters. You will see later in the model how many parameters and all the factors that we had in the model. After identifying your factors, you go to the next one, which is the data collection. For this case, we used RASF data, where they notify the different food safety cases. I will give you in the next slide a short introduction to RASF data for those who doesn't know what RASF means. We extract all notification related to hazard and fruit and vegetable from 2014 for the main producers or the main exporters of fruit and vegetables to Europe. Then the last step is linking all notifications that you have in this database to the indicators that the expert defined, like climate, agricultural data, economical data, and all the parameters defined. As I mentioned just this is a short jump to what is RASF. RASF features a central system required by food law, managed by European Commission, and all EU national food safety authorities, EFSA, Norway, Leuchttinstein, Iceland, and Switzerland are in. The objective is to provide food and feed control authorities with an effective tool to exchange information about measures taken responding to serious risk detected in relation to food or food. If you type Google it, just type RASF portal, you will get to the system and the open access to everyone to see what is happening in food safety and feeding. In numbers, what we had notifications for fruit and veg in that period was around 6,000 cases. For the main countries like Netherlands and Turkey, India was around 4,000 cases. And the main product was figs, pepper, okras, apricot, pears, tomato, and the main has basically mycotoxins, pesticide, food additives, and flavoring, foreign bodies, and so on. Yes, we constructed the model. Yes, of course, we split our data to training and learning data set. We used 80% of the collated data to train our models and we used machine learning, arguing to build the BN model. And the remaining 20% of the cases randomly selected from the full data set was used to test and validate the accuracy of our models. And the accuracy for this one was really, really nice. It was higher than 96% in the quality of prediction of the hazard category. This is how it looks like, the fruit and veg basin network. In short, how I can explain it to you in an easy way. You see all these ellipses are the parameters. The green ones, for instance, the economical factors. You see yearly production, yearly import, monthly prices, agricultural GDP of the origin country, sown area in the origin country, agriculture index, prices yearly. This is also strategic operational and it depends on your A menu model and the production volumes, for instance, yearly. The orange ones here are related to the climate like mean temperature, monthly, precipitation yearly, precipitation days, minimal temperature. And the green, blue ones here are related to chemical use like insecticides, fertilizers, fungicides, and herbicides used in the origin country. And the ellipses, the white ones are basically the parameters given in the system like product name, the year, the origin country, the month, and if there is above emerald or below emerald. And the output parameter, what we were predicting was the hazard category. And you see here, like in these rectangles, you have the states here, the hazard category name, and the green bars are the distribution of your probability. And the numbers here, like 26 percent is pesticide residue, 57 is micro toxins, and so on. And like product also here, you see all the products that you have in your model. And it's the same for the other parameters. How can you use this model? You can play the power of it. You can play all scenarios that you have in mind. What if scenarios, what can happen before or in one year, next month, or given the parameters that you have now, what is the most, probably most likely hazard will be, will occur. And how I have some scenarios for you here, just for a few parameters because of sick of time. Here, for instance, I played the scenario where we imported figs from Turkey and we made just a variation in price, from zero to 1,000, 1,000, 2,000, and higher than 2,000 per ton. And you can see here, this is the output, which most likely chemical, if you have a question, okay, knowing the price of my figs today, which chemical should I check? Or if your question also is yearly, if I know the price of my figs in Turkey next year, this year, which chemical should I check next year? Then you will see here the effect of a price on your outcome, the things, the chemicals that you should check. For the first classes, when it's about zero to 2,000, it's always microtoxin, almost 100%, then when it becomes really higher than 2,000, really expensive, two times the normal price, you have foreign buddies that you see then questions of fraud and it's not only microtoxin, but you will have other food safety issues. Another scenario here is the same country, the same product, but the variation I was changing, the import, the volume, how much we are importing on tons. Then you see, for the first category from 1,000 to 5,000, it was microtoxins, then the second class microtoxins, then microtoxins. When you achieve a certain level of volume that you are importing from that country, you start, there is a limit for it. It becomes completely, you will have other aspects of hazard, non-patogenic microorganism, then organoleptic aspects, other completely other types of food safety hazards, and microtoxins becomes like below 40%. You can, what mice with this model, how we can use it. In that sense now for our authority, given the condition next year in the origin countries, you give me the temperature, the estimation, how much they will spend in agriculture, how much they will show on which product. We can give you 95% ahead, which product you should check and which chemicals also in your monitoring plan for next year. This is like strategic decision for one year. If you implement, as Hans mentioned, the system in real time, as we have it in our infrastructure, you can see any parameter if it's changing today, what is the effect of that one or which chemicals you need to check or you need to change your strategy with chemicals to change or which product. Yes, it depends on the dynamics on the market and the influence of these parameters, how they are moving daily. All this work was published and they are in open access. I think both of them. The first one of a daily case that Hans shared already with you in the first case and the second case, it's open access. Just type it and you will find it there. Yes, this one was about fruit and veg. Now let's move on to completely another AI approach, which is natural language processing. And to another topic, which is food supplement. And when I say food supplement, it's just, yeah, I think all of us, we know what is stimulant. It's a drug that produces a temporary increase of functional activity or efficiency of an organism or any of its parts that are used to treat obesity, increase focus on alertness, decrease appetite or decrease need for sleep. Yes, what's the problem with them? The market is really growing, especially after in this pandemic period, the consumption is increasing, the market is moving very quickly, and there is a lot of fraud and illegal compounds are sold as food stimulants. And the issue there, if they are illegal compounds, this means they can harm the health of the consumer, which becomes an issue for us, like a food safety institute and for the food safety authorities. Then there we had some research questions, which are, are there other compounds that should be added in our monitoring? When we say monitoring, we have a set of stimulants that we are checking daily to keep our stimulants and food safe for everyone. And what is being used and are there compounds which are not aware of what is happening in literature is something, what is happening in the markets is another thing, what is happening daily also is another story. Then the idea, like when we designed it, how to deal with this problem, we came up with this strategy, use text mining to mine Twitter, collect all the stimulants, tweeters and use sentiment analysis and text mining to get something from there. Another data source was a European media monitor that collects media, media reports worldwide, what is happening, sorry, what is happening in the world in media and mine that text using machine learning and consult with experts, you will get a set of new stimulants. And we use Drasaf also to see which ones are emerging in that sense, like an information and knowledge. And the last one that I would give more details about it, which is using scientific literature also, what is happening in science, in publications. And like data source, we used Europe PMC, which is a big database even bigger than PubMed. It collects scientific literature data with synonyms from chemical databases. Let's go to what we did, how we mined literature. We used like Europe PMC as our source of literature, which provides a signal point for access for the abstracts row, PubMed, PMC, full text and additional five millions. It's like if you take all scientific literature databases, Europe PMC is one of the biggest ones. And our aim is to find new stimulants in scientific literature that are currently not yet monitored by us using word embedding like an AI technique. All scientific literature that contained one or more of the compounds that we have and the synonyms in the title and the abstract were collected from 1990 to 2019. And the result of the harvest was 2.1 million articles, which is in size, there's no scientists who can read all this material and which is really a big data set. And the word embedding model was trained with this data to try to understand if there is any new stimulants there. Just here a short introduction what is word embedding is an AI technique used to identify words that occur in the same textual context. It's what is word embedding, is a high-dimension vector of numbers instead of text and converts all words into a vector of numbers based on all other words in the data set that you use to train your model. In short, we used word to check another network. Then when we trained our word embedding model, we used stimulants like a vector and for stimulants we find which compounds were interacted for were close to stimulants. And from this set, the compounds from the original four to eight compounds, the set of the chemicals that we have in the list and the synonyms were removed from that. And the top 15 found compounds were checked for the validity. From that we found this is how it looks like if you see your word embedding projected in 2D or 3D dimension here and the stimulant is here in the middle and you see all these compounds were close to stimulant. And from the trained word embedding model of science on this exercise, 15 new stimulants were found. In addition to that, mining EMM like a system, not the literature, but what is happening in media, we using text mining, we found 10 compounds were completely new stimulants. This work was just published I think this week and it's available online in open access for everyone if you are interested to have really the details of the approach from A to Z, not just a short one like I gave it now. And what I can to conclude, let's say, as mentioned by Hans and Nikos already AI and system approach are really needed, not only needed for us, we believe that it's a must in food safety, where you can include all these factors, economical, climate factors and social factors, technology factors, all the factors that are in the system. BN is one of the best approaches when you want to use system approach and AI, because you can explain it to expert in food safety, especially in food safety in a conservative area. You need interaction with the expert, you need to explain what is happening in the system. And always they have new questions, then you can play easily what if scenarios and ask strategy, operational and tactical questions in seconds and without going back to build any other model. In addition to that, using NLPs we succeeded to find a way, now we are developing this area also finding unknowns, we did it for stimulants and we are doing it for other food safety hazard. And the same technology also can be scaled up to any product or a topic or hazard easily. You need just to do the same exercise and retrain your models. In addition to that, we have like you need to later on if you want to make it in real time, that's you will have other issues related to your infrastructure, the power of computation, data governance, what you can share, what you cannot share and such kind of new challenges that are raising in food safety generally. Yes, again I started with a team, I will close my talk with the team. Thanks for all these guys who really added the big value to this work. And thanks for all of you for attending and looking for your questions and a nice discussion. Thank you Yamin, thank you. I have a couple of questions and while I will be asking a couple of questions I will give the time to our participants to submit their own in the chat. Let me see if I can share again my slides, perfect because my questions are around the practical steps or challenges in putting this in operation in daily operation. So what I hear you describing is first of all an essential step in involving the experts and their knowledge in identifying the key factors that can serve in a reliable way as potential data signals to consider in the model. So this was an essential step in what you are describing. I wonder how much time does it take to run such a study? If we want to look at a particular area, how much time do we need in order to decide on the key factors that we should then incorporate in our model? Shall I address that Yamin? Okay, I guess if I would estimate of the project budget 60% is used for this. This is the most important part of it. The modeling is and the validation is the least part. We also have investigated to what extent can we use the extract knowledge from the experts because when you have the Bayesian network, for example, you can also influence the direction of impact. I mean this is only the main impact that you see in the figure, but you can also manipulate in first the direction of impact. For example, if they think it is not logical. Often when we do that and the expert says, yeah, that's not logical. I would expect this and this and this and if you ask them, okay, do you have proof for that? No, got through it. When we do it, then all the time the model performance decreases significantly. We learn from that and we also have done together with social scientists also in the office study on it. The reason for it is the experts are really experts on a very narrow knowledge path. But in a questionnaire or in an interaction in an expert elicitation methodology, you ask much more of which they think they know, but not 100% sure. And when you have a let's say a complex interactions where these direct paths and the rules are not clear and are not published and well established, then you come into this domain and then we prefer to say, okay, we use the experts for let's say what things around out of the box things should be considered and and and let the data speak for them for itself. And because when we do it in that way, the models that we produce when of course with the data set that we use that we have has the highest prediction, but also it's it remains often very high the same level in the future data that we don't have. Let's say when we let's say when we do the same thing, try to predict with a model that is four years old with the data that is now, then we have we achieve more or less the same prediction accuracy, except if something really strange has happened like the COVID for example, that's something that you that is out of the ordinary that has an impact in the whole system that never has something similar in the past, then then you have something then you have a different situation. And how much time does this take? If you start from scratch and you want a month, you first have to identify a first you go the first thing we do is literature study. So checking the literature that costs a lot a couple of weeks, then you summarize and and then you have to identify at the same time you start at the same time identify the experts and in in the domain itself you contact them and you try to figure out which domains they think is relevant and you find experts in these domains then you have to draw up a questionnaire if you do the questionnaire and interviews also really good because then you have a more interactive but then of course you have fewer number of experts that you consult and Delphi is a different way to do it but that's more costly so but if it depends so much money you have available let's say for the consultation easily half a year. So it's half a year and some investment has to go in identifying the initial the starting factors and then what I hear you describing is that the data can come and complete the picture but we have a starting point by getting the the experts and the right experts in the process. Yeah I also wanted to yeah please I have another question that is also practical. Yeah but you need to know which methodology, expert elicitation you want to use in what purpose and where so you need an expert on that so I just not just sending out a question. I understood that this is clear that the social part the social science part is very important as and complementary to the computer science part and when we're looking down the road in terms of how far you can look at you talked about yes the future you talked about the real-time day-to-day updates of probabilities in terms of hazards in given product categories what did your model show in terms of some years months or years down the road how far can you go right now with the models that you have for sure a year what but we haven't checked all of them we have checked one that we developed on food fraud and the prediction what we did five years ago was a forecast of something that was going to happen in the in a specifically in China we made a forecast of China they also made a forecast of others but this case was in China and we supposed that something what's based on the let's say implementation of food safety regulation and so on that some of the parameters and the food safety control we would improve and that was I think six years ago now but we and we say okay supposing five years time this and this and this a situation is happening we would expect that this and this and this in the food fraud and we did the simulation I think last year with this data and it was exactly the same so the prediction was accurate so for the typical question of the largest organizations in the industry that look three to five years down the road you say that we can trust the models enough but it's something to further develop yeah I mean yes you see the question as you know in any modeling modeling is very important what do you want to do your model you cannot have a model that does everything or answer all questions that's not possible but if you want to integrate the operational and the tactical and the strategic level in your model you can do it by designing the beginning because sometimes like you see in all our models here we add a year even the expert says I am not interested in the year is not one of the factors but for us like an authority we know we need to prepare the sampling plan of next year and the year after then we need the year like parameter to play the yearly decisions to see what will happen in two years three years we have the weekly one we want to decide weekly then if you want to go to daily level which is like predicting price or whatever then you need to include those parameters in that sense you should know in addition I am predicting food safety I am going to use it for strategic questions let's say it depends on your horizon as Hans mentioned in food fraud for us it was five years we have it already the years were there then but if your strategy is one year then you need to include those parameters like not factors that influence the output but factors where you can play scenarios what if in that sense so here they should be in your model different models or different factors in different versions of the model in depending on the horizon that you want to cover the future horizon that you want to go yes and it depends on the complexity of the questions you always start with simple yes then if you don't don't say yeah the ideal scenario you will get my result from the first one of course but it can happen also that you will get poor results in that sense then you need to separate the tactical the operational and the strategy from each other then you can put them in sequence instead of having them all of them together it depends on the complexity but yes the bn is one of really in my in my life like a modeler is one of well it's a it's a big machine really yeah I can say we helped to implement it in other domains in Wageningen where we predict let's say yields we predict animal disease outbreaks and they are quite accurate well it's a bit baseless so this is the the view that we have in especially looking down the road in large organizations but there is also a question that is very very interesting from our audience and I'm going to the questions from our audience that talks about smaller organizations because what you were describing in terms of data collection training the model and putting the model in practice requires some resources what would you recommend to people working in smaller organizations like a small fast developing company which part should they implement or can they implement in-house and which part can they use from other sources from external services or from your services okay I can answer maybe a part of the question look there is different ways of building a patient network what we describe here is expert-based way where you consult with the expert you need to collect and there is data driven one 100% from data then if like a small company he wants to analyze his quality control data put them in the model to predict which ones to okay which okay I have a product coming from this supplier which quality aspect that they have to prioritize which ones to tackle first if you have the data there then it's straightforward that's clear because you don't need you are the quality expert and you have the data in-house then yes the being can be really quickly decided just concerned with your expert quickly you have the question and you are doing maybe modeling other techniques then you know the question already then that one is data driven automatically and quicker but if you go to complex questions where you have different factors you need expert interactions and combining both together I think also the data availability if you have your own data it is easier and combined with external and then it can be made much more rapid if you have daily data or yes and you can you can have also small models if okay if you are a small company it depends okay if you want to use it in quality check or suppliers before you import something or in your production line saying which default or I can have in the product then you are recording already your data then you can use just those parameters that you have in your daily data that you are collecting to predict and to become more efficient more targeting more everything better in that sense then it's straightforward good good okay thank you for the answer and I think this was the question that we have enough time to answer we had enough time to answer from the participants I see people saying that they would like to to contribute more and this is an excellent way to contribute which takes me to the call to action be part of the interest group feel free to register at the interest group so that we can have more opportunities for continuing the conversation more webinars exposure and testing and playing with systems and models like the one that you presented I would wrap up I want to thank both of you Yamin and Hans for this very very interesting set of presentations you you have put you have set some light in areas that were not clear to me as well the social science part was an important part that I didn't know of and that now I understood much better I also like a lot the way that you have used scientific literature to mine the unknown hazards and the upcoming hazard so to feed in the hazard taxonomy or catalog with new ones coming from scientific literature I don't know if you have any last words closing Hans it was another to present this and I hope it has inspired people and if they want to learn more please contact us I'm happy to respond thank you Yamin the same for me thanks Nikos and Ola Grono guys for inviting us and for all our participants for attending and for all your very nice questions let's hope to see you in another webinar or wherever where we can share our results and discuss about food safety thank you very much it was a pleasure and an honor to have you with us I want to thank all the participants enjoy the rest of the day or the starting day and enjoy the month as well and keep in touch and stay tuned for more things that will come thanks a lot thank you so much bye bye