 So, thank you very much. Thank you all for being here this afternoon. First, let me apologize because my colleague, José Alonso, wasn't able to attend this conference, so it's my duty to do it today this afternoon. So, the title is about explaining the unexplainable AI. Basically, what we will do is to focus on this aspect of explainable artificial intelligence, which is now currently being one of the most relevant topics in research on ocean applications. We will focus on this aspect of using natural language generation for trying to explain these some models or some consequences or some tasks that AI systems do. Our research center, the CITUS in Santiago Compostela, has a number of scientific programs related with AI, particularly one related with natural language generation and in general natural language technologies. So, we are working in projects and doing research in this topic, mostly in the area of explainability. So, first of all, in order to start just a few examples or a few short motivations about this idea of this need of endowing with explainability our AI systems. There are a lot of dimensions related with explainability. I will just overview a few of them. For example, we can consider the capability of the fairness in our systems. Fairness is defined as opposite to having bias or preferences on some examples, on some communities, on some users. Also, AI systems have to be robust or should be robust in the sense that should be reliable, should be safe, should be consistent in their operation. Also, the characteristic of explainability is related with making the systems intelligible so that users or people that suffer the consequences of these systems have to understand how they operate or how these outcomes of these systems are produced and why are produced. And also, it's very important the line edge. So, this is a long cycle feature because it's involving every aspect of the design of the implementation previously on the definition of the systems, also maintenance, evolution. So, it's the need of having all this trustability of the operation of these systems. To some extent, some people are relating this idea of explainability or, in general, trustworthy AI with the general idea of quality. This means that we need to assure the quality of our systems. We need to validate. We need to test them properly. And this is a key issue in some of the applications we will overview today. Particularly, when machine learning or in general learning techniques are in this system, what happens with this evolution, with the change these systems are having. So, which is the trace we can follow from the initial system to the final system when this is evolving and changing through the time. Some examples of these features of AI. Fairness. There are a number of examples we could use for showing how BIAS is introduced in many AI systems. This is an example by Ricardo Baez and Karma Pero. Any of you can try with your world translator or similar tools. In this case, what we have is a translation from English to Turkish language. We have to keep in mind that Turkish language does not have gender in the pronouns. So, when we translate, for example, she is an art or she is the first person in her country to win an over prize, the translation into Turkish is missing this gender feature of the pronouns. But when we do the reverse translation and get back to English, the translation has a certain BIAS which is due to the data probably that was training in this translator. In the first one, we have she is a nurse, but in the second one, the translation, we produce he is the first Nobel prize in his country. So, when it comes to decide the gender of the output in this translation, part of the BIAS that we had in the data that were used to train the translator are represented in this outcome. There are a number of similar examples. This one is also very recent, it's in the area of healthcare. In that case, there is a company opting with an algorithm that tried to estimate who should receive extra medical care The point here is that they tried to provide this extra medical care to people that in the future will have more cost to the system. So they tried to anticipate these extra costs by preventing or by producing or endowing these people with this extra medical care. So the point is that because of socio-economic indicators, it happens that black people have lower cost in the healthcare. So the outcome at the end is that for similar people with similar health conditions there is a bias against black people that favors white people in the contrary. So in this case, there is not explicitly raised as a variable in the system. What happens is that this future cost for each patient has implicitly this information inside it. So here is not only a matter of data but also it has to do with the way that algorithms are able to select the variables and to produce the models that finally are used for making the estimations. Typically they are the following gaining information criteria and in this case, this gaining information is inheriting somehow some features which are hidden in the data and producing these outcomes at the end. Also in the medical domain we could have similar examples but let's go to the second feature, robustness in our A.I. system should be reliable, should be safe. Here we have a couple of very well known examples in the areas of classifying images and also with different models and with different approaches. In the second case, in the right we have that the system is able to classify correctly the images in the left but when they are disturbed when they are perturbed with this pattern in the middle images to our eyes are still very similar and should be classified with no problem by the systems but in the case of using a neural approach which is the case for that problem, all the features in the other images in the right are classified broadly in this case all of them are classified as raw streets. It's important that in this topic of images we can relate this idea of being robust, of being safe even with the idea of hacking because it's interesting to see the results of this study. It's also a very recent study that was conducted with images of city lungs and in that case what they did was to introduce artificially fake cancerous nodes and this was able to mislead the diagnosis by the experts by the radiologists. You can see that 90% of the time the diagnosed cancer when one of these fake nodes were introduced in inmates and on the contrary when they erased a real nodule the diagnosis was healthy in 94% of the cases. So we have that at the end the idea of having safe reliable systems is important and we need to build our systems with these features with the characteristics. What's important for today which is the explainability feature making our systems intelligent to understand what's the operation of the systems, what's the outcome of the systems. You know we have a lot of different types of models we have neural ones, we have graphical, we have decision trees it seems that explainability is of course easier for the graphical or typical graphically represented models like decision trees for example they are on the right but when we increase the accuracy and we go into the black box models like neural networks we have a lot of difference but explainability is also at the end we have the classical trade-off between explainability between interpretability and precision is now revisited with the eruption of neural networks of deep neural networks in particular and huge amount of new services they are providing to us. So here we have a very relevant or key debate the debate here is we should put the balance on what, on or why why is related with the strategy to know the problems that our models can have, what is related with immediate actions after a given model produces an outcome so at the end we will have to balance between accuracy or knowledge about the processes because it seems evident that the models that have less accuracy in general can be more interpretable and the contrary the models with high accuracy do not have this capability. Related with traces or how to trace from the beginning of one AI system to the, during its life how it's operating how it's behaving it's also a relevant fundamental question here these are the two well known examples of car accidents involving casualties and it's not also the point that when we have these problems and in these cases we have some real abilities and responsibility legal responsibility of these accidents has to be given to the car designer to the algorithm designer to the people that provided the data the test data is the testing or the validation strong enough in order to account for the use of these systems that will last for many years in very different conditions of operation that the conditions that they had when they were trained and where they were tested so where's the responsibility here when there is a failure in these kind of systems related also with trustworthy it's interesting to note that we don't need to explain, to provide explainability or to make trustworthy systems to all the users of these systems we have to do that mostly with the people that is that are responsible for their operation because there is a certain transitivity property with trustworthy you have an example of of a flight of people flying in that are confident in the pilot and the pilot is confident in the autopilot so at the end the people in the in the plane is also confident in the autopilot not because they have the experience of using that but they are confident on the person that has confidence on that and is a professional that they trust on many other dimensions in this problem of explainability of trustworthy we have also the destrutability so we should also endow their systems with the capacity that any user or professional or specific users are able to know what's going wrong in these models in order to explain what's happening with them and so we have a need, a real need for endowing the systems with these features, with these characteristics but it's not only a matter of having fairness or having accountability or having transparency in our algorithms, in our systems we have also legal or normative frameworks that are becoming more and more important in our days do you all know that a couple of years ago, not a couple but almost a couple of years that the GDPR was informed in May 2018 and there are a number of very interesting I will say sentences in the articles of this law for example you can see there that any user should have the right not to being to support a decision based on automatically processing this includes profiling another tasks but the point here is that at a given point any user should be should have the right of having human intervention in the process so if a decision is taken only on automatic processing of data and of course if this is being done with models that are difficult to understand or that anyone is able to understand how they operate this right to the explanation that every citizen in Europe has can be affected by this lack of knowledge about our systems in particular it's also very interesting to look at the recital a part of them in this law you can see that for example there is even a certain legal obligation of minimizing errors or the risk of errors also to avoid discriminatory effects so it's not just a matter of well operating of a system but also a normative or a legal existence of this kind of systems actually there are some interpretations of that that with these regulations all citizens have a certain right to the explanation from automatically taking decisions so in any corporation in any government whatever it is the service that is taking or doing or perform only with automatic intervention with artificial intelligence at some point the users in particular the citizens will need the capability of being explained what the decisions are taking and how even more recently the European Parliament has published a governance framework for accountability and transparency and it's also very interesting to look I have highlighted some of the sentences I think are of more interest but in particular we can see that they even say that let's say technical reasons for this framework state that it's very unlikely that explaining the steps of an algorithm making a detailed explanation step by step of how the algorithm performs this will be informative enough for users it seems that they have the preference of explanations that are related with the general behavior of the systems it seems that for them this is more feasible than doing the opposite in the technical point of view both things can be combined and at the end may be a good explanation of the complete behavior of the system should be impart at least based on how the algorithm that is implemented in the system is working and how its steps are being taken for producing a given outcome there is also a third reason let's say implementation and it's also the real need that we have in many applications let me show this is a photo of our campus in Santiago this is one of the garden areas we have you see also the sidewalks the roads you see the grass and the trees and we can see there these white lines that are the footpaths that people walking was at the end when they tried to move around the campus so the designer thought that the sidewalks would be enough for ensuring mobility but real people was optimizes the trajectories and is able to do a different process that the process that was the design that was made by the technical people so this has to do a lot with an relevant area in AI which is business process managing in particular a process mining which is also related with big data and has enormous impact in explainability to sandwich things business processes are related with defining a number of tasks for providing a service or a product this task should be performed by persons or by by any system this is a simple example it could be an e-commerce or a logistic chain doing a number of tasks in order to produce a final service for the users this is the design process but reality at the end happens this way most of these not the systems in general but also the people involved is able to find different alternatives for doing the same tasks and even in this example we can see that for example a key issue like quality check could be avoided by doing a different a different path in the process and also what's also relevant here is that some loops can exist we can go from activity to another and back and forth and back so in this case it's related with inefficiencies so at the end different we have even very simple process the learning or the mining of this process or the definition of this process could be like this one you have here this is a real process mined from data in this case is related with a health service of a hospital this is what in the literature of process mining they call spaghetti processes the reason is clear because of the shape of the representation graphical representations are the usual way of combining information in this area of BPM so it's evident also that in this case with this representation no real information no explanations no useful knowledge can be obtained so it's important to have some analysis or to have some extra tools that go deep inside this process and provide relevant information to the users this can be done with analytics in particular with business process analytics and for example with these tools we can obtain cycles like the one you can see over there which is our petition of different tasks that probably are showing some inefficiencies in the systems but graphic representation for these cases even for simple cases is not enough for example in this case which is a process in a hospital doctors in the hospital are not able to take to understand any relevant or useful information from that representation so in that cases information could be enhanced with natural language description like the report you can see below the graphic where emphasis can be put not only in the process itself but in some of the indicators some metrics that can be obtained from the logs of the system but also at the end of the report we can also explain what's relevant about the process for example this repetition we mentioned it before so in many cases the professional for example in this case cardiologist demand not only the graphic representation but also natural language written or talked explanation of the visualization so how to explain AI we have graphic representations as we mentioned before the language we have a number of different options the underlying idea about all this problem is trying to explain this explainable artificial intelligence typically the black box is algorithms in particular network and similar so the paradigm here is how to open this black box in order to show what we have inside this interest from explainability mostly arrived when the agents in the United States started a program like three years ago this program is interesting that this was one of the motivating examples they used for this launching this idea of explainability you can see in this case that the classification this is a very simple classification problem in this case this is a cat but explanations here are pointing to both to two different ways of providing the explanations we have a written explanation which is this a cat because it has fair whiskers and claws but also we have a graphic representation sometimes providing similar examples of particular features that are relevant not in language, not in written language but in a photo or selecting one part of the image that is relevant for the explanation this is also a convenient way of providing the explanation the general idea is to endow the systems with the models or the blocks that are able to provide these explanations and the user should decide at the end not only based on the outcome systems but also in the explanation of how this outcome was obtained this DARPA project is still alive actually it will end by mid 2021 so we are seeing from year to year many of the results in particular in these last years they are doing the part of the assessment and comparative evaluations it's a project that is focusing on many many aspects of the problem not only the AI related problem which could be the models of course they are doing research and in general the community is doing research on trying to explain the models but also we need to build a new type of interfaces that are able to take advantage of this new information we have to provide so we have also challenges related with how these new interfaces should be and what's also very important is not only a matter of computer science or artificial intelligence but when it comes to conveying information with convincing with explaining there is a lot of information and science that come from the social sciences, psychologists, philosophy, logic so this aspect of the other problem should also be taking into account not only consider this from a computer science or from an engineer and point of view so it's very interesting to look to the people that has been doing science about these topics for many years and try to use how do they do things and bring them back to our models in particularly I selected here two examples one is important maybe when it comes to natural language explanations it's not a matter of just producing the explanation but also have some kind of interaction that helps the user to focus with more detail in the part of the information that needs maybe we can provide just an initial simple explanation but after that this should be refined according to the real needs from the user but also argumentation is important not only how do we link different how we reason and how we combine arguments in order to be convincing with our explanations a couple of examples of how these explanations are built from the area of AI an example that's related with images in particular the approach that they did here was to look at morphological or patterns at simple features features that every human would in general be able to detect or to understand and trying to build the explanations based on these features if we go to technical details for example for the biology of this bird it will be impossible that any of us which is understanding especially could understand the information about why this bird is from one type or of another so at the end what they do is to focus on these features and try to answer if these features are present or not in the examples in the animals in this case and at the end the explanations will be a number of these features that are the information which was taken by the system for doing the classification there are also some other lines of research in this area there are models that try to provide directly an explanation of that directly an approach is related with trying to understand the model itself is this idea that for example the European Council is not very fun of it and there are also a different approach which is being produced now very interesting results which is trying to provide to use transparent models at least for explaining part of the behavior of a complex of black box systems of course we can always do reverse engineering we can just take the input outputs of one of these black box systems and train with this data a new system which is interpretable and in this way to use this information for trying to explain how the black box system performs two examples of this idea of systems that provide information locally from the black box classifiers or from the black box predictors one of the most recent one is this line approach which at the end does not provide a real explanation in language but provide evidence is that the user, the final user is able to understand and that may be convincing for him or for her in these examples we have a number of features for classifying making diagnosis of one person of having flu the idea with the local interpretable model and the explainer is that they focus on the most relevant features, some of them provide evidences against, some of them provide evidences in favor of the prediction and this information is provided to the user which is finally who takes the decision based on this additional information the way that this model operates is trying to look for similar cases to the one that has to be explained some of them will belong we have a similar prediction than the one that has to be explained another will have different different predictions but with all this information about favorable favorable favorable cases and cases in contrary the final user is able to make the decision it's not only a matter of using words this is a general approach this can be used also for images and in this case what they do is to provide these pieces of evidences that support the outcome for example if we have to classify or to say do we have a guitar or a labrador dog or different patterns in a scene with these models they select the part of this thing that is related with explanation in this case we can see for example the the first case electric guitar the neck of the guitar of course this is not an electric guitar this is a Spanish or acoustic guitar but since the neck is similar there is a certain evidence that this could be an electric guitar but you can see that the part of the image that is selected for supporting acoustic guitar and labrador are very similar so there are a lot of projects of initiatives in Europe related with this also in Spain but I would like to go to the idea of how can we use natural language in particular natural language natural language generation for combining with explanations first of all natural language generation is one part of what is commonly called natural language processing typically natural language processing is actually the name of a complete topic is related with producing automatically text or content from other sources or information that could be data that could be data numerical data or different of different services do we need this natural language generation there are some studies and some perspectives that say yes and we can see that some of the areas some of the fields of application of an LG are nowadays being very very relevant for example the conversational systems all the assistants are tools that are we using commonly very in daily lives it's important to record here that an LG is a part of them there are a number of different areas in particularly the data to text area which is able to produce explanations or descriptions of information that is given in numerical data sets when it comes to the particularly too big data typically the way of conveying information is the visualization so we can have doubts if we need this other area of providing information which is an LG there are we can use many examples of how graphic representations can be misleading this is a very well known example in the literature here we have a representation about how the number of murders evolved after in Florida they passed a law that allowed users to people to use their firearms so here what happens is that the representation of the data is not following the usual standard that the vertical axis has zero below and upper values in the vertical direction but if they did it in the reverse way so what seems here that there is a distance of this number of casualties of victims actually represented that was an increase so it can be in many aspects being misleading to have graphic representation because the text at the end is a matter of the user so here what we do in LG is to process the data basically this is the content determination state is to analyze the data and distract all the potentially useful information this information is related about what we have to say in the final text and after that we have two stages related with how to organize the text in order to produce text corrects fluents and that are understandable but what's important in this context is that this energy approach can be combined with any other model we talked before that the line model was an anostic model for trying to provide explanations and LG is also an anostic model because you can use it and combine it or integrate it with any other system that provides the information and at the end what's important for doing the narratives of the text is to have the analytics of the data the content determination this is related with the problem so the rest of the pipeline can be integrated with any other application well in this area there are a number of examples we could use but let me just finish with a very simple one where we relate this energy with explanation this is one of the tools we have developed it's related with explaining classifiers this is a real life example we could use is having to classify VRs in this case using some attributes if you do that with decision tree like this one which is itself one of these explainable models we can also integrate with us a written explanation that is related with which was the path that produces the classification in this case so at the end you can support the classification by explaining which were the decisions in the decision tree that were taken for the classification or for giving the outcome of the system this of course can be integrated not only with decision trees but also for example with faster decision trees or with faster knowledge based systems like this other one which is very well known in the community called Fourier so in this context there is a very strong research movement in order to produce not only explainability but also explainably related with natural language in particularly we are launching now a European project and ITN which is specifically focusing on how to combine explainability with natural language generation in particular in this context we are focusing on how to explain very different models black boxes, grey boxes which is for example fast rule based and similar ones so just for finishing the key idea here is that with energy we can combine we can use that for explaining different models different outcomes different ways of representing the explanations graphical, numerical or other or in other way so that's it thank you very much just let me finish with a kind of invitation because we will host next year the most relevant scientific conference in Europe related with years and I'm delighted to come to Santiago to participate in it so thank you very much sorry I had a question first of all thank you very much it was a fantastic talk I I was wondering because I used to work in the finance industry a couple of years ago and one of the things we were doing to explain our models was actually just build I wouldn't say a more dumb model in parallel sort of using that and the features that you get from it to say well look this is the path that our explainable model follows so therefore to a certain degree we can say that the advanced model must probably take the same path I suppose the question that I'm trying to get to with this is what is the degree of explainability that you need to achieve because there needs to be some sort of a threshold to say well this is good enough this explanation is good enough how do you decide on what that threshold is I don't know if I'm clear on this this is an important point because how to validate the quality or to assess the quality of the explanations is hot topic now in research because you cannot do a general explanation which is satisfactory to everyone it's always very related with how to consume this information or this explanation so at the end what we did in that case is to validate that by trying to in two ways there is intrinsic evaluations which can be done if we have for example corpuses of explanations that can with some metrics like typically they do in PL in natural language processing and using that metrics you can see how your explanation matches intrinsic and also there is the most challenging and demanding way of evaluating is doing the intrinsic which is just asking a representative number of people that should be similar profiles that the one that has to consume information to evaluate the quality of the explanations quality in terms of linguistic quality also in terms of the contents of this information if this is relevant or not it's focused so there are a lot of dimensions you have to put in this question as you post to that people thank you thank you very much for the talk it was great I read something in the astrath about how to explain black box models such as deep learning or neural networks by means of grey models or something like that can you explain this the idea here is not only grey but also white box models are the ones that are more interpretable grey models are some between for example you have a rule based model this is interpretable but if this model includes uncertain information as fuzzy expressions this is what we call a grey model so any of those can be used to explain partially one black box model in a similar way of example I was before locally you can try to match parts of the classification that the black box model does with a white one or a grey one that has a similar behaviour for that specific area of the space and in that case you can use deformation from that explainable models for providing an explanation for the black black box model but not related specifically with energy there is also an important way or a relevant way of doing that is for example when it comes to images you can provide examples you can provide similar cases to the one that you are trying to predict in order to support your explanation so you don't need to go inside how the model is working but to provide similar situations and in this way parallelism between your real case and similar cases you can more or less understand how the classification is done with knowing nothing about the insights or how the model works because in this deep neural network most of course it's almost impossible to understand how things are going on how this is working inside in order to provide information from that so at the end this is a matter of comparison between examples thank you okay hi thank you for the talk because for me it was really interesting because I'm working in the healthcare sector so this interpretability of models is especially important because the decisions of the model are particularly critical so it was really interesting I just wanted to put another side of the topic so to speak to know your opinion because I think it's important also to stress and I think it's not being stressed enough the difference between extracting interpretation of one model and extracting is not the same it's not the same that a factor is important for the model to make a prediction and to say that this factor was the cause of the outcome so I think it's not being stressed enough and in particular in the healthcare sector it's causing some confusion I fully agree with that actually in this final project I mentioned this ITM one of the topics that we'll be working on is to explain Bayesian causal models for example because there was some work in this aspect many many years ago after that it was abandoned let's say but by now there are not relevant results on how to explain for example causal models this is an important topic to work on now yes thank you