 AI-based systems are increasingly making decisions for and on behalf of end-users, and therefore it is tremendously important for AI to be able to explain its reasoning and decision-making process. This kind of AI is accessible, just and fair, across different stakeholder groups. It is transparent about its capabilities and about the motivations of AI creators. The problem is that there is also untrustworthy AI that may miss some of the properties or in worst case scenario it could be willfully the opposite of trustworthy AI. This is very important to sort of build AI systems that are sort of working with humans closely in today's world because a lot of systems try to like make recommendations to people or try to make a routine task faster for human beings. Even though on past research data sets or on experimental results they perform superb like 98% accuracy but still in the real world when these systems are deployed they fail to maybe get the desired accuracy that we hope to achieve from these systems. We are designing interactive visualization tools that can help domain experts and end-users to understand the internal behavior of those AI models and so that the domain expert can understand what decisions these AI models are making that can have an impact at the life of end-users. The central problem I'm trying to investigate is this communication gap between machine learning technologies for healthcare and what the goals and needs of clinicians are. There is a trial period where in every member of the team tries to interact with this model and figures out how are the different ways in which it is working and they try to relate this performance of the model with their own practical knowledge or clinical knowledge. Now what happens is maybe the model will perform better in the short term but over long term it might lead to like very disastrous consequences that are on the patient side. This is why pathologists and radiologists are very skeptical when interacting with the new model. They might have positive experiences in the short term but over long term it might lead to more harm. These existing math-based explanations are not understandable to these domain experts, to these end-users because often they do not match their mental models or they might actually even be misleading for these stakeholder groups. When these come in the context of human interactions there are a lot of other sort of nuances that come into play like how do we perceive these recommendations? Do we really trust the AI systems? Do we really benefit from the recommendations in a specific context or is it context dependent or not? Our interactive tools actually hails domain experts to understand different scenarios from the model and the data and they can simulate different behaviors correlating different features. Your model can tell you this particular pixel is tumor or this is not tumor but can it actually show you how short it is while predicting tumor? Our future work is also looking into ways that we can create interactions that help end-users make these AI-based systems accountable.