.png)
Explainability is extremely crucial in AI systems that are responsible for carrying out mission-critical tasks. For example, in healthcare, if scientists are relying on AI models to help them figure out whether the patient will have cancer or not, they need to be 100% sure of their diagnoses otherwise this can result in death, a lot of lawsuits and a lot of damage done in trust. Nature is this problem is so intense that explainability sits at the core of this problem: the data scientists and the human operators, in this case, the doctors need to understand how the machine learning system is behaving and how did it come to a decision.
Explainable AI is also extremely important in financial services or FinTech in particular due to the growing adoption of machine learning solutions for credit scoring, loan approval, insurance, investment decisions and so on. Here again, there is a clear cost associated with implementing wrong decisions by the machine learning system: so there is a huge need to understand how the model actually works.
Using black-box AI increases business risk and exposes the businesses to a deep downside. The reason why black-box models are not desirable becomes more clear when we look at how the business functions as a whole:
For the business decision-maker, data scientists need to answer the question of why they can trust our models, for IT & Operation, data scientists need to tell them how can they monitor and debug if an error occurs, for the data scientist, they need to know how they can further improve the accuracy of their models and finally, for regulators and auditors, they need to be able to get an answer to whether our AI system is fair or not?
Explainable AI aims at providing clear and transparent predictions - an end-to-end system that translates complicated reasoning into simple language and relies on automated feedback to constantly improve the AI system. Remember, xAI is highly driven by the feedback so is a two-way interaction between the human and the AI system.
In the end, we should understand why behind the model, the impact of the model, where the model fails and what recommendations to provide.
explainx is an open-source explainable AI platform created by explainX.ai. Explainx aims to help data scientists explain, monitor and debug black-box AI models — the purpose is to enable companies to build robust, unbiased and transparent AI solutions.
Within the explainx architecture, explainx provides access to state of the art interpretability techniques in just a single line of code within our Jupyter notebook or IDE.
For this example, we will use the HELOC dataset provided by FICO. The customers in this dataset have requested a credit line in the range of USD 5,000–150,000. Our job is to predict RiskPerformance: whether they will make timely payments over a two year period. The prediction can then be used to decide whether the homeowner qualifies for a line of credit.
For this example, we will train CatBoost Classifier model. After the training is done, we will use the explainx xAI module to explain our model and build a narrative for business stakeholders to understand! You can also follow the tutorial by simply copying the code in your own notebook!
Let’s start with opening up our Jupyter notebooks and installing the explainx library.
You can also clone the repository from the link: https://github.com/explainX/explainx
So let's begin!
Note: If you want to view it inline, simple pass mode="inline” argument into the explainx function.
For this tutorial, we will not go in the nitty-gritty of model building, model metrics and evaluation. Instead, we will dive right into the explaining AI powered decisions and understand how models behave. So let’s start opening up the black-box!
In explainX, we can explain the model on four levels:
1. Global Level Explanation
2. Local Prediction Explanation
3. Scenario Analysis
4. Feature Interaction & Distributions
Let's use the overall feature importance and overall feature impact graph to identify top/bottom performing features.
Interpretation:
This tells that according to the CatBoost model, ExternalRiskEstimate, MSinceMostRecentInq and PercentTradesNeverDelqare the top three variables with the largest impact on RiskPerformance. These three represents Risk Estimate, Credit Inquiries and Debt Level information: extremely important categories when assessing risk.
This information gives us a general idea of feature contribution but to understand whether each of these features has a positive or a negative impact on RiskPerformance we need to look at the feature impact graph.
Interpretation:
This graph gives us even more insight into the model logic. We can clearly observe that ExternalRiskEstimate impacts the predicting variable positively — pushes it towards “Good Credit Risk Performance” and that matches our intuition as well. We have to use a little bit of our domain knowledge in finance for this: ExternalRiskEstimate is a consolidated version of some risk markers (higher is better) so automatically we learn that this variable will always positively affect the prediction. Then we have NumSatisfactoryTrades the number of “satisfactory” accounts (“trades”) has a significant positive effect on the predicted probability of good credit.
However, in the lower end, delinquency variables push the overall prediction towards 0 (in this case, application denied). This is interesting and we can even dig deeper and see when does this negative effect of delinquency wears off? (Something to try on your own!)
Now as we have an idea of how each feature/variable affects the overall decision, we can move on to explaining the decision for a specific customer. For that, we will be using an impact graph or a decision plot that will help us get attribution scores for that particular prediction. To further understand model behavior, we will also find similar customers that had a similar profile and decision to further explain the model.
So let’s explain the decision for customer # 9 in our data. For this specific customer, the application was Approved because the RiskPerformance was “Good” and our model classified it correctly!
Interpretation:
So this graph clearly shows the top three positive and top three negatively impacting variables. According to the model, MSinceMostRecentInq had the most positive impact on the variable. So this tells us that the higher value of this variable means that there is no penalty for having more than one month since the most recent inquiry. Then we have ExternalRiskEstimate that again plays a positive role in pushing the prediction towards “good credit behavior”. However, PercentTradesNeverDelq affected the prediction negatively: this might be the case if the value of this variable is extremely small because the smaller value for this variable lowers the probability of good credit score - and we can confirm this by looking at the data.
To keep it short, these findings match our mental-models as the contribution or impact score of each variable is correctly assigned. To further explain the model and support our analysis, we will find similar customers!
ExplainX comes with an in-built prototypical analysis module which provides a much more well rounded and comprehensive view of why the decision for the applicant may be justifiable by identifying similar customers!
The above table depicts the five closest customers to the chosen applicant. Based on importance weight assigned to each profile by the method, we see that the prototype under column zero is the most representative user profile by far. This is (intuitively) confirmed from the feature similarity where more than 50% of the features (12 out of 23) of this prototype are identical to that of the chosen user whose prediction we want to explain. Also, the bank employee looking at the prototypical users and their features surmise that the approved applicant belongs to a group with high values of ExternalRiskEstimate. This justification gives the bank employees more confidence in approving the user’s application.
Now, let’s explore different scenarios and see how the model performs - we can use scenario analysis to identify biases within our model. It might be the case where for specific group of customers, the model is behaving differently. We can apply the data filters within the data table (no need to write SQL queries to filter your data) and explain multiple instances and scenarios very easily.
This is extremely useful when you are trying to understand behavior on a specific cluster or group of data. For example, we want to see how the model behaved when the ExternalRiskEstimate was > 60 and MSinceOldestTradeOpen was greater than 200.
Interpretation:
When the ExternalRiskEstimate is greater than 60, it is seen as a positive sign and most of the applications were approved which matches our internal mental model as well: ExternalRiskEstimate is actually monotonically decreasing which means that as the value of ExternalRiskEstimate increases, the probability of Bad Credit Score decreases! So in this cluster where the ExternalRiskEstimate is greater than 60, we will have more approved applications and customers who were extended the credit line. We can confirm this by visualizing the marginal effect of changing ExternalRiskEstimate has on the model decision.
We can clearly see that pattern that as the value of ExternalRiskEstimate increases, the model is more likely to approve the application of the customer. We see more concentration of red dots as the ExternalRiskEstimate value goes up and that makes a lot of sense! Let's not forget that we have to look at other variables as well to learn more but now, with explainX, we finally have an in-depth insight into how the model is actually behaving.
Data Scientists can easily use explainX to further explore patterns by looking at interactions between different variables and how they impact the overall decision of the model. So let’s summarize our learning for a business user to understand:
I hope you all enjoyed this case study. Explainability is extremely crucial and more than relevant today — so the ability to present a narrative that shows your understanding of how AI works is a vital skill for a data scientist. This is the real essence of human-AI understanding and democratization of AI.
