Downloaded 50 times
![Example: Regression (Ames Housing dataset)
Model: XGBoost
Test: 0.16332
Validation : 0.13830
Results (metric is Root Mean Squared Error of Log (RMSEL) )
Kaggle notebook “Stacked Regressions : Top 4% on LeaderBoard”
Cf: Score with the same method (XGBoost), but with feature engineering: 0.11420
This time, it is much better than simple baseline.
Cf: Baseline (linear regression in terms of
OverallQual): 1.1608
It is discretized
because OverallQual is
[1,…,10].
Prediction/actual value for validation data](https://image.slidesharecdn.com/explainableaislideshare-200711110451/75/Explainable-AI-26-2048.jpg)
Uploaded byArithmer Inc.
Explainable AI
The document discusses explainable AI (XAI), focusing on the importance of making AI interpretable to foster trust and improve model selection. It reviews methods like LIME and SHAP for explaining predictions in various machine learning tasks, including classification and regression, as well as image classification techniques using attention branch networks (ABN). Recent trends in XAI research highlight the increasing interest in developing models that provide clear rationale for their predictions.
More Related Content
![[ppt]](https://cdn.slidesharecdn.com/ss_thumbnails/ppt3441-thumbnail.jpg?width=640&height=640&fit=bounds)
![[ppt]](https://cdn.slidesharecdn.com/ss_thumbnails/ppt2931-thumbnail.jpg?width=640&height=640&fit=bounds)
Explainable AI
- 1. 2 0 20 / 0 7 / 1 1 Explainable AI (XAI) Daisuke Sato Reference (Survey on XAI) • A. Adadi, M. Berrada, https://ieeexplore.ieee.org/document/84665 90/ • Finale Doshi-Velez, Been Kim, arXiv:1702.08608 Codes Notebook in Kaggle by Daisuke Sato • “Random forest/XGBoost, and LIME/SHAP with Titanic” • “Predict Housing price”
- 2. Self-Introduction • Daisuke Sato,Ph.D. • Graduate school • Kyoto university • Research topic: Theoretical Physics (Quark-Gluon Plasma) • Previous jobs • Postdoc (US., Italy, Germany) • Current position • CTO
- 3. Contents • Concept/Motivation • Recenttrends on XAI • Method 1: LIME/SHAP • Example: Classification • Example: Regression • Example: Image classification • Method 2: ABN for image classification
- 4. Concept/Motivation We want AIto be explainable because… Generally speaking, AI is a blackbox. 1. Users should trust AI to actually use it (prediction itself, or model) Ex: diagnosis/medical check, credit screening People want to know why they were rejected by AI screening, and what they should do in order to pass the screening. G. Tolomei, et. al., arXiv:1706.06691
- 5. 2. It helpsto choose a model from some candidates Classifier of text to “Christianity” or “Atheism” (無神論) Green: Christianity Magenta: Atheism Both model give correct classification, but it is apparent that model 1 is better than model 2. Concept/Motivation
- 6. Cf: Famous exampleof “husky or wolf” 3. It is useful to find overfitting, when train data is different from test data Training dataset contains pictures of wolfs with snowy background. Then, the classifier trained on that dataset outputs “wolf” if the input image contains snow. Concept/Motivation
- 7. Contents • Concept/Motivation • Recenttrends on XAI • Method 1: LIME/SHAP • Example: Classification • Example: Regression • Example: Image classification • Method 2: ABN for image classification
- 8. Recent trends # ofpapers which includes one of explanation-related words (“intelligible”, “interpretable”,…) AND one of AI-related words (“Machine learning”, “deep learning”,…) FROM 7 repositories (arXiv, Google scholar, …) Recently, researchers are studying XAI more and more.
- 9. Contents • Concept/Motivation • Recenttrends on XAI • Method 1: LIME/SHAP • Example: Classification • Example: Regression • Example: Image classification • Method 2: ABN for image classification
- 10. Method 1: LocalInterpretable Model-agnostic Explanations (LIME) Objects: Classifier or Regressor M. Ribeiro, S. Singh, C. Guestrin, arXiv:1602.04938 Basic idea: Approximate original ML model with interpretable model (linear model/DT), in the vicinity of specific features. 𝑥: specific input Interpretable model
- 11. prediction 𝑥 ∈ ℝ 𝑑 features ℝ 𝑓(𝑥) ∈ 𝑓 Original model prediction 𝑧′ ∈ {0,1}𝑑′ Absence/presenc e of features ℝ 𝑔(𝑧′) ∈ 𝑔 Interpretable model Weights of sampling 𝜋 𝑥 𝑧 = exp(−𝐷(𝑥, 𝑥′)2/𝜎2) Linear model Complexity of model (depth of threes for DT/ # of non-zero weights for linear model) Distance of predictions of original model (f) and its approximation (g) Method 1: Local Interpretable Model-agnostic Explanations (LIME)
- 12. Explaining individual explanation 1.Original model predicts from features (sneeze, headache,…) whether the patient is flu or not. 2. LIME approximates the model with linear model in the vicinity of the specific patient. 3. The weights of the linear model for each feature give “explanation” of the prediction
- 13. desirable features ofLIME • Interpretable • Local fidelity • Model-agnostic (Original model is not affected by LIME at all) • Global perspective (sample different inputs and its predictions)
- 14. Shapley Additive exPlanations(SHAP) Actually, they are utilizing concepts of cooperative game theory: • Shapley regression values • Shapley sampling values • Quantitative Input Influence S. Lundberg, S-I. Lee, arXiv:1705.07874 Generalization of methods for XAI, • LIME • DeepLIFT • Layer-Wise Relevance Propagation A. Shrikumar et. al., arXiv:1605.01713 Sebastian Bach et al. In: PloS One 10.7 (2015), e0130140
- 15. Example: Classification (Titanicdataset) Variable Definition Key survival Survival 0 = No, 1 = Yes pclass Ticket class 1 = 1st, 2 = 2nd, 3 = 3rd sex Sex Age Age in years sibsp # of siblings / spouses aboard the Titanic parch # of parents / children aboard the Titanic ticket Ticket number fare Passenger fare cabin Cabin number embarked Port of Embarkation C = Cherbourg, Q = Queenstown, S = Southampton Label Features # of training samples = 891 # of test samples = 418 All values are unique 77% are NaN →Train : Validation = 8 : 2
- 16. GBDT Decision Tree (DT) Selectionof features and its threshold is done, so that the mutual information is maximized. Entropy at a node is maximized when the samples in that node are uniformly distributed to all the classes. →Child nodes tends to be purely distributed Mutual information: 𝐼 𝐻 𝐷 𝑝 − 𝐼 𝐻 𝐷𝑙𝑒𝑓𝑡 − 𝐼 𝐻 𝐷𝑟𝑖𝑔ℎ𝑡 Entropy in parent node Entropy in child nodes “Boyness” value • Simple to understand and interpret • Can represent interaction among features unlike linear model (at least up to # of depth_tree) • No need to NaN preprocessing/standardization • Tends to overlearn (if trees are deep)
- 17. GBDT Gradient Boosting DecisionTree (GBDT) T. Chen, C. Guestrin, arXiv:1603.02754 In ensemble method, the final value is calculated from average of many trees. How to decide the structure of tree ensemble (forest): 1. Start with one tree 2. Add another tree so that the loss function is minimized (gradient boost) Cf: In random forest, all the trees are generated in parallel, at once. Compared with it, GBDT can learn from the misprediction in the previous tree, so the result is expected to be better. Very popular method in Kaggle • Overlearning problem is improved • Less interpretability • Many hyperparameters to be tuned
- 18. GBDT I used oneof GBDT (XGBoost) as a ML model, under the following conditions: • No standardization of numerical features (as DT does not need it) • No postprocessing of NaN (GBDT treats NaN as it is) • No feature engineering • Hyperparameter tuning for n_estimators and max_depth (optuna is excellent) Not satisfying… Cf: Baseline (all women survive, all men die): 0.76555 Test score: 0.77033 Best parameters: {'n_estimators': 20, 'max_depth': 12} Validation score: 0.8659217877094972 Results: Review of know-how on feature engineering by experts: Kaggle notebook “How am I doing with my score?” Cf: Reported best score with ensemble method: 0.84210
- 19. GBDT Cf: Test scoreat Kaggle competition Cheaters… (Memorizing all the names of the survivors? Repeat trying with GA?)
- 20. GBDT Linear correlation betweenfeatures and label Fare has large correlation with Pclass, as you can easily expect (~0.4-0.6). Also with Embarked_C (~0.27), which I do not know why. Sex_female 0.543351 Pclass_3 0.322308 Pclass_1 0.285904 Fare 0.257307 Embarked_C 0.168240 Embarked_S 0.155660 Correlation between Survived and features The method which you try first, to select important features. You can work when the number of features is small. It is likely that the model trained with this data put large importance on the features with large correlation. (explanation of data itself, not model)
- 21. GBDT Importance of featuresin GBDT (explanation of whole model, not for specific sample) Women had high priority for getting on boats Passengers in 3rd class are not likely to survive. (Sign of feature’s contribution can not be read, only the magnitude) The average gain (for mutual information, or inpurity) of splits which use the feature Top 3 agrees with that in linear correlation.
- 22. GBDT However, this methodhas a problem of “inconsistency” (when a model is changed such that a feature has a higher impact on the output, the importance of that feature can decrease) This problem is overcome by LIME/SHAP.
- 23. LIME (explanation of modelprediction for specific sample) Explanation by LIME https://qiita.com/fufufukakaka/items/d0081cd38251d22ffebf My code did not work on Kaggle kernel, because of a bug in LIME package… So, here I quote results from other person. As LIME approximates the model with linear function locally, the weights of the features are different depending on sample. In this sample, the top 3 features are Sex, Age, and Embarked.
- 24. SHAP Results of SHAP Top3 does not agree with that in linear correlation/XGBoost score (Age enters). Red points（Sex_female=1:female） have positive SHAP values, while the blue points （Sex_ｆemale=0:male） negative. Women are likely to survive Passengers in 3rd class are likely to be killed Sorted with mean of |SHAP value| SHAP is consistent (unlike feature importance of XGBoost) and has local fidelity (unlike linear correlation), I would trust SHAP result than the other two.
- 25. Example: Regression (AmesHousing dataset) Variable Definition Key SalePrice the property's sale price in dollars - OverallQual Rates the overall material and finish of the house 10:Very Excellent 9:Excellent … 1:Very Poor GarageCars Size of garage in car capacity - KitchenAbvGr Kitchens above grade Ex:Excellent Gd:Good TA:Typical/Average Fa:Fair Po:Poor …(80 features in total) … … Label Features Dataset describing the sale of individual residential property in Ames, Iowa, from 2006 to 2010. # of training samples = 1460 # of test samples =1459 →Train : Validation =75 : 25
- 26. Example: Regression (AmesHousing dataset) Model: XGBoost Test: 0.16332 Validation : 0.13830 Results (metric is Root Mean Squared Error of Log (RMSEL) ) Kaggle notebook “Stacked Regressions : Top 4% on LeaderBoard” Cf: Score with the same method (XGBoost), but with feature engineering: 0.11420 This time, it is much better than simple baseline. Cf: Baseline (linear regression in terms of OverallQual): 1.1608 It is discretized because OverallQual is [1,…,10]. Prediction/actual value for validation data
- 27. Example: Regression (AmesHousing dataset) Linear correlation between features and label (explanation of data itself, not model) OverallQual 0.790982 GrLivArea 0.708624 GarageCars 0.640409 GarageArea 0.623431 TotalBsmtSF 0.613581 1stFlrSF 0.605852 FullBath 0.560664 TotRmsAbvGrd 0.533723 YearBuilt 0.522897 YearRemodAdd 0.50710 … Correlation between SalePrice and features Rates the overall material and finish of the houseAbove grade (ground) living area square feet Size of garage in car capacity
- 28. Example: Regression (AmesHousing dataset) Importance of features in GBDT (explanation of whole model, not for specific sample) Top 3 is different from linear correlation (KitchenAbvGr).
- 29. Example: Regression (AmesHousing dataset) (explanation of model prediction for specific sample) Explanation by SHAP GrLivArea: Above grade (ground) living area square feet TotalBsmtSF: Total square feet of basement area Top 3 is different from linear correlation/LIME.
- 30. Example: Image Classification Resultsof LIME The model seems to focus on the right places. However, there are models which can not be approximated with LIME. Ex: Classifier whether the image is “retro” or not considering the values of the whole pixels (sepia?)
- 31. Example: Image Classification Huskyor wolf example By looking at this explanation, it is easy to find that the model is focusing on snow.
- 32. Example: Image Classification •Anchor Gives range of features which does not change the prediction M. Ribeiro, et. al., Thirty-Second AAAI Conference on Artificial Intelligence. 2018. • Influence Gives training data which the prediction is based on P. Koh, P. Liang, arXiv:1703.04730 Training images which affect the prediction of the test imageTest image Other approaches
- 33. Contents • Concept/Motivation • Recenttrends on XAI • Method 1: LIME/SHAP • Example: Classification • Example: Regression • Example: Image classification • Method 2: ABN for image classification
- 34. Method 2: ABNfor image classification B. Zhou, et. al., arXiv:1512.04150 Class Activation Mapping (CAM) • Decrease classification accuracy because fully- connected (Fc) layer is replaced with GAP. 𝑓𝑘(𝑥, 𝑦) 𝑥, 𝑦: spatial coordinate 𝑘: channel 𝑓1(𝑥, 𝑦) 𝑆𝑐 = 𝑘 𝑤 𝑘 𝑐 𝐹𝑘 = 𝑥,𝑦 𝑀𝑐(𝑥, 𝑦) Score for class c 𝐹𝑘 = 𝑥,𝑦 𝑓𝑘(𝑥, 𝑦) Global Average Pooling (GAP) for channel c 𝑴 𝒄(𝒙, 𝒚) = 𝒌 𝒘 𝒌 𝒄 𝒇 𝒌(𝒙, 𝒚)
- 35. Method 2: ABNfor image classification Attention Branch Network (ABN) H. Fukui, et. al., arXiv:1812.10025 No Fc layer… Can add Fc layers!! Basic idea: Divide attention branch from classification branch so that Fc layers can be used in the latter branch.
- 36. Method 2: ABNfor image classification • Improved classification accuracy because it can use Fc layers. • Actually, using attention map in the input of perception/loss function improves accuracy. Perception branch uses also attention map not only features from feature extractor, as its input. Loss function has attention/perception terms. Both consists of cross entropy for classification task.
- 37. Method 2: ABNfor image classification Results of ABN As I do not have domain knowledge, I can not judge whether the model is focusing on correct features… Are the highlighted parts characteristic for each maker/model? It is interesting that, the attention is paid to different parts depending on the task. Attention map improves accuracy!
- 38. Method 2: ABNfor image classification The model seems to be focusing on correct features.
- 39. Method 2: ABNfor image classification M. Mitsuhara et. al., arXiv: 1905.03540ABN and human-in-the-loop Basic idea: By modifying attention map in ABN using human knowledge, try to improve the accuracy of image classifier.
- 40. Method 2: ABNfor image classification Modifying attention map manually helps classification Perception branch uses also attention map not only features from feature extractor, as its input.
- 41. Method 2: ABNfor image classification Results Modifying the attention map improved the classification accuracy
- 42. Method 2: ABNfor image classification ABN with HITL Feature extractor is not updated (fine-tuning)
- 43. Method 2: ABNfor image classification Results HITL improves the attention map (not very apparent), and also classification accuracy.
- 44. Method 2: ABNfor image classification HITL improves the attention map (focusing on relevant parts, not the whole body), and also classification accuracy.
- 45. Method 2: ABNfor image classification So maybe, being model-agnostic is not always useful: Sometimes it is better to be able to touch the model. HITL is possible because the explanation (attention map) is given as a part of their model in ABN, and it is reused as input of perception branch. Unlike LIME, ABN is not model-agnostic.
- 46. Summary I introduced afew recent methods on XAI. 1. LIME for • Classification of structured data • Regression of structured data • Classification of image 2. ABN for • Classification of image I also introduced application of human-in-the-loop to ABN
- 47.