kdd2019explainableaitutorial 190806180502

Fang

2019/08/09 发布于 研究 分类

可解释人工智能

XAI 

文字内容
1. Explainable AI in Industry KDD 2019 Tutorial Sahin Cem Geyik, Krishnaram Kenthapadi & Varun Mithal Krishna Gade & Ankur Taly 1 https://sites.google.com/view/kdd19-explainable-ai-tutorial
2. Agenda ● Motivation ● AI Explainability:'>Explainability: Foundations and Techniques ○ Explainability concepts, problem formulations, and evaluation methods ○ Post hoc Explainability ○ Intrinsically Explainable models ● AI Explainability:'>Explainability: Industrial Practice ○ Case Studies from LinkedIn, Fiddler Labs, and Google Research ● Demo ● Key Takeaways and Challenges 2
3. Motivation 3
4. Third Wave of AI Symbolic AI Statistical AI Explainable AI Logic rules represent knowledge Statistical models for specific domains training on big data Systems construct explanatory models No learning capability and poor handling of uncertainty No contextual capability and minimal explainability Systems learn and reason with new tasks and situations Factors driving rapid advancement of AI GPUs , On-chip Neural Network Data Availability Cloud Infrastructure New Algorithms
5. Need for Explainable AI User Current AI Systems Finance Marketing ● Why did you do that? ● Machine Learning centric today. ● ML Models are opaque, non- Security Logistics ● Why not something else? ● When do you succeed or fail? ● How do I correct an error? ● When do I trust you? intuitive and difficult to understand Explainable AI and ML is essential for future customers to understand, trust, and effectively manage the emerging generation of AI applications
6. Black-box AI creates business risk for Industry
7. Black-box AI creates confusion and doubt Can I trust our AI decisions? Business Owner Why I am getting this decision? How do I answer this customer complaint? Customer Support Poor Decision How can I get a better decision? Black-box AI How do I monitor and debug this model? IT & Operations Data Scientists Is this the best model that can be built? Are these AI system decisions fair? Internal Audit, Regulators
8. What is Explainable AI? Black Box AI Data Black-Box AI Decision, Recommendation AI product Confusion with Today’s AI Black Box ● ● ● ● Why did you do that? Why did you not do that? When do you succeed or fail? How do I correct an error? Explainable AI Feedback Data Explainable AI Explainable AI Product Clear & Transparent Predictions Decision Explanation ● ● ● ● I understand why I understand why not I know why you succeed or fail I understand, so I trust you
9. Why Explainability: Verify the ML Model / System Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18 9
10. Why Explainability: Improve ML Model Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18 10
11. Why Explainability: Learn New Insights Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18 11
12. Why Explainability: Learn Insights in the Sciences Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18 12
13. Why Explainability: Debug (Mis-)Predictions Top label: “clog” Why did the network label this image as “clog”? 13
14. Why Explainability: Laws against Discrimination Citizenship Immigration Reform and Control Act Sex Equal Pay Act of 1963; Civil Rights Act of 1964 Race Civil Rights Act of 1964 Age Age Discrimination in Employment Act of 1967 Disability status Rehabilitation Act of 1973; Americans with Disabilities Act of 1990 And more... 14
15. Fairness Privacy Transparency Explainability 15
16. GDPR Concerns Around Lack of Explainability in AI “ Companies should commit to ensuring systems that could fall under GDPR, including AI, will be compliant. The threat of sizeable fines of €20 million or 4% of global turnover provides a sharp incentive. Article 22 of GDPR empowers individuals with the right to demand an explanation of how an AI system made a decision that affects them. ” - European Commision VP, European Commision
17. Fairness Privacy Transparency Explainability 17
18. Fairness Privacy Transparency Explainability 18
19. SR 11-7 and OCC regulations for Financial Institutions 19
20. “Explainability by Design” for AI products Model Debugging Model Visualization Feedback Loop Train Model Diagnostics Root Cause Analytics Debug Model Evaluation Compliance Testing Performance monitoring Fairness monitoring Explainable AI Monitor QA Deploy A/B Test Model Comparison Cohort Analysis Predict Explainable Decisions API Support Model Launch Signoff Model Release Mgmt
21. Example: Facebook adds Explainable AI to build Trust
22. Foundations and Techniques 22
23. Achieving Explainable AI Approach 1: Post-hoc explain a given AI model ● Individual prediction explanations in terms of input features, influential examples, concepts, local decision rules ● Global prediction explanations in terms of entire model in terms of partial dependence plots, global feature importance, global decision rules Approach 2: Build an interpretable model ● Logistic regression, Decision trees, Decision lists and sets, Generalized Additive Models (GAMs) 23
24. Achieving Explainable AI Approach 1: Post-hoc explain a given AI model ● Individual prediction explanations in terms of input features, influential examples, concepts, local decision rules ● Global prediction explanations in terms of entire model in terms of partial dependence plots, global feature importance, global decision rules Approach 2: Build an interpretable model ● Logistic regression, Decision trees, Decision lists and sets, Generalized Additive Models (GAMs) 24
25. Top label: “fireboat” Why did the network label this image as “fireboat”? 25
26. Top label: “clog” Why did the network label this image as “clog”? 26
27. Credit Lending in a black-box ML world Credit Line Increase ? Request Denied Bank Query AI System Credit Lending Score = 0.3 Why? Why not? How? Fair lending laws [ECOA, FCRA] require credit decisions to be explainable Credit Lending Model
28. The Attribution Problem Attribute a model’s prediction on an input to features of the input Examples: ● Attribute an object recognition network’s prediction to its pixels ● Attribute a text sentiment network’s prediction to individual words ● Attribute a lending model’s prediction to its features A reductive formulation of “why this prediction” but surprisingly useful :-)
29. Application of Attributions ● Debugging model predictions E.g., Attribution an image misclassification to the pixels responsible for it ● Generating an explanation for the end-user E.g., Expose attributions for a lending prediction to the end-user ● Analyzing model robustness E.g., Craft adversarial examples using weaknesses surfaced by attributions ● Extract rules from the model E.g., Combine attribution to craft rules (pharmacophores) capturing prediction logic of a drug screening network 29
30. Next few slides We will cover the following attribution methods** ● Ablations ● Gradient based methods ● Score Backpropagation based methods ● Shapley Value based methods **Not a complete list! See Ancona et al. [ICML 2019], Guidotti et al. [arxiv 2018] for a comprehensive survey 30
31. Ablations Drop each feature and attribute the change in prediction to that feature Useful tool but not a perfect attribution method. Why? ● Unrealistic inputs ● Improper accounting of interactive features ● Computationally expensive 31
32. Feature*Gradient Attribution to a feature is feature value times gradient, i.e., xi* 𝜕y/𝜕 xi ● Gradient captures sensitivity of output w.r.t. feature ● Equivalent to Feature*Coefficient for linear models ○ ● First-order Taylor approximation of non-linear models Popularized by SaliencyMaps [NIPS 2013], Baehrens et al. [JMLR 2010] Gradients in the vicinity of the input seem like noise 32
33. score Interesting gradients uninteresting gradients (saturation) 1.0 intensity 0.0 Baseline … scaled inputs ... … gradients of scaled inputs …. Input
34. Integrated Gradients [ICML 2017] Integrate the gradients along a straight-line path from baseline to input IG(input, base) ::= (input - base) * ∫0 -1▽F(𝛂*input + (1-𝛂)*base) d𝛂 Original image Integrated Gradients
35. What is a baseline? ● Ideally, the baseline is an informationless input for the model ○ E.g., Black image for image models ○ E.g., Empty text or zero embedding vector for text models ● Integrated Gradients explains F(input) - F(baseline) in terms of input features Aside: Baselines (or Norms) are essential to explanations [Kahneman-Miller 86] ● E.g., A man suffers from indigestion. Doctor blames it to a stomach ulcer. Wife blames it on eating turnips. Both are correct relative to their baselines. ● The baseline may also be an important analysis knob.
36. Why is this image labeled as “clog”? Original image “Clog”
37. Why is this image labeled as “clog”? Original image Integrated Gradients (for label “clog”) “Clog”
38. Detecting an architecture bug ● Deep network [Kearns, 2016] predicts if a molecule binds to certain DNA site ● Finding: Some atoms had identical attributions despite different connectivity
39. Detecting an architecture bug ● Deep network [Kearns, 2016] predicts if a molecule binds to certain DNA site ● Finding: Some atoms had identical attributions despite different connectivity ● Bug: The architecture had a bug due to which the convolved bond features did not affect the prediction!
40. Detecting a data issue ● Deep network predicts various diseases from chest x-rays Original image Integrated gradients (for top label)
41. Detecting a data issue ● Deep network predicts various diseases from chest x-rays ● Finding: Attributions fell on radiologist’s markings (rather than the pathology) Original image Integrated gradients (for top label)
42. Score Back-Propagation based Methods Re-distribute the prediction score through the neurons in the network ● LRP [JMLR 2017], DeepLift [ICML 2017], Guided BackProp [ICLR 2014] Easy case: Output of a neuron is a linear function of previous neurons (i.e., ni = ⅀ wij * nj) e.g., the logit neuron ● Image credit heatmapping.org Re-distribute the contribution in proportion to the coefficients wij 42
43. Score Back-Propagation based Methods Re-distribute the prediction score through the neurons in the network ● LRP [JMLR 2017], DeepLift [ICML 2017], Guided BackProp [ICLR 2014] Tricky case: Output of a neuron is a non-linear function, e.g., ReLU, Sigmoid, etc. Image credit heatmapping.org ● Guided BackProp: Only consider ReLUs that are on (linear regime), and which contribute positively ● LRP: Use first-order Taylor decomposition to linearize activation function ● DeepLift: Distribute activation difference relative a reference point in proportion to edge weights 43
44. Score Back-Propagation based Methods Re-distribute the prediction score through the neurons in the network ● LRP [JMLR 2017], DeepLift [ICML 2017], Guided BackProp [ICLR 2014] Pros: ● Conceptually simple ● Methods have been empirically validated to yield sensible result Image credit heatmapping.org Cons: ● Hard to implement, requires instrumenting the model ● Often breaks implementation invariance Think: F(x, y, z) = x * y *z and G(x, y, z) = x * (y * z)
45. So far we’ve looked at differentiable models. But, what about non-differentiable models? E.g., ● Decision trees ● Boosted trees ● Random forests ● etc.
46. Shapley Value [Annals of Mathematical studies,1953] Classic result in game theory on distributing gain in a coalition game ● Coalition Games ○ Players collaborating to generate some gain (think: revenue) ○ Set function v(S) determining the gain for any subset S of players
47. Shapley Value [Annals of Mathematical studies,1953] Classic result in game theory on distributing gain in a coalition game ● ● Coalition Games ○ Players collaborating to generate some gain (think: revenue) ○ Set function v(S) determining the gain for any subset S of players Shapley Values are a fair way to attribute the total gain to the players based on their contributions ○ Concept: Marginal contribution of a player to a subset of other players (v(S U {i}) - v(S)) ○ Shapley value for a player is a specific weighted aggregation of its marginal over all possible subsets of other players Shapley Value for player i = ⅀S⊆N w(S) * (v(S U {i}) - v(S)) (where w(S) = N! / S ! (N - S -1)!)
48. Shapley Value Justification Shapley values are unique under four simple axioms ● Dummy: If a player never contributes to the game then it must receive zero attribution ● Efficiency: Attributions must add to the total gain ● Symmetry: Symmetric players must receive equal attribution ● Linearity: Attribution for the (weighted) sum of two games must be the same as the (weighted) sum of the attributions for each of the games
49. Shapley Values for Explaining ML models SHAP [NeurIPS 2018], QII [S&P 2016], Strumbelj & Konenko [JMLR 2009] ● Define a coalition game for each model input X ○ Players are the features in the input ○ Gain is the model prediction (output), i.e., gain = F(X) ● Feature attributions are the Shapley values of this game
50. Shapley Values for Explaining ML models SHAP [NeurIPS 2018], QII [S&P 2016], Strumbelj & Konenko [JMLR 2009] ● Define a coalition game for each model input X ○ Players are the features in the input ○ Gain is the model prediction (output), i.e., gain = F(X) ● Feature attributions are the Shapley values of this game Challenge: Shapley Values require the gain to be defined for all subsets of players ● What is the prediction when some players (features) are absent? i.e., what is F(x_1, , x_3, …, )?
51. Modeling Feature Absence Key Idea: Take the expected prediction when the (absent) feature is sampled from a certain distribution. Different approaches choose different distributions ● [SHAP, NIPS 2018] Use conditional distribution w.r.t. the present features ● [QII, S&P 2016] Use marginal distribution ● [Strumbelj et al., JMLR 2009] Use uniform distribution ● [Integrated Gradients, ICML 2017] Use a specific baseline point
52. Computing Shapley Values Exact Shapley value computation is exponential in the number of features ● Shapley values can be expressed as an expectation of marginals 𝜙(i) = ES ~ D [marginal(S, i)] ● Sampling-based methods can be used to approximate the expectation ● See: “Computational Aspects of Cooperative Game Theory”, Chalkiadakis et al. 2011 ● The method is still computationally infeasible for models with hundreds of features, e.g., image models
53. Evaluating Attribution Methods
54. Human Review Have humans review attributions and/or compare them to (human provided) groundtruth on “feature importance” Pros: ● Helps assess if attributions are human-intelligible ● Helps increase trust in the attribution method Cons: ● Attributions may appear incorrect because model reasons differently ● Confirmation bias
55. Perturbations (Samek et al., IEEE NN and LS 2017) Perturb top-k features by attribution and observe change in prediction ● Higher the change, better the method ● Perturbation may amount to replacing the feature with a random value ● Samek et al. formalize this using a metric: Area over perturbation curve ○ Plot the prediction for input with top-k features perturbed as a function of k ○ Take the area over this curve Area over perturbation curve Prediction for perturbed inputs Drop in prediction when top 40 features are perturbed 10 20 30 40 50 60 Number of perturbed features
56. Axiomatic Justification Inspired by how Shapley Values are justified ● List desirable criteria (axioms) for an attribution method ● Establish a uniqueness result: X is the only method that satisfies these criteria Integrated Gradients, SHAP, QII, Strumbelj & Konenko are justified in this manner Theorem [Integrated Gradients, ICML 2017]: Integrated Gradients is the unique path-integral method satisfying: Sensitivity, Insensitivity, Linearity preservation, Implementation invariance, Completeness, and Symmetry
57. Some limitations and caveats
58. Attributions are pretty shallow Attributions do not explain: ● Feature interactions ● What training examples influenced the prediction ● Global properties of the model An instance where attributions are useless: ● A model that predicts TRUE when there are even number of black pixels and FALSE otherwise
59. Attributions are for human consumption ● Humans interpret attributions and generate insights ○ Doctor maps attributions for x-rays to pathologies ● Visualization matters as much as the attribution technique
60. Attributions are for human consumption ● Humans interpret attributions and generate insights ○ Doctor maps attributions for x-rays to pathologies ● Visualization matters as much as the attribution technique Naive scaling of attributions from 0 to 255 Attributions have a large range and long tail across pixels After clipping attributions at 99% to reduce range
61. Other types of Post-hoc Explanations
62. Example based Explanations Learned prototypes and criticisms from Imagenet dataset (two types of dog breeds) ● Prototypes: Representative of all the training data. ● Criticisms: Data instance that is not well represented by the set of prototypes. Figure credit: Examples are not Enough, Learn to Criticize! Criticism for Interpretability. Kim, Khanna and Koyejo. NIPS 2016 62
63. Influence functions ● Trace a model’s prediction through the learning algorithm and back to its training data ● Training points “responsible” for a given prediction Figure credit: Understanding Black-box Predictions via Influence Functions. Koh and Liang ICML 2017 63
64. Local Interpretable Model-agnostic Explanations (Ribeiro et al. KDD 2016) Figure credit:'>credit: Ribeiro et al. KDD 2016 Figure credit:'>credit: Anchors: High-Precision Model-Agnostic Explanations. Ribeiro et al. AAAI 2018 64
65. Anchors Figure credit: Anchors: High-Precision Model-Agnostic Explanations. Ribeiro et al. AAAI 2018 65
66. Figure credit: Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) Kim et al. 2018 66
67. Global Explanations 67
68. Global Explanations Methods ● Partial Dependence Plot: Shows the marginal effect one or two features have on the predicted outcome of a machine learning model 68
69. Global Explanations Methods ● Permutations: The importance of a feature is the increase in the prediction error of the model after we permuted the feature’s values, which breaks the relationship between the feature and the true outcome. 69
70. Achieving Explainable AI Approach 1: Post-hoc explain a given AI model ● Individual prediction explanations in terms of input features, influential examples, concepts, local decision rules ● Global prediction explanations in terms of entire model in terms of partial dependence plots, global feature importance, global decision rules Approach 2: Build an interpretable model ● Logistic regression, Decision trees, Decision lists and sets, Generalized Additive Models (GAMs) 70
71. Decision Trees Is the person fit? Age < 30 ? Yes Eats a lot of pizzas? Yes Unfit No Fit No Exercises in the morning? Yes No Fit Unfit 71
72. Decision List Figure credit: Interpretable Decision Sets: A Joint Framework for Description and Prediction, Lakkaraju, Bach, Leskovec 72
73. Decision Set Figure credit: Interpretable Decision Sets: A Joint Framework for Description and Prediction, Lakkaraju, Bach, Leskovec 73
74. GLMs and GAMs Intelligible Models for Classification and Regression. Lou, Caruana and Gehrke KDD 2012 Accurate Intelligible Models with Pairwise Interactions. Lou, Caruana, Gehrke and Hooker. KDD 2013 74
75. Case Studies from Industry 75
76. Case Study: Talent Search Varun Mithal, Girish Kathalagiri, Sahin Cem Geyik 76
77. LinkedIn Recruiter ● Recruiter Searches for Candidates ○ Standardized and free-text search criteria ● Retrieval and Ranking ○ ○ Filter candidates using the criteria Rank candidates in multiple levels using ML models 77
78. Modeling Approaches ● Pairwise XGBoost ● GLMix ● DNNs via TensorFlow ● Optimization Criteria: inMail Accepts ○ Positive: inMail sent by recruiter, and positively responded by candidate ■ Mutual interest between the recruiter and the candidate 78
79. Feature Importance in XGBoost 79
80. How We Utilize Feature Importances for GBDT ● Understanding feature digressions ○ ○ Which a feature that was impactful no longer is? Should we debug feature generation? ● Introducing new features in bulk and identifying effective ones ○ ○ An activity feature for last 3 hours, 6 hours, 12 hours, 24 hours introduced (costly to compute) Should we keep all such features? ● Separating the factors for that caused an improvement ○ ○ Did an improvement come from a new feature, or a new labeling strategy, data source? Did the ordering between features change? ● Shortcoming: A global view, not case by case 80
81. GLMix Models ● Generalized Linear Mixed Models ○ ○ ○ Global: Linear Model Per-contract: Linear Model Per-recruiter: Linear Model ● Lots of parameters overall ○ For a specific recruiter or contract the weights can be summed up ● Inherently explainable ○ ○ Contribution of a feature is “weight x feature value” Can be examined in a case-by-case manner as well 81
82. TensorFlow Models in Recruiter and Explaining Them ● We utilize the Integrated Gradients [ICML 2017] method ● How do we determine the baseline example? ○ ○ ○ ○ ○ ○ Every query creates its own feature values for the same candidate Query match features, time-based features Recruiter affinity, and candidate affinity features A candidate would be scored differently by each query Cannot recommend a “Software Engineer” to a search for a “Forensic Chemist” There is no globally neutral example for comparison! 82
83. Query-Specific Baseline Selection ● For each query: ○ ○ ○ ○ Score examples by the TF model Rank examples Choose one example as the baseline Compare others to the baseline example ● How to choose the baseline example ○ ○ ○ ○ Last candidate Kth percentile in ranking A random candidate Request by user (answering a question like: “Why was I presented candidate x above candidate y?”) 83
84. Example 84
85. Example - Detailed Feature Description Difference (1 vs 2) Contribution Feature………. Description………. -2.0476928 -2.144455602 Feature………. Description………. -2.3223877 1.903594618 Feature………. Description………. 0.11666667 0.2114946752 Feature………. Description………. -2.1442587 0.2060414469 Feature………. Description………. -14 0.1215354111 Feature………. Description………. 1 0.1000282466 Feature………. Description………. -92 -0.085286277 Feature………. Description………. 0.9333333 0.0568533262 Feature………. Description………. -1 -0.051796317 Feature………. Description………. -1 -0.050895940 85
86. Pros & Cons ● Explains potentially very complex models ● Case-by-case analysis ○ ○ ○ ○ Why do you think candidate x is a better match for my position? Why do you think I am a better fit for this job? Why am I being shown this ad? Great for debugging real-time problems in production ● Global view is missing ○ ○ Aggregate Contributions can be computed Could be costly to compute 86
87. Lessons Learned and Next Steps ● Global explanations vs. Case-by-case Explanations ○ ○ Global gives an overview, better for making modeling decisions Case-by-case could be more useful for the non-technical user, better for debugging ● Integrated gradients worked well for us ○ ○ Complex models make it harder for developers to map improvement to effort Use-case gave intuitive results, on top of completely describing score differences ● Next steps ○ Global explanations for Deep Models 87
88. Case Study: Model Interpretation for Predictive Models in B2B Sales Predictions Jilei Yang, Wei Di, Songtao Guo 88
89. Problem Setting ● Predictive models in B2B sales prediction ○ ○ E.g.: random forest, gradient boosting, deep neural network, … High accuracy, low interpretability ● Global feature importance → Individual feature reasoning 89
90. Example 90
91. Revisiting LIME ● Given a target sample 𝑥𝑘, approximate its prediction 𝑝𝑟𝑒𝑑(𝑥𝑘) by building a sample-specific linear model: 𝑝𝑟𝑒𝑑(𝑋) ≈ 𝛽𝑘1 𝑋1 + 𝛽𝑘2 𝑋2 + …, 𝑋 ∈ 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟(𝑥𝑘) ● E.g., for company CompanyX: 0.76 ≈ 1.82 ∗ 0.17 + 1.61 ∗ 0.11+… 91
92. xLIME Piecewise Linear Regression Localized Stratified Sampling 92
93. Piecewise Linear Regression Motivation: Separate top positive feature influencers and top negative feature influencers 93
94. Impact of Piecewise Approach ● Target sample 𝑥𝑘=(𝑥𝑘1, 𝑥𝑘2, ⋯) ● Top feature contributor ○ ○ LIME:'>LIME:'>LIME:'>LIME: large magnitude of 𝛽𝑘𝑗 ⋅ 𝑥𝑘𝑗 xLIME:'>LIME:'>LIME:'>LIME: large magnitude of 𝛽𝑘𝑗− ⋅ 𝑥𝑘𝑗 ● Top positive feature influencer ○ ○ LIME:'>LIME:'>LIME:'>LIME: large magnitude of 𝛽𝑘𝑗 xLIME:'>LIME:'>LIME:'>LIME: large magnitude of negative 𝛽𝑘𝑗− or positive 𝛽𝑘𝑗+ ● Top negative feature influencer ○ ○ LIME:'>LIME:'>LIME:'>LIME: large magnitude of 𝛽𝑘𝑗 xLIME:'>LIME:'>LIME:'>LIME: large magnitude of positive 𝛽𝑘𝑗− or negative 𝛽𝑘𝑗+ 94
95. Localized Stratified Sampling: Idea Method: Sampling based on empirical distribution around target value at each feature level 95
96. Localized Stratified Sampling: Method ● Sampling based on empirical distribution around target value for each feature ● For target sample 𝑥𝑘 = (𝑥𝑘1 , 𝑥𝑘2 , ⋯), sampling values of feature 𝑗 according to 𝑝𝑗 (𝑋𝑗) ⋅ 𝑁(𝑥𝑘𝑗 , (𝛼 ⋅ 𝑠𝑗 )2) ○ ○ ○ ○ ● In 𝑝𝑗 (𝑋𝑗) : empirical distribution. 𝑥𝑘𝑗 : feature value in target sample. 𝑠𝑗 : standard deviation. 𝛼 : Interpretable range: tradeoff between interpretable coverage and local accuracy. LIME, sampling according to 𝑁(𝑥_ , 𝑠 2). 𝑗 𝑗 96
97. Summary 97
98. LTS LCP (LinkedIn Career Page) Upsell ● A subset of churn data ○ ○ Total Companies: ~ 19K Company features: 117 ● Problem: Estimate whether there will be upsell given a set of features about the company’s utility from the product 98
99. Top Feature Contributor 99
100. 100
101. Top Feature Influencers 101
102. Key Takeaways ● Looking at the explanation as contributor vs. influencer features is useful ○ ○ Contributor: Which features end-up in the current outcome case-by-case Influencer: What needs to be done to improve likelihood, case-by-case ● xLIME aims to improve on LIME via: ○ ○ Piecewise linear regression: More accurately describes local point, helps with finding correct influencers Localized stratified sampling: More realistic set of local points ● Better captures the important features 102
103. Case Study: Relevance Debugging and Explaining @ Daniel Qiu, Yucheng Qian 103
104. Debugging Relevance Models 104
105. Architecture 105
106. What Could Go Wrong? 106
107. Challenges 107
108. Solution 108
109. Call Graph 109
110. Timing 110
111. Features 111
112. Advanced Use Cases 112
113. Perturbation 113
114. Comparison 114
115. Holistic Comparison 115
116. Granular Comparison 116
117. Replay 117
118. Teams ● ● ● ● ● ● Search Feed Comments People you may know Jobs you may be interested in Notification 118
119. Case Study: Integrated Gradients for Adversarial Analysis of Question-Answering models Ankur Taly** (Fiddler labs) (Joint work with Mukund Sundararajan, Kedar Dhamdhere, Pramod Mudrakarta) **This research was carried out at Google Research 119
120. Tabular QA Visual QA Reading Comprehension Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls. He is also the oldest quarterback ever to play in a Super Bowl at age 39. The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age 38 and is currently Denver’s Executive Vice President of Football Operations and General Manager Q:'>Q:'>Q:'>Q: How many medals did India win? A:'>A:'>A:'>A: 197 Q:'>Q:'>Q:'>Q: How symmetrical are the white bricks on either side of the building? A:'>A:'>A:'>A: very Neural Programmer (2017) model 33.5% accuracy on WikiTableQuestions Kazemi and Elqursh (2017) model. 61.1% on VQA 1.0 dataset (state of the art = 66.7%) Q:'>Q:'>Q:'>Q: Name of the quarterback who was 38 in Super Bowl XXXIII? A:'>A:'>A:'>A: John Elway Yu et al (2018) model. 84.6 F-1 score on SQuAD (state of the art) Robustness question: Do these models understand the question? :-) 120
121. Visual QA Kazemi and Elqursh (2017) model. Accuracy: 61.1% (state of the art: 66.7%) Q: How symmetrical are the white bricks on either side of the building? A: very 121
122. Visual QA Kazemi and Elqursh (2017) model. Accuracy: 61.1% (state of the art: 66.7%) Q:'>Q: How symmetrical are the white bricks on either side of the building? A:'>A: very Q:'>Q: How asymmetrical are the white bricks on either side of the building? A:'>A: very 122
123. Visual QA Kazemi and Elqursh (2017) model. Accuracy: 61.1% (state of the art: 66.7%) Q:'>Q:'>Q:'>Q: How symmetrical are the white bricks on either side of the building? A:'>A:'>A:'>A: very Q:'>Q:'>Q:'>Q: How asymmetrical are the white bricks on either side of the building? A:'>A:'>A:'>A: very Q:'>Q:'>Q:'>Q: How big are the white bricks on either side of the building? A:'>A:'>A:'>A: very 123
124. Visual QA Kazemi and Elqursh (2017) model. Accuracy: 61.1% (state of the art: 66.7%) Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q: How symmetrical are the white bricks on either side of the building? A:'>A:'>A:'>A:'>A:'>A:'>A:'>A: very Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q: How asymmetrical are the white bricks on either side of the building? A:'>A:'>A:'>A:'>A:'>A:'>A:'>A: very Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q: How big are the white bricks on either side of the building? A:'>A:'>A:'>A:'>A:'>A:'>A:'>A: very Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q: How fast are the bricks speaking on either side of the building? A:'>A:'>A:'>A:'>A:'>A:'>A:'>A: very 124
125. Visual QA Kazemi and Elqursh (2017) model. Accuracy: 61.1% (state of the art: 66.7%) Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q: How symmetrical are the white bricks on either side of the building? A:'>A:'>A:'>A:'>A:'>A:'>A:'>A: very Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q: How asymmetrical are the white bricks on either side of the building? A:'>A:'>A:'>A:'>A:'>A:'>A:'>A: very Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q: How big are the white bricks on either side of the building? A:'>A:'>A:'>A:'>A:'>A:'>A:'>A: very Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q:'>Q: How fast are the bricks speaking on either side of the building? A:'>A:'>A:'>A:'>A:'>A:'>A:'>A: very Test/dev accuracy does not show us the entire picture. Need to look inside! 125
126. Analysis procedure ● Attribute the answer (or answer selection logic) to question words ○ Baseline: Empty question, but full context (image, text, paragraph) ■ By design, attribution will not fall on the context ● Visualize attributions per example ● Aggregate attributions across examples
127. Visual QA attributions Q: How symmetrical are the white bricks on either side of the building? A: very How symmetrical are the white bricks on either side of the building? red: high attribution blue: negative attribution gray: near-zero attribution 127
128. Over-stability [Jia and Liang, EMNLP 2017] Jia & Liang note that: ● Image networks suffer from “over-sensitivity” to pixel perturbations ● Paragraph QA models suffer from “over-stability” to semantics-altering edits Attributions show how such over-stability manifests in Visual QA, Tabular QA and Paragraph QA networks
129. Over-stability During inference, drop all words from the dataset except ones which are frequently top attributions ● E.g. How many red buses are in the picture? Visual QA Top tokens: color, many, what, is, how, there, … 129
130. Over-stability During inference, drop all words from the dataset except ones which are frequently top attributions ● E.g. How many red buses are in the picture? Visual QA 50% of final accuracy with just one word “color” 80% of final accuracy reached with just 100 words (Orig vocab size: 5305) Top tokens: color, many, what, is, how, there, … 130
131. Attack: Subject ablation Replace the subject of a question with a low-attribution noun from the vocabulary ● This ought to change the answer but often does not! Low-attribution nouns 'tweet', 'childhood', 'copyrights', 'mornings', 'disorder', 'importance', 'topless', 'critter', 'jumper', 'fits' What is the man doing? → What is the tweet doing? How many children are there? → How many tweet are there? VQA model’s response remains the same 75.6% of the time on questions that it originally answered correctly 131
132. Many other attacks! ● Visual QA ○ Prefix concatenation attack (accuracy drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop: 61.1% to 19%) ○ Stop word deletion attack (accuracy drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop: 61.1% to 52%) ● Tabular QA ○ Prefix concatenation attack (accuracy drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop: 33.5% to 11.4%) ○ Stop word deletion attack (accuracy drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop: 33.5% to 28.5%) ○ Table row reordering attack (accuracy drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop:'>drop: 33.5 to 23%) ● Paragraph QA ○ Improved paragraph concatenation attacks of Jia and Liang from [EMNLP 2017] Paper: Did the model understand the question? [ACL 2018]
133. Fiddler Demo 133
134. Fiddler is an explainable AI engine designed for the enterprise Pluggable Platform Integrate, deploy, visualize a wide variety of custom models Explainable AI Deliver clear decisions and explanations to your end users Trust & Governance Easy governed access helps teams build and understand Trusted AI Simplified Setup Lean and pluggable AI platform with cloud or on-prem integrations
135. Fiddler - Explainable AI Engine All your data Any data warehouse Custom Models Fiddler Modeling Layer Explainable AI for everyone APIs, Dashboards, Reports, Trusted Insights 135
136. Benefits Across the Organization More trust and customer retention Customers Business Owner DevOps and IT have greater visibility into AI Transparent AI workflow and GDPR compliance Data Scientists are more agile and effective Dev Ops Data Scientists
137. Fiddler enables building Explainable AI Applications like this!
138. Can explanations help build Trust? ● Can we know when the model is uncertain? ● Does the model make the same mistake as a human? ● Are we comfortable with the model? *Zachary Lipton, et al. The Mythos of Model Interpretability ICML 2016
139. Can explanations help identify Causality? ● Predictions vs actions ● Explanations on why this happened as opposed to how *Zachary Lipton, et al. The Mythos of Model Interpretability ICML 2016
140. Can explanations be Transferable? ● Training and test setups often differ from the wild ● Real world data is always changing and noisy *Zachary Lipton, et al. The Mythos of Model Interpretability ICML 2016
141. Can explanations provide more Information? ● Often times models aid human decisions ● Extra bits of information other than model decision could be valuable *Zachary Lipton, et al. The Mythos of Model Interpretability ICML 2016
142. Challenges & Tradeoffs ● Lack of standard interface for ML models makes pluggable explanations hard ● Explanation needs vary depending on the type of the user who needs it and also the problem Fairness Performance at hand. ● The algorithm you employ for explanations might depend on the use-case, model type, ? data format, etc. ● There are trade-offs w.r.t. Explainability, Transparency User Privacy Performance, Fairness, and Privacy. 142
143. Reflections ● Case studies on explainable AI in practice ● Need “Explainability by Design” when building AI products 143
144. Fairness Privacy Related KDD’19 sessions: 1.Tutorial: Fairness-Aware Machine Learning: Practical Challenges and Lessons Learned (Sun) 2.Workshop: Explainable AI/ML (XAI) for Accountability, Fairness, and Transparency (Mon) 3.Social Impact Workshop (Wed, 8:15 – 11:45) 4.Keynote: Cynthia Rudin, Do Simpler Models Exist and How Can We Find Them? (Thu, 8 - 9am) 5.Several papers on fairness (e.g., ADS7 (Thu, 10-12), ADS9 (Thu, 1:30-3:30)) 6.Research Track Session RT17: Interpretability (Thu, 10am - 12pm) Transparency Explainability 144
145. Thanks! Questions? ● Feedback most welcome :-) ○ krishna@fiddler.ai, sgeyik@linkedin.com, kkenthapadi@linkedin.com, vamithal@linkedin.com, ankur@fiddler.ai ● Tutorial website: https://sites.google.com/view/kdd19-explainable- ai-tutorial ● To try Fiddler, please send an email to info@fiddler.ai 145