A risk model turns down an application, and someone asks the obvious question: why? The answer usually arrives as a chart. A row of colored bars, a heatmap smeared across a sentence, a thin line pointing at one word. Those artifacts come from three different tools, and they are not interchangeable. Each one answers a narrower question than the one that was asked, which is exactly why they get misread in meetings where nobody has time to ask what the axes mean.
Below is a working guide to what each technique produces, where it breaks, and how to keep a reasonable explanation from turning into a confident wrong one.
Three Tools, Three Different Questions
It helps to phrase the question before reaching for a library.
- SHAP asks: for this single prediction, how much did each input feature push the output up or down, relative to a baseline?
- LIME asks: what simple rule, fitted around this one data point, roughly mimics the model nearby?
- Attention maps ask: which tokens did this layer route information from, for this sequence?
None of the three reveals the model's reasoning in the way a person would describe their own. They are measurements of behavior, and every measurement has a unit, a baseline, and a blind spot.
SHAP: Credit Assignment, Not Cause
SHAP borrows from cooperative game theory. Features are players, the prediction is the payout, and each feature receives a share based on its average marginal contribution across many orderings. The result is additive by construction: the baseline value plus every feature's contribution equals the prediction. That property is what makes the plots readable, and it is also what tempts people to over-read them.
Reading a SHAP plot correctly
Check three things before you say anything out loud. First, what is the baseline? A missing or arbitrary reference point shifts every bar. Second, what unit is the output in? For a classifier, contributions are often in log-odds, not probability, so a bar of 0.8 is not "80 percent of the decision." Third, is this a local explanation or a global summary? A single waterfall plot describes one row of data. A global bar chart of mean absolute values describes the model's overall sensitivity, which is a different claim entirely.
The correlated-feature trap
When two inputs move together, Shapley values split credit between them in ways that depend on the sampling scheme. You may see a plausible feature get a near-zero contribution simply because a near-duplicate absorbed the credit. Similarly, a feature can carry a large contribution and still have no causal effect on the outcome. SHAP tells you what the model leaned on. It cannot tell you what would happen if you changed the world.
LIME: A Local Sketch, Drawn in Pencil
LIME perturbs a single instance, weights the perturbed samples by how close they are to the original, and fits a simple model to that neighborhood. In text, this usually means deleting words and watching the prediction move. The output is a short list of features with signed weights.
Two properties matter in practice. It is local: the sketch applies near the point you drew it, and often nowhere else. And it is stochastic: sampling, kernel width, and the number of perturbed examples all change the story. Run LIME twice with different seeds and you can get visibly different word lists for the same prediction. That is not a bug to hide. It is a signal to report a distribution rather than a single tidy chart.
LIME is most useful when you want a quick, human-readable sanity check on a single case — especially with text or sparse tabular data where feature-level SHAP output is hard to parse. It is least useful when someone treats the word list as the model's internal checklist.
Attention Maps: Routing, Not Reasoning
Attention weights show how a layer mixed information across positions. They are normalized per query, they differ wildly between heads and layers, and they are one component inside a much larger computation. A token receiving high attention is not automatically the token that drove the prediction. Researchers have shown that attention patterns can be substantially altered, sometimes scrambled, with little change in model output. If the output barely moves when the pattern changes, the pattern was not the decision.
Where attention maps earn their keep: debugging. They help you spot a model attending to padding tokens, to a boilerplate disclaimer instead of the substance, or to the wrong span of a long document. That is inspection, not explanation.
Five Misreadings to Avoid
- Treating importance as causation. A large contribution describes the model, not the world. Interventions, not attributions, answer causal questions.
- Generalizing from one local plot. One explanation for one case says nothing about the model's overall behavior.
- Ignoring correlated inputs. Check for near-duplicate features before you interpret any ranking.
- Forgetting the output scale and baseline. Log-odds, raw margins, and probabilities produce charts that look alike and mean different things.
- Choosing the explanation that supports the decision. If you generate several explanations and keep the convenient one, you have built a justification engine, not an interpretability practice.
An explanation is a claim about a model's behavior. It deserves the same review you would give the model itself.
A Short Review Routine That Holds Up
Write down the question first, in one sentence. "Why did this case score high?" and "Where is the model fragile across the population?" lead to different tools and different plots.
Then fix the plumbing: pin the library version, set random seeds where the method allows, and record the baseline and output scale alongside any figure you publish. Run at least two methods and note where they disagree. Disagreement is the interesting part, and it usually points to correlations, sparse regions, or a genuinely unstable decision boundary.
Next, validate with a perturbation. Remove or alter the feature the explanation highlighted, rerun the model, and see whether the prediction moves as predicted. This step catches more bad explanations than any amount of plot interpretation. If you can, route the result past someone who knows the domain but not the model — they will spot a nonsense feature faster than any metric.
Finally, write the limits into the report: which cases the explanation covers, what it does not establish, and how much variability to expect across reruns. For decisions that affect people's access to credit, employment, housing, or care, bring in legal and domain reviewers early rather than presenting an explanation chart as settled fact.
Used this way, SHAP, LIME, and attention maps are not oracles. They are instruments, and like any instrument, they reward the person who knows what the needle is actually measuring.
