Explainable AI Without Pretending the Model Is Transparent

Large language models are often described as black boxes.

That description is partly right. A model contains billions of learned parameters, and it is rarely possible to point to one internal calculation and say, with confidence, "this is why the model reached that conclusion."

But an opaque model does not require an opaque system.

Spillway is built around a more practical claim:

Individual model computations may be difficult to inspect. System behavior can still be made observable, testable, and explainable.

The goal is not to invent a story about what happened inside the model. The goal is to understand what information entered the system, what the model produced, how that output was used, and what other evidence supported or contradicted it.

Explanation begins before the model runs

A useful explanation starts with the inputs.

Spillway can record which information was available to a classification or extraction step, such as:

the message text and thread context used;
deterministic facts about senders, links, attachments, dates, and calendar data;
relevant prior user decisions;
the model and prompt version;
the processing stage that owned the conclusion;
the source and version of any deterministic knowledge package.

That does not reveal every neural computation. It does establish the informational conditions under which the output was produced.

Model outputs are observations

Spillway does not have to treat a model response as truth.

A model output can instead be treated as an observation with attributes such as:

what claim it proposed;
which processing stage produced it;
which model and prompt produced it;
how confident or uncertain the response appeared;
which input evidence was present;
whether another model or deterministic rule agreed;
whether a person later confirmed or corrected it;
whether the observation is still fresh.

This is closer to the way a social scientist treats measurement.

A survey response, expert judgment, behavioral observation, or coded text is not automatically the underlying truth. It is data produced under identifiable conditions. Its reliability can be studied. Its errors can be compared. Its relationship to other measures can be examined.

AI output can be handled in the same disciplined way.

Provenance tells us where evidence came from; weight tells us how much it should matter

A complete evidence trail needs both.

Provenance answers questions such as:

Which model, rule, package, or person produced this claim?
Which inputs and software versions were involved?
Was the claim observed directly, inferred, resolved from a knowledge package, or confirmed by a user?

Weight answers a different set of questions:

How reliable has this source been for this task?
How specific is the evidence to this message?
How recent is it?
Is it independent corroboration or a repetition of the same signal?
How certain is the claim?
Does stronger contrary evidence exist?

These dimensions should not be collapsed into one universal confidence score. A deterministic parser may be almost perfectly reliable about a narrow fact while saying nothing about the larger decision. A language model may offer a useful interpretation while remaining uncertain. A human action may carry high authority but still represent an exception rather than a durable general rule.

Research on human beliefs offers a useful analogy. Josh Pasek and colleagues have shown that an incorrect answer held uncertainly is substantively different from a confidently held misperception. More recent certainty-weighted work models belief content, certainty, and influence as related but distinct parts of judgment. For an explainable AI system, the corresponding lesson is that identifying a claim is not enough: the system should also preserve why it received the influence it did.

The psychological-data analogy

We often cannot know precisely why another person reached a conclusion.

Instead, we study:

what information they encountered;
what judgment they expressed;
how confident they were;
whether similar judgments were accurate in the past;
how their judgment changed when the information changed;
how their answer compared with other observers;
whether the decision was stable or sensitive to framing.

Those observations support strong inferences without pretending we can inspect every mental process.

The same principle can improve AI systems.

Rather than asking only, "Why did the AI say this?", we can ask a richer set of questions:

What evidence was supplied?
Which parts were deterministic facts and which were interpretations?
Which stage produced the claim?
Did other evidence agree?
How much weight did each source receive, and why?
Has this model been reliable for this kind of task?
Did a person confirm or reject the result?
Would the conclusion change if one input were removed?
Is the output still current?

These questions do not make the model internally transparent. They make the system scientifically inspectable.

Communication context is part of the evidence

The same person can communicate differently across a survey, social-media post, email, meeting, calendar response, or document edit.

That does not necessarily mean one expression is authentic and another is false. Different platforms invite different audiences, lengths, norms, timing, and forms of behavior. They therefore reveal different constructs.

Research by Josh Pasek and colleagues comparing Twitter sentiment about a president with survey measures of presidential approval found that the two were not interchangeable. They could move together in broad, long-term ways while telling different stories at a more granular level or over shorter periods.

That principle matters for explanation. If a future Spillway conclusion combines an email request, a calendar acceptance, and a task completion, the explanation should preserve which source contributed which kind of evidence. “The user cares about this project” is more credible when the system can show the distinct behaviors behind the inference instead of presenting several context-dependent traces as if they were copies of the same fact.

Explanation is more than a generated rationale

Asking a model to explain its own answer is useful, but insufficient.

A fluent rationale may be persuasive even when it is incomplete or reconstructed after the fact. The explanation itself is another model output and should be treated accordingly.

A stronger explanation combines several kinds of information:

Direct observations — what was literally present in the message or artifact.
Deterministic resolutions — what stable parsing and lookup established.
Model proposals — what the AI inferred from the available context.
Provenance — which component, version, and inputs produced each claim.
Weight — why some evidence influenced the result more than other evidence.
Agreement and conflict — which other signals supported or challenged it.
Human action — whether a person confirmed, corrected, dismissed, or overrode it.

That kind of explanation can remain honest even when the model's internal computation is unavailable.

Explanations should match the decision

Not every output needs the same level of scrutiny.

A suggested newsletter category may need only a short explanation:

Spillway recognized the sender and recurring subscription language, then suggested Newsletters based on similar messages you previously confirmed.

A more consequential suggestion should expose more:

Spillway identified an explicit date, interpreted nearby action language as a deadline candidate, and found a related request in the thread. This remains a suggestion because the date's role is uncertain.

The explanation should reveal the evidence, relative weight, and uncertainty relevant to the decision rather than overwhelm the user with implementation detail.

Corrections are part of explainability

An explanation is incomplete if the user cannot act on it.

Spillway is being designed so people can distinguish among several responses:

this suggestion is right;
this message is an exception;
this source is usually interpreted differently;
this fact is wrong;
the evidence was right but the conclusion was wrong;
the conclusion was once right but is no longer current.

Those corrections improve the system and also reveal where the error occurred.

A mistaken deterministic fact, an unreliable model inference, an overweighted weak signal, and an outdated user rule require different remedies. Preserving those distinctions is essential for meaningful learning.

Auditing behavior over time

A system that records provenance and weight can be evaluated empirically.

Developers can ask:

Which models perform best on which stages?
Which deterministic facts most improve accuracy?
Which evidence sources receive too much or too little influence?
Which prompts generate unstable conclusions?
Which source types produce the most corrections?
Do newer observations appropriately supersede older ones?
Are errors concentrated in particular accounts, languages, platforms, or workflows?
Does a model remain calibrated after software or model updates?

This is explainability at the system level: not a single attractive sentence, but an evidence trail that supports diagnosis, comparison, and improvement.

What Spillway can and cannot claim

Spillway cannot promise to expose every internal reason a neural model produced a token.

It can aim to show:

what went in;
what came out;
which component produced it;
what deterministic and probabilistic evidence supported it;
how much influence different evidence received and why;
how the result affected the product;
what uncertainty or conflict remained;
how the user can correct it;
whether the same process behaves reliably over time.

That is a more modest claim than making the model transparent.

It is also a more useful one.