Skip to content

Explainable AI Without Pretending the Model Is Transparent

Large language models are often described as black boxes. That is partly right: it is rarely possible to point to one internal calculation and say with confidence, “this is why the model reached that conclusion.”

But an opaque model does not require an opaque system.

Individual model computations may be difficult to inspect. System behavior can still be made observable, testable, and explainable.

The architecture that separates sources, observations, evidence, knowledge, suggestions, and decisions is described in From Data to Knowledge. This page asks a narrower question: what does that separation make possible when a user wants to understand or challenge an AI-assisted result?

Explanation begins with the conditions of the decision

A useful explanation starts with what the system actually had available.

Spillway can preserve information such as the message/thread context supplied to a step, deterministic facts, relevant prior decisions, the model and prompt version, the processing stage, and the sources of any contextual knowledge.

That does not expose every neural computation. It establishes the informational conditions under which the result was produced.

A generated rationale is not provenance

Asking a model to explain its own answer can be useful, but a fluent rationale may be incomplete or reconstructed after the fact.

The explanation itself is another model output.

A stronger system-level explanation combines:

  • what was observed directly;
  • what deterministic resolution established;
  • what the model proposed;
  • which sources and versions were involved;
  • what evidence supported or contradicted the result;
  • what uncertainty remained; and
  • what a person later confirmed, corrected, or rejected.

That evidence trail is more useful than pretending the model's internal reasoning has become transparent.

Provenance and influence answer different questions

Provenance asks where a claim came from. Influence asks why it mattered as much as it did.

A complete explanation may therefore need to say not only that a claim came from a model, rule, Reference Source, or human action, but also why that evidence was treated as reliable, specific, recent, independent, or authoritative for the decision at hand.

Those dimensions should not be collapsed into one universal confidence number.

Explanations should match the consequence

Not every suggestion needs the same depth.

A low-stakes newsletter category may need only a short account of the decisive cues. A possible hard deadline or consequential workflow action should expose more of the supporting evidence, conflict, and uncertainty.

The goal is not maximal detail. It is enough detail for the user to understand the basis of the decision and decide whether to trust or correct it.

Corrections are part of explanation

An explanation is incomplete if the user cannot act on it.

Useful correction paths distinguish among cases such as:

  • the conclusion is right;
  • this message is an exception;
  • the source fact is wrong;
  • the evidence was right but overweighted;
  • the model interpretation was wrong; or
  • the conclusion used to be right but is no longer current.

Those are different errors and should not all produce the same learning response.

Context and trajectories should survive explanation

If a conclusion depends on several contexts, the explanation should preserve them rather than flattening them into one score.

For example, “opened on phone, explicitly deferred, then opened from Work on Mac and replied” is more informative than “high engagement.” The first sequence shows awareness, handoff, device opportunity, and eventual action. The second hides the structure that made the behavior meaningful.

From Evidence to Action develops the behavioral and device-context implications of this distinction.

Explainability also supports auditing

Once the system preserves the conditions under which decisions were made, developers can ask empirical questions:

  • Which models perform best on which stages?
  • Which sources or prompts produce concentrated errors?
  • Does reliability differ by category or workflow?
  • Are apparently independent cues actually redundant?
  • Do newer observations supersede stale evidence appropriately?
  • Does performance change after a model or software update?

This is explainability at the system level: an evidence trail that supports diagnosis and improvement over time.

What Spillway can and cannot claim

Spillway cannot promise to expose every internal reason a neural model produced a token.

It can aim to show what went in, what came out, where the supporting evidence came from, what uncertainty or conflict remained, how the result affected the product, and how the user can correct it.

That is a more modest claim than making the model transparent—and a more useful one.

From Data to Knowledge · Computational Fallacies · References and Further Reading

Documentation provenance: Iterative human–AI construction. See Documentation Provenance.