Why Spillway uses local AI

The basic idea

Spillway uses artificial intelligence to help sort email, summarize messages, identify requests and dates, and reason about what may deserve attention.

For the AI to do that work, the relevant content has to be processed somewhere.

There are two broad options:

• Local AI runs on hardware you control.
• Cloud AI runs on infrastructure operated by another company and reached over the internet.

Spillway is designed to make local AI a practical option, especially for email that may contain private, confidential, or institutionally restricted information.

What “local” means

When Spillway uses a local AI model, the text being analyzed is sent to an AI service running on your Mac, such as Ollama.

In the normal local setup, that request does not need to go to OpenAI, Anthropic, Google, or another cloud AI company.

A local endpoint can also be configured to run elsewhere, but once the endpoint is another computer or hosted service, the privacy boundary is that endpoint—not the Mac merely because Spillway calls it through the same interface.

Ollama is a runtime, not the model

Ollama is software that can download and run many compatible language models. Spillway does not depend conceptually on one particular model, and there is no permanent “Spillway model.”

That distinction matters because model recommendations age quickly. New open models appear often, existing models receive new versions and quantizations, and what works well depends on both the task and the Mac running it.

The current setup is therefore:

1. install Ollama from ollama.com;
2. choose and install a compatible model from the Ollama model library;
3. confirm that Ollama is running;
4. configure Spillway to use the model and accounts you intend.

Ollama normally listens on:

http://localhost:11434

“Localhost” means the service is on your own computer.

Choosing a model: capability is only one dimension

It is tempting to ask simply, “What is the smartest model my Mac can run?” That is usually the wrong operational question.

A useful local model has to balance several things:

• capability — how well it performs on the kinds of reasoning and extraction Spillway asks it to do;
• memory use — whether the model and its working context fit while leaving enough headroom for macOS and other apps;
• latency — how long repeated calls take;
• throughput — how much work can be completed over time;
• context length — how much text and instruction the model can consider in one request; and
• stability under real use — whether running the model makes the rest of the computer unpleasant.

A larger model that technically loads can still be a poor choice if it leaves the machine under constant memory pressure or makes ordinary interaction sluggish.

Model size, parameters, and quantization

Model names often include a rough parameter count: 3B, 8B, 14B, 32B, and so on. “B” means billions of learned parameters.

Parameter count is not a complete measure of capability, but larger models generally need more memory and computation.

Local runtimes often use quantized versions of models. Quantization stores model weights with fewer bits, reducing memory and storage requirements and often improving speed. More aggressive quantization can introduce some quality loss, so it is another tradeoff rather than a free compression step.

On Apple-silicon Macs, CPU and GPU work draw from the same unified-memory pool. The model, its context, Spillway, macOS, browsers, Xcode, and everything else you are running all compete for that memory.

The practical rule is simple:

Choose a model that leaves the whole computer useful, not merely one that can be forced into memory.

Context length: how much can the model consider at once?

A model's context window is the amount of input and generated text it can consider in a single request.

Longer context can be useful when a task needs:

• a long email thread;
• multiple related messages;
• reference material;
• category definitions and prior decisions; or
• several pieces of provenance-bearing evidence at once.

But context has a cost. Larger configured contexts can require substantially more working memory and computation, and filling the window with irrelevant text can make a model slower without improving the answer.

For local inference, context length should therefore be set to what the workload actually needs rather than automatically maximized. A model advertising a very large theoretical window does not mean every Spillway request should use it.

How to compare current models

Static recommendations become stale too quickly to be trustworthy. For current comparisons, consult a live benchmark or model index and then test promising models on your own hardware and workload.

Useful starting points include:

• the Hugging Face Open LLM Leaderboard and related benchmark collections;
• LM Arena for comparative human preference data across many models; and
• the Ollama model library to see which models and variants are directly available through Ollama.

No general leaderboard is a Spillway benchmark. Chat quality, benchmark scores, memory use, speed, structured-output reliability, and performance on repeated email classification can diverge. The best choice is the model that performs well enough for the relevant Spillway tasks while fitting comfortably on the system.

What local AI does not know

A smaller local model may know less about the general world than a frontier cloud model. That limitation is real.

But general model knowledge is only part of the problem Spillway is solving.

Neither a local nor a cloud model automatically knows:

• which people are in your current course or project;
• which sender identities belong to which real-world relationships;
• what a particular committee or category means to you;
• which messages you previously corrected;
• what work you have already completed;
• which private documents or historical emails matter now; or
• what an institution-specific identifier means inside your environment.

Much of that knowledge is absent because it is private, local, recent, or personal—not because the model is small.

Some missing knowledge is surprisingly easy to teach

Spillway's architecture separates source observations from interpretations. That makes it possible to give a model useful context without pretending the model already knew it.

For example, Spillway can eventually combine things such as:

• deterministic facts extracted from messages and links;
• user-confirmed category meanings and relationships;
• Reference Sources such as a roster supplied by the user;
• known sender associations;
• temporal information; and
• learned patterns whose provenance and authority are preserved.

A model can then reason from those facts. The important point is not that local AI becomes omniscient. It is that high-value private context can compensate for some gaps in generic world knowledge, and a cloud model would not possess that private context by default either.

Knowledge can sometimes cross boundaries even when source content cannot

Different accounts may have different AI policies. A personal mailbox might be allowed to use a cloud or local model while a work account might prohibit sending message text to any AI service.

That does not necessarily mean every useful lesson must remain trapped inside one account.

A user-authored category definition, a high-level workflow preference, or another appropriately abstracted and provenance-bearing conclusion may sometimes be safe and useful across accounts even when the underlying restricted message cannot leave its policy boundary.

The architectural principle is:

Transfer permitted knowledge, not prohibited source content.

This is a direction and constraint, not a claim that every conceivable cross-account learning mechanism is already implemented. Spillway should never use an AI-eligible account as a way to smuggle restricted work content into AI processing.

Why local AI still matters for email

Email often contains much more sensitive information than people realize. A mailbox may include conversations with family and friends, student or employee information, unpublished research, legal or financial discussions, travel plans, private links, attachments, and patterns that are revealing only when combined.

Keeping inference on hardware the user controls can reduce the amount of that information sent to outside services. It can also avoid per-call cloud charges, continue working offline after setup, and give the user direct control over which model version is installed.

The tradeoff is that local inference consumes the user's own storage, memory, power, and compute. Spillway treats those as real product resources rather than pretending local AI is free merely because there is no API bill.

Is cloud AI unsafe?

Not necessarily. Cloud providers can offer very capable models and may be the right choice for some users, tasks, and accounts.

Using cloud AI means selected information leaves the device and is processed under that provider's terms, account settings, data controls, and security practices. Spillway does not assume that every mailbox is permitted to cross the same boundary.

Compare local and cloud AI.

Institutional or managed Macs

An employer or institution may restrict installing software, running local services, or processing particular information with AI at all. A model running locally is not automatically compliant merely because it is local.

Users should follow the policies that govern the data and account involved.

Spillway's principle

Some users may choose local AI for everything. Others may use cloud AI for ordinary personal mail while keeping work or regulated accounts local-only or AI-disabled.

The product is being built so those choices can differ by account and so the surrounding evidence system—not one particular model—carries the durable knowledge.

---

Documentation provenance: Iterative human–AI construction. See Documentation Provenance.