Trust Scoring

The scoring layer your dispatcher is missing

At some point a fleet stops being manageable by inspection. You start with one agent, grow to a dozen, and the knowledge of who handles what cleanly lives in your head, built session by session. That knowledge guides every routing decision until the fleet grows past what you can hold in working memory.

The evidence is not missing. It sits in the event log — task completions, escalations, handoffs, all timestamped and typed. What is missing is the layer that surfaces those records at routing time, converts them into something a dispatcher can act on in the moment of assignment, and maintains them across sessions without reconstruction. This book builds that layer.

Seven pieces between the log and the routing decision

• score-store — An append-only file keyed by agent ID where task outcomes accumulate as credit deltas. Delta records are the source of truth; the running total is a precomputed convenience so the dispatcher never sums the whole file.
• ingestion-pipeline — A scheduled duty that reads completed and escalated tasks from the event log and converts them into credits on a watermarked schedule, so nothing is double-counted and nothing is missed.
• tier-thresholds — The configuration that maps a raw credit total to a routing-grade label, plus the pure resolver function that makes the mapping deterministic and testable.
• crossing-detector — Fires a tier_crossed event when a total crosses a threshold — but only after a gate independently re-derives the total from raw records and verifies the crossing. A mismatch suppresses it until the pipeline is corrected.
• deliberation-weighting — A weighted aggregator over a multi-agent deliberation result, giving each response mass proportional to the speaker’s current tier. Quorum reflects the fleet’s actual experience rather than a flat headcount.
• routing-signal — A tier-aware dispatcher that reads both the task’s tier requirement and the tier index before selecting an agent, with a fallback that emits a coverage-gap event when no agent at the required tier is free.
• score-aging — A half-life decay function applied at query time, so older records weigh less and a six-month peak does not permanently define an agent’s standing.

A preview

A dispatcher selecting from twelve available agents has access to whatever the task envelope says and whatever the roster says. The task envelope names the work. A roster entry confirms the agent exists and is available. Neither one says how this agent performed on the last twenty tasks of this type. That record sits in the event log. The dispatcher selects by availability — which is a fact — rather than by track record, which requires a query.

Every artifact here is in production: score store, ingestion pipeline, tier resolver, crossing detector, weighted aggregator, tier-aware dispatcher, aging function. The failure hooks that open each chapter are real failures — the kind that happen before any of this machinery exists.

Operators whose fleet outgrew their memory

Operators whose fleet has grown past what one person can hold in working memory, where routing by remembered track record stops working. The prerequisites are hard: AI Teams (Book 5) for role contracts and task queues, Self-Verifying Pipelines (Book 7) for the event log and gate architecture, and Agent Messaging (Book 11) for the message envelope, the event log schema, and the deliberation protocol the weighting function wraps. Skip any of the three and the code here will not connect.

A longer excerpt is available to newsletter subscribers.

Buy Ebook on Amazon Buy Paperback on Amazon