Fleet dispatch quality eval harness, spec

Status: spec draft, armed 2026-10-21 by orchestrator O4. No implementation. Owner: tcpsystems ops (the harness). Scope: a lightweight eval harness that grades dispatch quality per task type with rubrics, in the pass@k style the harness already uses.

This PR is documentation only. It adds this file and nothing else: no runner, no eval definitions, no rubric template files. The proposal inside it extends the evals definition schema by one optional key and one validation rule; the diff does not change the schema because the diff contains no code.

Conclusion first

The existing eval runner (harness/evals.mjs) is code-graded only: every check is a shell command with an expected exit code. Dispatch quality needs a verifier that scores a completion against a rubric, because a gate exit code cannot judge “was the plan coherent” or “did the review catch the planted defect”. This spec proposes a model-graded path beside the code-graded path, reuses the runner’s pass@k and pass^k semantics, and leaves every existing code-graded definition valid and unchanged.

Why this is the right shape

The dispatch standards state the two facts this harness encodes:

• Verification is the number one failure mode; self-correction without a verifier degrades.
• Sampling and best-of-N are only sound where a verifier selects.

A graded rubric makes the verifier explicit: one model (DeepSeek Flash, the reviewer tier in the dispatch policy) scores each sampled completion on named axes, and pass@k means at least one sample crossed the rubric threshold. Code-graded evals stay for things a command can check; model-graded evals cover the reasoning and planning tasks a command cannot.

Schema extension

Keep the current definition intact and add an optional grader object beside checks. A definition carries exactly one of checks (code-graded) or grader (model-graded).

The snippet below is illustrative. It is not added under evals/ by this PR, and the current check would reject it today (checks[] required at harness/checks/evals.mjs:21); it becomes a valid definition only when the one-of rule in Acceptance lands beside it.

{
  "id": "dispatch-planning",
  "kind": "capability",
  "title": "Planning outputs satisfy the scaffold rubric",
  "grader": {
    "model": "deepseek/deepseek-flash",
    "rubric": "planning",
    "task": "given the spec, produce an ordered increment plan",
    "axes": [
      { "id": "coverage", "weight": 1, "max": 4 },
      { "id": "order", "weight": 1, "max": 4 },
      { "id": "no-invented-facts", "weight": 2, "max": 4 },
      { "id": "cost-ceiling-named", "weight": 1, "max": 4 }
    ],
    "passThreshold": 0.7,
    "reference": "docs/fixtures/planning-reference.md"
  },
  "threshold": { "passAtK": 1 }
}

The example carries the same four axes as the planning rubric below, so the worked example and the template cannot drift apart. Weights sum to 5 and each axis maxes at 4, so the weighted maximum is 20; passThreshold 0.7 is a weighted score of at least 14 of 20, which is reachable exactly because every axis score is an integer.

The reference field points at a fixture that does not exist yet. It is to be created with the runner; until then the fixture is a proposal, not a citation of an existing file.

The axes are graded 0 to 4 by the verifier against the task and the reference. A sample passes when its weighted score over max reaches passThreshold. pass@k is “at least one of k samples passed”; pass^k is “all k passed”, used by regression evals as it is today.

Per-task-type rubrics

Four rubric templates are proposed here, not shipped. This PR adds no template file, so nothing in the tree yet carries these axis lists; they are the reviewer-facing draft that the runner work will turn into files.

• planning: coverage, order, no invented facts, cost ceiling named.
• reasoning: premise cited, chain valid, conclusion hedged, unknown labelled.
• code: acceptance encoded, minimal diff, gate green, no silent no-op.
• evaluation: defect caught, severity right, evidence cited, no false green.

Each rubric names the defect class it is built to catch. The planning rubric’s fourth axis exists because a plan that ignores its own cost or budget is the defect the dispatch policy keeps producing; leaving it out of the rubric is what made the earlier draft’s example and template disagree.

Determinism for a graded pass^k

pass^k means “all k samples passed”, and the harness reuses it for regression evals at passPower: 1. On the code-graded path that is sound: a shell command returns the same exit code for the same tree. A model-graded path is stochastic, so a graded regression eval can flake against a pass^k = 1 gate unless determinism is bounded. The bound is part of the proposal, not an implementation detail:

• Pin the verifier. Record the exact provider model id and the provider’s model revision string in the result. A verifier version change invalidates earlier graded results, so every graded regression eval re-baselines on the pin change.
• Sample and grade at temperature 0 with a fixed seed, and record the seed with the result. If the provider ignores temperature or seed, the definition must set a nondeterministic flag and may then be graded as pass@k only; it may never back a pass^k = 1 gate.
• Grade each sample three times and take the per-axis median as the axis score. A disagreement that straddles the pass threshold (one grade above, one below) is a fail, not a pass: the graded path fails closed.
• Re-grade the same sample set twice. If the two passes disagree on any sample’s pass/fail, the eval reports flaky and the gate exits 1, so a flaky grader cannot present as a green regression.

The code-graded path keeps its existing behaviour; none of these rules touch it.

Tiny runner

The only implementation this spec commits to is a new runner, evals-grade.mjs, sitting beside harness/evals.mjs at the harness root. It is to be written; it does not exist in this tree. It loads a definition with a grader, samples k completions, sends each to the verifier with the rubric, parses the scores, and reports pass@k and pass^k in the same format as harness/evals.mjs. It never grades with the model under test. The verifier call and the rubric parsing are the two seams to implement first; both are fenced behind the existing zero-runtime-dependency rule.

Acceptance

• The diff is docs only, and no existing eval definition changes meaning: every definition that carries checks still validates exactly as it does today.
• The evals check (harness/checks/evals.mjs) gains exactly one rule: a definition must carry exactly one of checks or grader. The checks[] required assertion at line 21 becomes a one-of assertion. There is no other change to that check, and “unchanged” is not claimed for it.
• The runner’s pass@k and pass^k semantics are reused unchanged.
• The verifier model is always the reviewer tier, never the model being evaluated, and never a free seat.
• A graded definition carries the determinism fields from the section above, or it is marked nondeterministic and barred from pass^k.

Open questions

• Whether a graded definition is a new kind value or reuses capability. Leaning: reuse capability with a grader present, to avoid touching the regression contract.
• Whether scores are stored beside the existing eval results or only reported. Leaning: report only in the first cut; storage is a follow-up.
• Owner of the four rubric templates: this spec proposes them; a reviewer approves the axis wording before implementation, and the files land with the runner.