status: snapshots, probes and incidents

---

import StatusPill from ‘../../../components/StatusPill.astro’;

The status surface at status.tcp.systems is a generated snapshot plus a live event stream. One row per component, one probe behind each row, and an append-only incident feed. A state is only as good as its observation: a row that cannot be probed says so, and does not read as healthy.

Components and probes

A component is one declared surface of the estate. The single declaration is surfaces.json in the tcpsystems/status repository: id, name, host, address, implementation and the probe that is allowed to reach it. docs, chat and radar are class host (infrastructure); the other seven are class product (user-facing).

A probe is the measurement behind a row. Each component declares an HTTP path and the status code that counts as healthy:

• ai probes GET /readyz and expects 200.
• console probes GET /api/health and expects 200.
• studio probes GET / and expects 302, because studiod sits behind the Pocket ID gate. A 200 there would mean the gate is off, so it is a mismatch, not a success.
• search probes GET /search?q=status-probe, not /readyz. Caddy answers every unknown path on that vhost from a static directory, so /readyz returns 200 with zero bytes regardless of the backend.

The probe declaration lives in surfaces.json (probe.kind, probe.path, probe.expect), enforced by gen/status-gen.mjs --check (tcpsystems/status:gen/status-gen.mjs).

Component states

Each row carries one of four states, measured at snapshot time:

State	Meaning	Example
up	The probe returned the expected code with a non-empty body	GET /readyz answers 200 with a body
down	No answer, the wrong code, or an empty 200	502, a timeout, or Caddy’s zero-byte catch-all
stub	The expected code, but a placeholder body instead of a service	The body says service not yet wired; the test fixture covers it (tcpsystems/status:gen/status-gen.test.mjs)
unprobed	Policy forbids the probe, or the row is the artifact itself	The 23.88.44.32 rows carry the no-probe contract

Two honesty rules make a green claim impossible to fake:

1. An empty 200 is down. Caddy’s file_server catch-all answers 200 with zero bytes for any path, which proves the web server is up and nothing about the service. classify() treats any 200 with an empty body as down (tcpsystems/status:gen/status-gen.mjs).
2. A blind spot is stated, not hidden. Rows that resolve to 23.88.44.32 are unprobed with the no-probe reason, and the gate fails if such a row is reported as anything else. status itself is unprobed: a self-probe from the generator cannot distinguish serving from generating, so the freshness badge carries the page’s health instead.

Snapshot freshness: fresh, stale, unknown

A snapshot is a measurement with a timestamp, not a standing claim. Every artifact carries generated and budget_s (600 seconds, one timer interval), and freshness is the reader’s computation against them:

• fresh: the snapshot is at most budget_s old.
• stale: older than budget_s. The badge turns red, because a stopped timer must not keep claiming up.
• unknown: the timestamp cannot be parsed, so no freshness claim is made.

The timer runs the generator every 5 minutes, so a page that keeps generating does not look stale. Stale means the scheduler stopped, and it is a non-green state by design.

Rollup states

The rows roll up into one page state with a fixed precedence:

Rollup	Condition
degraded	Any row is down
unprobed	No rows exist, or every row is unprobed
ok	Every row is up
partial	Everything else: some rows up with at least one stub or unprobed, none down

A partial rollup refuses the word ok on purpose: an unmeasured row or a placeholder row is not a measured healthy surface.

Incidents

Incidents are records, not notices. The feed is operator-curated at tcpsystems/status:state/incidents.json and copied verbatim into /api/incidents.json on each run; a generator run never rewrites or drops an entry. Each entry carries:

• id, opened, closed, status, severity and the affected surface.
• impact: what a user observes.
• evidence: the commands and outputs that confirm the problem. An incident without a source is not written; closing an incident appends the resolution evidence instead of deleting the original.
• detection: why the probe saw it (or why a weaker probe would not).
• owner_action: the next operational step.

Worked example: the 2026-11-02 search incident recorded 502 on the /search handle while /readyz answered 200 from Caddy’s catch-all, named the Caddy rule and the missing listener, and closed with the 200 plus 3,739 bytes of results that proved the backend back (tcpsystems/status:state/incidents.json).

Maintenance

There is no separate maintenance state. Planned work is recorded as an incident entry, so it travels through the same feed with the same evidence rules:

1. Before: open an incident with the planned window, the expected impact and the surfaces touched. Rows affected by the work will move on their own as probes observe the change.
2. During: append evidence as it is observed. Do not edit the HTML or the generated JSON; the next run regenerates them.
3. After: close the incident with the verification commands and their output, and confirm the affected rows return to up on the next snapshot.

Uptime honesty

The page reports measured uptime per snapshot, not a marketing percentage:

• The headline shows up/total from the current snapshot (10 surfaces on 2026-11-05).
• Four of ten rows are declared unprobed by contract, so ok is not the steady state; partial and degraded are honest answers, not failures of the page.
• Unprobed, stub, stale and unknown are never rendered as green. A reader can always tell a measured fault from a declared blind spot and a placeholder row. No row among the ten was stub in the 2026-11-05 snapshot; the state exists so a half-wired surface cannot read as up.

Subscriber interfaces

All three public interfaces come out of the same generator run as the page, so they cannot drift from what the page shows:

• GET /api/status.json: the machine snapshot. Schema, generated, budget_s, the realtime declaration, the rollup, every row with its probe result, and the C1/C2/C4/C6 verdicts.
• GET /api/incidents.json: the incident feed, copied verbatim.
• GET /api/events: an SSE stream that announces a new snapshot with its generated timestamp and rollup. It is a notification channel, not a prober: verdicts come only from the generator run, and a subscriber recomputes freshness from generated.

There is no push or webhook interface for subscribers, and no subscriber data is held by the service.

Reading the page during an incident

1. Read the freshness badge first. A red stale badge means the scheduler stopped; the rows are history, not current state.
2. Read the rollup and the down rows. Each row names its probe, the HTTP result and the byte count, so a fault is reproducible with one curl.
3. Read /api/incidents.json for operator context: impact, evidence and the named owner action.

For operating the service itself (deploy, rollback, restore, a stalled timer), see the status runbook.

Observed on 2026-11-05 from the status service: GET / answered 200, /api/status.json reported rollup degraded with studio and search down, four rows unprobed by contract, and budget_s 600.

This page describes the intended status surface. It is not proof that the status service answers now.

Snapshot (from prior docs)

• The status surface is intended to show component freshness, probe evidence, and an append-only incident feed.
• A missing probe should be labeled unverified, not healthy.

Use fresh verification before relying on status.tcp.systems or incident data.

The page exists because a static landing page can look alive while the service behind it is down. A 200 with an empty body is treated as down, and an unprobed host is labelled unprobed, not green.

When the docs surface itself degrades, the recorded state feeds the runbooks, not the other way: the docs runbooks hold the deploy, rollback and verification procedure, and source drift covers a stale reference page. An incident here names the command that observed it.