docs.tcp.systems (snapshot runbook)

import StatusPill from ‘../../../components/StatusPill.astro’;

Before you act

• Confirm product and host state first; owner status (reviewed 2026-11-05) marks docs dead until restored.
• Run commands from a clean checkout with node_modules installed and SSH access to the docs host.
• Use the deploy script’s printed rollback command; do not reconstruct rollback steps by hand.

The site is a static Astro build published into an owned subtree of the docs host. The host root is assembled by several generators, so this repository never publishes the root.

Shape

tcp:/srv/docs                     the served root (Caddy file_server)
tcp:/srv/docs/site/               this repository's build (DOCS_BASE=/site/)
tcp:/srv/docs/llms.txt            index of the whole served tree
tcp:/srv/docs/sitemap.xml         one index, one file
tcp:/srv/docs/docs.json           machine inventory of the whole served tree

Local and CI build

The gate is tools/ci.sh. It exits 0 on a clean checkout and non-zero on any broken stage, so it is the proof that the site builds and every reference page matches its source. Run it from the repository root.

bash tools/ci.sh --skip-deploy    # local: install, build, verify, no deploy probe
bash tools/ci.sh                  # also plans the deploy dry-run to DEPLOY_TARGET

What each stage proves, in order, and the failure it catches:

Stage	Command	Fails on
toolchain floor	node -e version check	Node below 22.23.4
lockfile parity	pnpm install --frozen-lockfile --lockfile-only --ignore-scripts	pnpm-lock.yaml no longer describes package.json
install	npm ci	a missing or stale package-lock.json
build	npm run build	any page that fails to render
manifest tests	node --test tools/live-site/manifest.test.mjs	the index tool regressed
drift gate	node tools/docs-engine/generate.mjs --check	a reference page no longer matches its source (skipped sources are reported, not failed)
copy lint	node --test tools/copy-lint/lint.test.mjs + cli.mjs	an em dash, emoji, a vague error, an unmeasured claim, or reading grade above 12
artifact assertions	[ -f dist/index.html ], dist/_astro, dist/llms.txt byte-equal to public/llms.txt	the build did not emit the routes it must
legacy config parity	npm run check:config	docs.json and mint.json drift apart (informational only; it does not prove the Starlight site)

A stage that exits non-zero fails the whole gate: ci.sh uses set -euo pipefail, so a failed command aborts before the next stage. Read the last ci: FAIL: line to find the stage; re-run that stage’s command on its own to isolate it.

--skip-deploy is the normal local form: the final stage is an rsync dry run that reaches the deploy host. A laptop without host access uses it. A runner under a no-probe rule sets DEPLOY_TARGET= (empty) instead, which also skips the probe without printing the dry run.

Authoritative drift check

tools/live-site/manifest.mjs derives llms.txt and sitemap.xml from a directory that mirrors the served root, so the indexes cannot advertise a page that is absent or omit one that is present. The authoritative check is offline (and therefore a gate, not a network probe):

rsync -a --delete tcp:/srv/docs/ /tmp/docs-live/        # mirror the served root
node tools/live-site/manifest.mjs --root /tmp/docs-live --check

--check exits 0 only when both indexes match the tree exactly, plus these boundary rules:

• llms.txt is missing, or does not match the served tree, is a failure.
• sitemap.xml is missing, or does not match, is a failure.
• a second sitemap (sitemap-*.xml, or any sitemap* that is not sitemap.xml) is a failure: one index only.
• robots.txt that does not point at sitemap.xml is a failure.

The date in llms.txt is normalised before comparison, so a regenerated index with only a newer stamp still passes. A non-zero exit prints manifest: FAIL - <problem> per problem and stops the gate. The tool never writes in --check.

Machine-index regeneration

The two indexes and docs.json describe the whole served tree, not just this build, because the root is assembled by several generators. They are regenerated from a mirror of the served root, never from dist/:

rsync -a --delete tcp:/srv/docs/ /tmp/docs-live/
node tools/live-site/manifest.mjs --root /tmp/docs-live --write   # llms.txt + sitemap.xml
node tools/live-site/manifest.mjs --root /tmp/docs-live --json    # inventory for docs.json

--write emits llms.txt and sitemap.xml (contract artifacts first, then every page). deploy.sh builds docs.json from the same --json inventory. After writing, re-run --check; it must exit 0 before any publish. Do not hand-edit the indexes: tools/deploy.sh writes them on every publish, so an edit outside a deploy is overwritten on the next one.

--verify is the only networked mode and is never part of the gate. It fetches every advertised URL and fails on any non-200:

node tools/live-site/manifest.mjs --root /tmp/docs-live --verify

Deploy

From a clean checkout with node_modules installed:

bash tools/deploy.sh --dry-run    # build, assert the artifact, print the rsync plan
bash tools/deploy.sh              # publish
DEPLOY_VERIFY=host bash tools/deploy.sh   # verify on tcp over loopback

The script refuses to publish when the build’s asset prefix does not match DOCS_BASE or when the prefixed build’s /site/llms.txt is incomplete, backs up the subtree and the three root files it overwrites, regenerates llms.txt, sitemap.xml and docs.json from a mirror of the served root (tools/live-site/manifest.mjs --write --publish-path /site/), checks them before upload and again on the bytes read back from the host, and then verifies every advertised URL answers 200 before printing the rollback. Never hand-edit the three root files: the next deploy overwrites them and the drift check fails.

The deploy contract, in order:

1. Build. DOCS_BASE=/site/ npm run build, then assert dist/index.html, dist/llms.txt and dist/_astro exist, and that dist/index.html references ${DOCS_BASE}_astro/. A prefix that was not applied would request assets from the shared root and 404, so the script fails before uploading.
2. Plan. rsync -azn --delete --delete-excluded (sitemap-*.xml excluded), scoped to /srv/docs/site/. --dry-run prints this plan and writes nothing.
3. Backup. tar czf subtree.tar.gz of the existing subtree plus a copy of llms.txt, sitemap.xml, docs.json and the Caddyfile (with sha256), under /root/backups/docs-deploy-<UTC>/. The script makes the backup before the install, so a failed install still has a rollback.
4. Install. rsync -az --delete --delete-excluded into /srv/docs/site/. Nothing outside that subtree is touched.
5. Re-index. mirror the served root, manifest.mjs --write the two indexes, rebuild docs.json from the same inventory, upload all three root files.
6. Verify. manifest --check offline, then every advertised URL must answer 200 (network, or loopback with DEPLOY_VERIFY=host). The run prints the rollback command and records it here.

DEPLOY_VERIFY=host runs the same checks on the target with the name pinned to loopback (--resolve docs.tcp.systems:443:127.0.0.1), so a runner under a no-probe rule still proves 200 without reaching the public address. Caddy needs no reload: the host serves a static tree with file_server, and the run records the Caddyfile hash to prove it untouched.

Backups go to /root/backups/docs-deploy-<UTC>/ on the host and always contain subtree.tar.gz, llms.txt, sitemap.xml, docs.json, Caddyfile and Caddyfile.sha256.

Rollback

The deploy prints the exact command; it is one of:

# restore the previous subtree and the three root files
ssh tcp "tar xzf /root/backups/docs-deploy-<UTC>/subtree.tar.gz -C /srv/docs && cp -a /root/backups/docs-deploy-<UTC>/{llms.txt,sitemap.xml,docs.json} /srv/docs/"
# or, when the subtree did not exist before this deploy
ssh tcp "rm -rf /srv/docs/site"

The <UTC> is the timestamp in the backup path the deploy printed. A rollback restores the root indexes from the backup’s copies, which are the root-form indexes (the whole served tree), not the artifact’s subtree-only copies. The Astro sitemap-*.xml pair is not part of the served root and is not recreated by a rollback.

Restore a single page

The subtree tarball is the source: extract it, take the file, rsync it back.

ssh tcp "tar xzf /root/backups/docs-deploy-<UTC>/subtree.tar.gz -C /tmp"
rsync -az /tmp/site/ acumen/api/ tcp:/srv/docs/site/acumen/api/

Verification

After any deploy, prove the served tree matches what was published:

1. Index parity (offline, on the mirror): manifest.mjs --check must exit 0.
2. URL reachability (network or loopback): manifest.mjs --verify must print ok for every page and artifact, exit 0.
3. Named contracts (deploy step 6): /, /llms.txt, /sitemap.xml, /docs.json, /site/, /site/acumen/api/, /site/consumer/keys/, /site/operations/toolplane/, /acumen/api/, /openapi/acumen-v1.openapi.json each answer 200.
4. No second sitemap: ssh tcp 'ls /srv/docs/site/sitemap-*.xml' returns nothing; tools/deploy.sh excludes those and --check fails if one appears.

A fresh mirror diffed against a clean DOCS_BASE=/site/ build should be identical after normalising three build-stamped values: the “Last updated” <time datetime>, the astro-island uid, and the Pagefind shard filenames. Any other byte difference is a real divergence to diagnose.

Incident

1. curl -sS -o /dev/null -w '%{http_code}\n' https://docs.tcp.systems/site/'; a 200 means the tree is served and the problem is one page.
2. ssh tcp 'systemctl is-active caddy'; a stopped Caddy serves nothing, including the pages that were fine.
3. ssh tcp 'ls -la /srv/docs/site | head'; an empty or partial subtree is a failed rsync, not a bad build; roll back first, diagnose after.
4. ssh tcp "ls -t /root/backups/docs-deploy-* | head -1"; the most recent backup is the rollback target. Take it before any further deploy.

For a stale or wrong reference page (the contract moved but the page did not), see Source drift, not this rollback: regenerating the page is the fix, and a content rollback to an older build would only re-serve the stale page.

Source drift (reference pages)

The reference pages (reference/api, /cli, /config, /errors, /rpc) are generated from the service repositories and gate-checked: when a source changes and the page does not, tools/ci.sh fails. The fix is to regenerate from a clean source, never to hand-edit the page. See Source drift for the full procedure and its safety boundary.

Rotate a key

There is no credential in this repository or in the published tree. Publishing uses the operator’s SSH key; the site contains no secret and no token. A page that needs a credential references its path, never its value.

Add a provider

Not applicable: the site publishes static pages and calls no provider. A new page is a directory change plus a sidebar entry in astro.config.mjs, and the gate (bash tools/ci.sh) proves it builds and is reachable.

Next steps

• Check product status before treating the docs surface as available.
• For gateway/operator procedures, use the acumen runbook.