Console runbook

Use this runbook when console is unreachable, sign-in fails, or an operator action returns an error. Record the date, route, HTTP status, and x-request-id for every incident.

Baseline checks

Run the public probes first:

curl -sSI https://console.tcp.systems/
curl -sS https://console.tcp.systems/healthz
curl -sS https://console.tcp.systems/readyz

Observed 2026-11-05 from this workspace (exit code 0):

• / returned 302 to /login.
• /healthz returned 200.
• /readyz returned 503 with {"ready":false,"service":"tcp-console","oidc":true,"origin_bound":false,"probe_ms":1}.

Treat those values as snapshots. Compare with the current response before acting.

Failure: root redirects but authenticated access fails

Symptoms

• Browser reaches /login.
• After Pocket ID sign-in, the callback returns 403, redirects back to login, or never establishes a session.

Diagnosis

1. Confirm the user completed Pocket ID sign-in and returned to /auth/callback (CONSOLE-BFF-CONTRACT.md:31,48-49; src/lib/server/session.ts:128-168).
2. Check for {"error":"insufficient_scope"}. The callback denies sessions with no resolved console scope (src/lib/server/session.ts:159-163).
3. Verify the deployment has the OIDC bootstrap values it needs: issuer, client id, client secret, redirect URI, and session key. Missing or invalid bootstrap config causes the auth flow to fail closed (src/lib/server/session.ts:97-101; src/lib/server/env.ts:13-20).
4. Check that the redirect URI matches the public callback path of the deployment you are configuring (snapshot 2026-11-05). Console contract docs record https://ai.tcp.systems/console/auth/callback as a redirect expectation in one historical contract context (dated 2026-11-05); use the OIDC client configuration of the deployment you check (snapshot 2026-11-05) as the source of truth, and record the date checked (CONSOLE-BFF-CONTRACT.md:31,64-66).

What not to do

• Do not copy session cookies, ID tokens, refresh tokens, client secrets, or user data into tickets.
• Do not move OIDC settings into a checked-in config file. Fix bootstrap refs or secret references through the deployment path.

Failure: health is good but readiness is false

Symptoms

• /healthz returns 200.
• /readyz returns 503 or ready:false.

Diagnosis

1. Read the readiness body. It reports OIDC and origin binding status (src/routes/readyz, behavior observed 2026-11-05).
2. If origin_bound is false, the console process is up but cannot reach or is not configured for a required upstream origin. This is not a browser cache problem.
3. Check bootstrap references for upstream bases and server-side tokens (src/lib/server/env.ts:21-29; docs/DEPLOY.md:44-52). These are deployment bootstrap refs, not runtime feature flags.
4. If the failure started after a deploy, compare the stamped version from /healthz with the deployment record. Deploy evidence from 2026-11-03 shows version:"node-70f5fcc" as a dated example (docs/evidence/DEPLOY-MATRIX-2026-11-03.md:76-77).

Recovery

• Restore the last known good deployment or bootstrap reference set through the authorized deployment path.
• Recheck /readyz until ready:true or until the failing field changes.
• This docs task does not authorize host edits, service restarts, or deploys.

Failure: operator action returns 400, 401, 403, 404, or 502

Symptoms

• A page loads but a state-changing action fails.
• The API or UI reports missing idempotency, invalid origin, insufficient scope, route not found, or upstream unavailability.

Diagnosis

1. 400 idempotency_key_required: mutating routes require Idempotency-Key, except analytics ingest (src/hooks.server.ts:36-44,72-74). Retry with a fresh idempotency key only when the client did not send one.
2. 400 or 403 origin failure: the request origin must be same-host (src/hooks.server.ts:73). Check browser origin, proxy headers, and public URL alignment.
3. 401 or redirect to login: the session is missing or expired. Create a new session; the signed cookie is intentionally short-lived (src/hooks.server.ts:49-54; src/lib/server/session.ts:10,165-166).
4. 403 insufficient_scope: the user lacks ops:read or admin for the attempted route (CONSOLE-BFF-CONTRACT.md:88-111; src/hooks.server.ts:56-61).
5. 404 route_not_found or another honest upstream refusal: confirm whether the action answers in the deployed upstream (as of 2026-11-05). Pending console matrix actions must pass through acumen’s refusal rather than pretend success (CONSOLE-BFF-CONTRACT.md:128-136). Deploy evidence recorded an honest /v1/ops/audit 404 on 2026-11-03 (docs/evidence/DEPLOY-MATRIX-2026-11-03.md:91-99).
6. 502 or api_unavailable: check /readyz, upstream health, and origin binding. Do not replace the response with cached or sample data.

Escalation data

Provide the route, HTTP status, error code, request id, UTC timestamp, console version from /healthz if available, and the relevant source path. Do not provide secrets or customer data.