Skip to content
tcpsystems
Version: Current

Search Architecture

Search Architecture

Section titled “Search Architecture”

Status as of 2026-11-05: Search is not operational. It is marked as “dead until restored”. This page describes the intended design for restoration work.

This page covers the planned search pipeline. It follows Blueprint 06 and estate egress law.

Pipeline

Section titled “Pipeline”

The planned flow is:

  1. A user sends a query to the gateway.
  2. The federation layer picks sources.
  3. Fetch and extract collect and parse content.
  4. Rerank scores results with Reciprocal Rank Fusion (RRF).
  5. Cited synthesis builds an answer with source links.
  6. All outbound traffic uses the massprox proxy pool.

Federation

Section titled “Federation”

The federation layer routes each query by:

  • Source type (web search, OSINT tools, breach lookups)
  • Operator permissions
  • Egress limits (which proxy can reach which host)
  • Content class (public, licensed, breach data)
Source Type Notes Status (2026-11-05)
Web search SearXNG or websurfx style source Not deployed (as of 2026-11-05)
OSINT tools RDAP, Wayback, Common Crawl, urlscan Cataloged, not deployed (as of 2026-11-05)
Breach lookups HIBP k-anonymity and approved API clients Implemented, not deployed (as of 2026-11-05)
Corporate sources Internal knowledge bases Not implemented (as of 2026-11-05)

Fetch and extract

Section titled “Fetch and extract”

Fetch and extract do these jobs:

  • Translate the user query for each source.
  • Pick a proxy from the massprox pool.
  • Retrieve content.
  • Parse HTML, JSON, or other formats.
  • Add metadata such as time, source, and confidence.

Rerank

Section titled “Rerank”

Rerank uses Reciprocal Rank Fusion (RRF).

Parameter Default Meaning
k 60 RRF normalization factor
weights Source-specific Weight per source type
cutoff 100 Max results to rank

Cited synthesis

Section titled “Cited synthesis”

Cited synthesis must:

  • Link each factual claim to a source.
  • Show confidence based on source quality and recency.
  • Keep original context where useful.
  • Prefer higher quality sources.
  • Remove duplicate content.

Massprox integration

Section titled “Massprox integration”

Massprox provides the proxy pool. It enforces egress rules for search traffic.

Proxy checks include:

  • TCP dial
  • HTTP or HTTPS verify
  • Latency and success scoring
  • Recheck of verified proxies

See Massprox Architecture.

Egress law and safety

Section titled “Egress law and safety”

Search and massprox must follow egress law.

Required:

  • Route outbound traffic through the controlled gateway and proxy pool.
  • Block IPv6 egress for the search UID.
  • Use only approved DNS resolvers (1.1.1.1, 9.9.9.9).
  • Enforce proxy eligibility and operator permissions.

Not allowed:

  • Direct egress that bypasses controls.
  • Probing restricted addresses.
  • Unsafe network scans or tests.
  • Credential or secret exposure in docs or logs.

See Verify Egress Compliance.

Configuration boundaries

Section titled “Configuration boundaries”

Runtime settings must be editable at runtime. They must be stored in PostgreSQL suite with audit trails. Bootstrap DSN and secret references may use environment variables. Do not store runtime config in files, JSON, YAML, or SQLite in touched areas.

See Search Configuration and Massprox Configuration.

Restoration status (dated)

Section titled “Restoration status (dated)”
Component Status (2026-11-05) Evidence
Federation layer Not deployed (as of 2026-11-05) Design only
Fetch/extract Not deployed (as of 2026-11-05) No active pipelines
Reranking Not deployed (as of 2026-11-05) RRF defined, not active
Cited synthesis Not deployed (as of 2026-11-05) No active engine
Massprox pool Partially present (as of 2026-11-05) 206,808 candidates, 16 verified (2026-10-20 snapshot)
Egress enforcement Active on target host (as of 2026-11-05) IPv4 pinning and IPv6 blocking reported

Restoration pointers

Section titled “Restoration pointers”
  1. Restore and verify massprox first. Confirm egress guard tests pass.
  2. Deploy federation with source and permission checks.
  3. Deploy fetch and extract with proxy routing.
  4. Deploy rerank with RRF settings from suite.
  5. Deploy cited synthesis with attribution rules.
  6. Add monitoring, audits, and runbook drills.

See Search Restoration Runbook and Massprox Operator Runbook.

Section titled “Related pages”

References

Section titled “References”
  • Blueprint 06: Search Federation Architecture (design reference)
  • Estate canon: egress law
  • tcpsystems/search and tcpsystems/massprox repositories (operator context)