Ten31 Signal Engine (pilot)

A recurring pipeline that ingests a growing corpus of audio (podcasts, YouTube) and text (SEC filings, earnings-call transcripts), extracts structured propositions, and surfaces signal over time, filtered through Ten31's thesis as a relevance lens — with every surfaced signal logged as a falsifiable prediction so the system is scored against reality.

Spec / source of truth: ten31-signal-engine-handoff.md. It wins over this README on any conflict. Section refs below (§) point into it.

---

The spine (do not violate — §2, §5)

Statistics & graph structure NOMINATE candidates; the frontier model only JUDGES and FANS OUT a pre-filtered shortlist. The prediction ledger is the final arbiter.

• Job A — Discovery (§1): surface what Grant doesn't yet see — emergent themes (independent cross-cluster convergence, scored on acceleration not size), credible contrarian minority stances, and the prize: their intersection (a consensus about to flip).
• Job B — Conviction-action gap (§1.1): fan held convictions to 2nd/3rd-order derivatives and fire conviction(high) × exposure(low) × rising independent corroboration. Countermeasure to the 2023 "power is the binding constraint" call where the seed was held but derivatives under-acted.
• Nominate-then-judge boundary: geometry/stats emit a shortlist that already cleared a quantitative bar; the frontier model judges/synthesizes/fans-out only. A fanned-out derivative is a hypothesis, not a signal, until independent corpus corroboration confirms it (§2.3, §4.6).
• The lens tags relevance; it must NOT gate truth (§5.7). The engine must be able to surface an accelerating signal against Ten31's thesis (the B1–B3 breakers). thesis_seam is a tag, never a filter.

---

Locked pilot decisions

Topic	Decision	Ref
Packaging	Ship as a StartOS 0.4.0 s9pk (operator call). Code is package-ready; TS wrapper generated once the daemon/UI/dependency contract stabilizes.	§10, §13
Earnings transcripts	FMP Ultimate transcript API as backbone (audio isn't reliably fetchable: no uniform feed, ~30–90d replay expiry). EDGAR filings are the durable core. Self-transcribed audio deferred.	§4.1, §12
Conviction exposure	Coarse NAV bands: none / lt2 / 2to10 / gt10. Grant sole editor, monthly review.	§3.1
Topic vocabulary	Hybrid: seeded controlled list + emergent topics batch-merged on a schedule.	§4.2
Frontier model	Default claude-opus-4-8 (override via FRONTIER_MODEL). Scrubbed-vs-unscrubbed quality A/B is a pilot deliverable.	§4.6

---

Architecture (modules map 1:1 to spec layers)

ingest/        §4.1  RSS/YouTube/EDGAR/FMP fetch + scheduler; long-audio chunking; speaker stitch
extract/       §4.2  local LLM → 0..N structured claim units per chunk (willing to emit ZERO)
embedstore/    §4.3  embed DISTILLED propositions + client-side BM25 → Qdrant hybrid
signals/       §4.4  cluster acceleration · stance distributions · bridge edges · scoring
independence/  §4.5  source graph + voiceprint library → convergence discounting
frontier/      §4.6  judge · synthesis · conviction fan-out (scrub→frontier→rehydrate; SHORTLIST/SEEDS only)
ledger/        §4.7  dual-evaluation ledger + conviction log + earned credibility
spark/         §13   THE single chokepoint for all Spark Control HTTP (no other module knows the URL)
store/               SQLite schema + seed loaders
backfill/      §13.4 client-side GPU-hours queue (extraction = heavier serial load; audio sequential)

Two invariants enforced in exactly one place each:

• All gateway HTTP funnels through spark/ — spark/client.py holds the base URL, the self-signed TLS skip, 503 retry/backoff, and a process-wide audio lock (sequential audio, §4.1).
• The "model never sees Grant's rating first" rule (§6.7) is structural: ratings live in a separate human_evaluations table; the model-facing code reads ledger, which has no grant_rating column.

Data stores

• SQLite (store/schema.sql): sources, documents, claims, topics, source_edges, voiceprints, conviction_log, fanout_nodes, ledger, human_evaluations, backfill_jobs. The whole system state is a SELECT.
• Qdrant (via Spark Control): one propositions collection, hybrid dense bge-m3 (1024-d) + sparse BM25 (Qdrant/bm25, modifier: idf). Points are distilled propositions, never raw chunks.

---

Backfill queue (§13.4) — measured in GPU-hours, not real-time

Scheduler jobs are producers (fetch/dedup/enqueue); a single worker drains the GPU queue one job at a time → no parallel audio → no 503 by construction. Jobs are leased (crash-safe resume), idempotent on hash(content + prompt-version). Extraction is the binding load. Rough pilot estimate (~300 episodes + 25 companies): transcription ~6 GPU-h, extraction ~90 GPU-h, wall-clock ~4–6 days (audio on Spark 2 and extraction on Spark 1 run in parallel). The queue self-calibrates from measured gpu_seconds.

---

Build order (§11) & status

1. ✅ Foundation — config, spark/ client, SQLite schema, conviction-log seed, ledger scaffold live day one.
2. ✅ Ingestion + backfill queue — backfill queue; EDGAR + FMP earnings; audio path live-proven (download → Parakeet transcribe + Sortformer diarize → align → voiceprint-stitch → speaker-attributed transcript); 20 podcast feeds resolved (seeds/podcast_feeds.resolved.yaml). Broad corpus ingested: 785 company docs (379 filings + 406 earnings), 802 extract jobs. ⬜ Remaining: podcast audio backfill (RSS-full + dated YouTube pulls) + audio-cache cleanup; foreign-filer forms (20-F/6-K for CCJ/TSM/IREN).
3. ✅ Extraction worker + §4.2 prompt — live on text (earnings: 26 claims, insight-heavy) AND audio (Dwarkesh: 5 claims, correct speaker attribution). Backfill draining in background (logs/extract-backfill.log). ⬜ Refinement: target Item 7 MD&A for filings.
4. 🟡 Embedding + storage — ✅ Qdrant propositions hybrid collection (bge-m3 + BM25) live; hybrid search+rerank verified surfacing the power-infra theme. ⬜ embed the growing claim set; clustering.
5. 🟡 Scoring brain (signal_engine/signals/) — ✅ EISC independence primitive (verified), as-of harness, windowed acceleration, under-acted-conviction (Job B), the quantitative bar, ledger writer, resolver stub, orchestrator. Speaker-name independence edges live. ⬜ Job A scorers (emergence/stance/intersection) deferred per blueprint.
6. ✅ Source-independence graph — voiceprint cosine + speaker-name edges feed EISC; "one guest doing the rounds" collapses to ~1 voice (verified).
7. 🟡 §7.1 backtest — runs end-to-end; corpus-gated finding: power-infra corroboration is currently single-cluster (energy companies), so it correctly won't clear the cross-cluster bar (k_eff≥2). Needs the cross-cluster podcast corpus (transcription backfill running). The scoring is disciplined, not broken.
8. ⬜ Frontier judge + synthesis + live fan-out (Anthropic key live; routed scrub→frontier→rehydrate) — deferred; v1 backtest uses a hand-written fan-out.
9. 🟡 Web UI (signal_engine/ui/, serve) — ✅ corpus management (dashboard, add/view sources, per-source claim inspection). ⬜ human-eval rating interface (§6.7) on the same app.
10. ⬜ Run the §7.1 backtest FIRST, then the forward pilot; disagreement analysis; scaling decision.

Scaling lever (post-pilot): bulk extraction can offload to the Gemini batch API (operator can provide a key) while local Qwen stays default — public corpus only, never conviction/exposure data (§4).

---

Running the foundation

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
DATA_DIR=./data .venv/bin/python -m signal_engine init-db
DATA_DIR=./data .venv/bin/python -m signal_engine seed-convictions
DATA_DIR=./data .venv/bin/python -m signal_engine seed-convictions --file seeds/conviction_log.backtest-2023.seed.yaml
# When the gateway URL is provided:
SPARK_CONTROL_URL=https://<lan-addr> .venv/bin/python -m signal_engine spark-status

Pipeline + UI commands (config from .env):

.venv/bin/python -m signal_engine seed-sources && python -m signal_engine load-feeds
.venv/bin/python -m signal_engine ingest-edgar   --ticker CEG --since 2023-01-01 --until 2023-12-31
.venv/bin/python -m signal_engine ingest-earnings --ticker NVDA --since 2023-01-01 --until 2023-12-31
.venv/bin/python -m signal_engine run-extract --limit 4        # local-LLM claim extraction
.venv/bin/python -m signal_engine embed-claims                 # → Qdrant hybrid collection
.venv/bin/python -m signal_engine search --query "power is the binding constraint on AI"
.venv/bin/python -m signal_engine serve                        # corpus UI → http://localhost:8000

Config is all env-driven (signal_engine/config.py) so the same code runs as a plain process now and as a StartOS daemon later (injected via the s9pk store.json FileModel).

---

Operator-provided items (block runtime, not the foundation)

• Spark Control LAN base URL + TLS cert (same-LAN self-signed → SPARK_VERIFY_TLS=false).
• Confirm the exact §13.2 model strings (currently defaulted: qwen3.6-35b-a3b-nvfp4, bge-m3, parakeet-tdt-0.6b).
• ANTHROPIC_API_KEY (frontier step) and FMP_API_KEY (earnings transcripts) — or defer FMP and start filings-only.
• §7.1 backtest corpus: which 2023-era podcast archives + filings we can actually fetch (the gating risk).
• StartOS host CPU arch (drives which .s9pk to install — the host, not the ARM Sparks) and Spark Control's health-check IDs + outbound-gateway = clearnet (not Tor) — verified against a live box at packaging time.