Initial commit: Ten31 Signal Engine (ingest, scoring brain, corpus seeds)
This commit is contained in:
Keysat
@@ -0,0 +1,135 @@
|
||||
# Ten31 Signal Engine (pilot)
|
||||
|
||||
A recurring pipeline that ingests a growing corpus of audio (podcasts, YouTube) and text
|
||||
(SEC filings, earnings-call transcripts), extracts structured **propositions**, and surfaces
|
||||
**signal over time**, filtered through Ten31's thesis as a *relevance lens* — with every surfaced
|
||||
signal logged as a **falsifiable prediction** so the system is scored against reality.
|
||||
|
||||
**Spec / source of truth:** [`ten31-signal-engine-handoff.md`](ten31-signal-engine-handoff.md). It wins
|
||||
over this README on any conflict. Section refs below (§) point into it.
|
||||
|
||||
---
|
||||
|
||||
## The spine (do not violate — §2, §5)
|
||||
|
||||
> **Statistics & graph structure NOMINATE candidates; the frontier model only JUDGES and FANS OUT a
|
||||
> pre-filtered shortlist. The prediction ledger is the final arbiter.**
|
||||
|
||||
- **Job A — Discovery (§1):** surface what Grant doesn't yet see — emergent themes (independent
|
||||
cross-cluster *convergence*, scored on **acceleration** not size), credible contrarian minority
|
||||
stances, and the prize: their **intersection** (a consensus about to flip).
|
||||
- **Job B — Conviction-action gap (§1.1):** fan held convictions to 2nd/3rd-order derivatives and fire
|
||||
`conviction(high) × exposure(low) × rising independent corroboration`. Countermeasure to the 2023
|
||||
"power is the binding constraint" call where the seed was held but derivatives under-acted.
|
||||
- **Nominate-then-judge boundary:** geometry/stats emit a shortlist that already cleared a quantitative
|
||||
bar; the frontier model judges/synthesizes/fans-out only. A fanned-out derivative is a **hypothesis,
|
||||
not a signal**, until independent corpus corroboration confirms it (§2.3, §4.6).
|
||||
- **The lens tags relevance; it must NOT gate truth (§5.7).** The engine must be able to surface an
|
||||
accelerating signal *against* Ten31's thesis (the B1–B3 breakers). `thesis_seam` is a tag, never a filter.
|
||||
|
||||
---
|
||||
|
||||
## Locked pilot decisions
|
||||
|
||||
| Topic | Decision | Ref |
|
||||
|---|---|---|
|
||||
| **Packaging** | Ship as a **StartOS 0.4.0 s9pk** (operator call). Code is package-ready; TS wrapper generated once the daemon/UI/dependency contract stabilizes. | §10, §13 |
|
||||
| **Earnings transcripts** | **FMP Ultimate** transcript API as backbone (audio isn't reliably fetchable: no uniform feed, ~30–90d replay expiry). EDGAR filings are the durable core. Self-transcribed audio deferred. | §4.1, §12 |
|
||||
| **Conviction exposure** | Coarse **NAV bands**: `none / lt2 / 2to10 / gt10`. Grant sole editor, monthly review. | §3.1 |
|
||||
| **Topic vocabulary** | **Hybrid**: seeded controlled list + emergent topics batch-merged on a schedule. | §4.2 |
|
||||
| **Frontier model** | Default `claude-opus-4-8` (override via `FRONTIER_MODEL`). Scrubbed-vs-unscrubbed quality A/B is a pilot deliverable. | §4.6 |
|
||||
|
||||
---
|
||||
|
||||
## Architecture (modules map 1:1 to spec layers)
|
||||
|
||||
```
|
||||
ingest/ §4.1 RSS/YouTube/EDGAR/FMP fetch + scheduler; long-audio chunking; speaker stitch
|
||||
extract/ §4.2 local LLM → 0..N structured claim units per chunk (willing to emit ZERO)
|
||||
embedstore/ §4.3 embed DISTILLED propositions + client-side BM25 → Qdrant hybrid
|
||||
signals/ §4.4 cluster acceleration · stance distributions · bridge edges · scoring
|
||||
independence/ §4.5 source graph + voiceprint library → convergence discounting
|
||||
frontier/ §4.6 judge · synthesis · conviction fan-out (scrub→frontier→rehydrate; SHORTLIST/SEEDS only)
|
||||
ledger/ §4.7 dual-evaluation ledger + conviction log + earned credibility
|
||||
spark/ §13 THE single chokepoint for all Spark Control HTTP (no other module knows the URL)
|
||||
store/ SQLite schema + seed loaders
|
||||
backfill/ §13.4 client-side GPU-hours queue (extraction = heavier serial load; audio sequential)
|
||||
```
|
||||
|
||||
**Two invariants enforced in exactly one place each:**
|
||||
- All gateway HTTP funnels through `spark/` — `spark/client.py` holds the base URL, the self-signed TLS
|
||||
skip, 503 retry/backoff, and a process-wide **audio lock** (sequential audio, §4.1).
|
||||
- The "model never sees Grant's rating first" rule (§6.7) is **structural**: ratings live in a separate
|
||||
`human_evaluations` table; the model-facing code reads `ledger`, which has no `grant_rating` column.
|
||||
|
||||
### Data stores
|
||||
- **SQLite** (`store/schema.sql`): sources, documents, claims, topics, source_edges, voiceprints,
|
||||
conviction_log, fanout_nodes, ledger, human_evaluations, backfill_jobs. The whole system state is a SELECT.
|
||||
- **Qdrant** (via Spark Control): one `propositions` collection, **hybrid** dense `bge-m3` (1024-d) +
|
||||
sparse BM25 (`Qdrant/bm25`, `modifier: idf`). Points are distilled propositions, never raw chunks.
|
||||
|
||||
---
|
||||
|
||||
## Backfill queue (§13.4) — measured in GPU-hours, not real-time
|
||||
|
||||
Scheduler jobs are **producers** (fetch/dedup/enqueue); a **single worker** drains the GPU queue one job
|
||||
at a time → no parallel audio → no 503 by construction. Jobs are leased (crash-safe resume), idempotent on
|
||||
`hash(content + prompt-version)`. Extraction is the binding load. Rough pilot estimate (~300 episodes +
|
||||
25 companies): transcription ~6 GPU-h, **extraction ~90 GPU-h**, wall-clock ~4–6 days (audio on Spark 2 and
|
||||
extraction on Spark 1 run in parallel). The queue self-calibrates from measured `gpu_seconds`.
|
||||
|
||||
---
|
||||
|
||||
## Build order (§11) & status
|
||||
|
||||
1. ✅ **Foundation** — config, `spark/` client, SQLite schema, conviction-log seed, **ledger scaffold live day one**.
|
||||
2. ✅ **Ingestion + backfill queue** — backfill queue; EDGAR + FMP earnings; **audio path live-proven** (download → Parakeet transcribe + Sortformer diarize → align → voiceprint-stitch → speaker-attributed transcript); 20 podcast feeds resolved (`seeds/podcast_feeds.resolved.yaml`). Broad corpus ingested: **785 company docs (379 filings + 406 earnings), 802 extract jobs**. ⬜ Remaining: podcast audio backfill (RSS-full + dated YouTube pulls) + audio-cache cleanup; foreign-filer forms (20-F/6-K for CCJ/TSM/IREN).
|
||||
3. ✅ **Extraction worker + §4.2 prompt** — **live** on text (earnings: 26 claims, insight-heavy) AND audio (Dwarkesh: 5 claims, correct speaker attribution). Backfill draining in background (`logs/extract-backfill.log`). ⬜ Refinement: target Item 7 MD&A for filings.
|
||||
4. 🟡 **Embedding + storage** — ✅ Qdrant `propositions` hybrid collection (bge-m3 + BM25) **live**; hybrid search+rerank verified surfacing the power-infra theme. ⬜ embed the growing claim set; clustering.
|
||||
5. 🟡 **Scoring brain** (`signal_engine/signals/`) — ✅ EISC independence primitive (verified), as-of harness, windowed acceleration, **under-acted-conviction (Job B)**, the quantitative bar, ledger writer, resolver stub, orchestrator. Speaker-name independence edges live. ⬜ Job A scorers (emergence/stance/intersection) deferred per blueprint.
|
||||
6. ✅ **Source-independence graph** — voiceprint cosine + speaker-name edges feed EISC; "one guest doing the rounds" collapses to ~1 voice (verified).
|
||||
7. 🟡 **§7.1 backtest** — runs end-to-end; **corpus-gated finding**: power-infra corroboration is currently single-cluster (energy companies), so it correctly won't clear the cross-cluster bar (k_eff≥2). Needs the cross-cluster podcast corpus (transcription backfill running). The scoring is disciplined, not broken.
|
||||
8. ⬜ Frontier judge + synthesis + live fan-out (Anthropic key live; routed scrub→frontier→rehydrate) — deferred; v1 backtest uses a hand-written fan-out.
|
||||
9. 🟡 **Web UI** (`signal_engine/ui/`, `serve`) — ✅ corpus management (dashboard, add/view sources, per-source claim inspection). ⬜ human-eval rating interface (§6.7) on the same app.
|
||||
10. ⬜ **Run the §7.1 backtest FIRST**, then the forward pilot; disagreement analysis; scaling decision.
|
||||
|
||||
**Scaling lever (post-pilot):** bulk extraction can offload to the Gemini batch API (operator can provide a key) while local Qwen stays default — public corpus only, never conviction/exposure data (§4).
|
||||
|
||||
---
|
||||
|
||||
## Running the foundation
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
|
||||
DATA_DIR=./data .venv/bin/python -m signal_engine init-db
|
||||
DATA_DIR=./data .venv/bin/python -m signal_engine seed-convictions
|
||||
DATA_DIR=./data .venv/bin/python -m signal_engine seed-convictions --file seeds/conviction_log.backtest-2023.seed.yaml
|
||||
# When the gateway URL is provided:
|
||||
SPARK_CONTROL_URL=https://<lan-addr> .venv/bin/python -m signal_engine spark-status
|
||||
```
|
||||
|
||||
**Pipeline + UI commands** (config from `.env`):
|
||||
```bash
|
||||
.venv/bin/python -m signal_engine seed-sources && python -m signal_engine load-feeds
|
||||
.venv/bin/python -m signal_engine ingest-edgar --ticker CEG --since 2023-01-01 --until 2023-12-31
|
||||
.venv/bin/python -m signal_engine ingest-earnings --ticker NVDA --since 2023-01-01 --until 2023-12-31
|
||||
.venv/bin/python -m signal_engine run-extract --limit 4 # local-LLM claim extraction
|
||||
.venv/bin/python -m signal_engine embed-claims # → Qdrant hybrid collection
|
||||
.venv/bin/python -m signal_engine search --query "power is the binding constraint on AI"
|
||||
.venv/bin/python -m signal_engine serve # corpus UI → http://localhost:8000
|
||||
```
|
||||
|
||||
Config is all env-driven (`signal_engine/config.py`) so the same code runs as a plain process now and as a
|
||||
StartOS daemon later (injected via the s9pk `store.json` FileModel).
|
||||
|
||||
---
|
||||
|
||||
## Operator-provided items (block runtime, not the foundation)
|
||||
|
||||
- **Spark Control LAN base URL + TLS cert** (same-LAN self-signed → `SPARK_VERIFY_TLS=false`).
|
||||
- **Confirm the exact §13.2 model strings** (currently defaulted: `qwen3.6-35b-a3b-nvfp4`, `bge-m3`, `parakeet-tdt-0.6b`).
|
||||
- **`ANTHROPIC_API_KEY`** (frontier step) and **`FMP_API_KEY`** (earnings transcripts) — or defer FMP and start filings-only.
|
||||
- **§7.1 backtest corpus:** which 2023-era podcast archives + filings we can actually fetch (the gating risk).
|
||||
- **StartOS host CPU arch** (drives which `.s9pk` to install — the host, not the ARM Sparks) and Spark Control's
|
||||
health-check IDs + outbound-gateway = clearnet (not Tor) — verified against a live box at packaging time.
|
||||
```
|
||||
Reference in New Issue
Block a user