Handoff: prune Current state to a snapshot; note shared .dockerignore gotcha

AGENTS.md

@@ -103,17 +103,13 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude
 
 ## Current state
 
-_Phase 0 substrate + Phase 1 thesis/outreach built; **box and repo at v0.1.0:86** (deployed & verified live 2026-06-17). **The fundraising grid + email capture is the canonical system of record** (decision 2026-06-16) — vestigial classic-CRM surfaces get pruned/repurposed. Longer-term backlog: `ROADMAP.md`._
+_Phase 0 + Phase 1 built; **box and repo at v0.1.0:86** (deployed & verified live 2026-06-17). **The fundraising grid + email capture is the canonical system of record** (2026-06-16) — vestigial classic-CRM surfaces get pruned/repurposed. Deploy/feature history lives in git log + `start9/0.4/startos/versions/`; longer-term backlog + debt in `ROADMAP.md` / `EVALUATION.md`._
 
 - **Matrix intake bot — DEPLOYED & LIVE (2026-06-17), `backend/matrix_intake/`:** a separate-process bot (its `matrix-nio` dep isolated from the stdlib CRM) turning a typed Matrix-room message into a proposed fundraising-grid add/edit, written only after **in-thread human approval** (`yes`/`edit field=value`/`no`). Parse = local Qwen via Spark Control (no Claude/scrub, like the digest); writes reuse the CRM's own `POST /api/fundraising/log-communication` tagged `source="matrix_intake"`; new-vs-existing via read-only `GET /api/intake/match` (returns the grid row id → no duplicate). **Runs on the Spark as a docker-compose service** (`modelo32`, container `matrix-intake`, `restart: unless-stopped` → survives a reboot; `docker-compose.yml` at the repo root + `backend/matrix_intake/Dockerfile` bundling `backend/matrix_intake` + the stdlib `backend/ingest` Spark client; retired the old nohup launch, 2026-06-17). A spark-control dashboard card is still pending (handoff: `docs/handoffs/add-intake-bot-to-spark-control.md`). **Live-smoked end-to-end** (new-investor create + existing-investor note matched & appended, no dup). Server side shipped to the box as **v0.1.0:84** (`/api/intake/match` + `source` provenance — these were missing on v83, so the bot 404'd until v84); then UX adds: main-timeline nudge pointer, top-level-`yes`→thread redirect, clearer commit wording, note text in the grid line (v85 dropped the `[note]` tag). M3 (business-card photo) deferred (no Spark vision model). Guide: `docs/guides/matrix-intake.md`.
 - **Matrix intake — fuzzy-match + conversational-edit pass — DEPLOYED & LIVE 2026-06-17 (box on v0.1.0:86, bot restarted on the Spark; `candidates` endpoint verified live); Matrix live-smoke still pending.** Closes the two locked post-deploy enhancements (ROADMAP). **(a) Fuzzy matching (server-side, ships in the s9pk):** `find_intake_candidates` in `server.py` (deterministic — stdlib `difflib` name similarity + token-set Jaccard, legal-suffix-aware via `_strip_legal_suffix`, + email Levenshtein ≤ 2; ranked, ≥0.62, top 5); `GET /api/intake/match` now returns `{match, candidates}`. The bot surfaces a numbered shortlist (`_stage="disambiguate"`) so a near-duplicate ("Charlie"/"Charles", "Acme Capital"/"Acme Capital LLC", a one-char email typo) is **confirmed by a human** instead of silently creating a second investor — never auto-attached. **The optional LLM-judge re-rank was deferred** (deterministic filter already surfaces the cases; LLM is the right shortlist *pruner* if noise proves real). **(b) Conversational edits (bot-side, ships on the Spark):** any in-thread reply that isn't `yes`/`no`/`edit field=value` → `parse.revise` re-runs `{proposal + instruction}` through local Qwen and re-renders the card; **email integrity preserved** (a changed address must literally appear in the instruction; the model's email field is never trusted); no-op revisions re-prompt (`same_fields`). **Deploy is split:** the `candidates` need an **s9pk build+install** (v86); the bot's disambiguation+revise need a **Spark `git pull` + restart** — a bot restart alone won't deliver `candidates` (box returns `[]`, bot safely proposes new). Tests green; **needs a Matrix live-smoke** (grammar + Qwen `revise` leg). Guide updated.
 - **Working (all draft-only):** CRM + ingest (chunk→embed→Qdrant + retrieval) + redaction boundary; Gmail capture (DWD) + email-activity propose→approve; Thesis Workshop + Architect (Claude) with dual-approval gate; Outreach Draft Assistant + follow-up radar + per-user voice + Tier-B in-thread Gmail draft creation.
-- **Deployed & verified live: v0.1.0:83** (box `$START9_BOX_HOST`/immense-voyage.local; `installed-version`→`0.1.0:83`, migration chain `…82→83` clean, server up on `:8080`, Gmail + ingest + digest schedulers all started; render-smoke gated the build) — **email search/query + windowed digest preview** (code-only, migrations no-op). Communications tab (`CommunicationsPage` + `email_integration/db.query_email_activity`): **fixed the investor dropdown** — the facet now mirrors the list with the digest's precedence (grid → org → contact → address) and **typed keys** (`fund:`/`org:`/`contact:`), so email matched only to a classic contact or org domain (no grid id — the common case, since `fundraising_contacts.email` is sparsely populated) now resolves to a real name and is selectable, instead of the dropdown being empty; added a **date-range filter** (`since`/`until`), and a **click-to-expand full-body view** (`GET /api/email/detail?id=` → `query_email_detail`, admin, soft-delete-gated, renders `body_text` escaped — never raw HTML). New **semantic content search**: a "Search content" toggle → `GET /api/email/search?q=` (`routes._h_search`) wrapping `ingest/search.py:hybrid_search` filtered to `doc_type='email'` (lazy import; **503** if Spark/Qdrant unreachable), **hydrated + soft-delete-filtered against SQLite** (`db.search_hit_emails` — never trust the derived index). **Daily Digest:** Settings → Admin now builds a digest over a chosen window (last 24h or since a date) as an **in-app preview** before sending (`POST /api/admin/digest/preview`); manual send uses the same window (`send-now` + `digest_scheduler.send_digest_window`); window resolved by `digest_builder.resolve_digest_window` (cap 92d). Both run the **real local-Spark summarizer** and **never touch the daily cursor**. Verified: 22/22 backend tests, `py_compile` clean, render-smoke pass. **Grant validated both live on the box 2026-06-16** — the digest windowed preview renders real Spark narratives over real activity, and the Communications dropdown / date filter / full-body view / content-search all work. Detail: `docs/guides/email.md`.
+- **Deploy history (done — git log + `start9/0.4/startos/versions/` + guides):** v74 security/path-traversal hardening; v78 retired `lp_profiles`/LP Tracker (grid is canonical); v80–83 email-activity Communications tab (typed investor facet, date filter, full-body view, semantic content search) + daily-digest windowed preview→send (`docs/guides/email.md`); v82 vendored + SRI-pinned front-end libs + jsdom render-smoke build gate (`docs/guides/packaging.md`).
-- **Deployed & verified live: v0.1.0:82** (box `$START9_BOX_HOST`/immense-voyage.local; `installed-version`→`0.1.0:82`, migration chain `…81→82` clean, server up on `:8080`, schedulers + Gmail integration up). **v82 vendored React 18.3.1 / ReactDOM 18.3.1 / @babel/standalone 7.29.7 into `frontend/assets/vendor/`**, served same-origin with `sha384` SRI (no CDN, no outbound-internet dependency to render the UI), and added **`start9/0.4/render-smoke.mjs`** — a jsdom check (shipped-Babel transform asserts classic/non-module + parseable; real mount asserts the login UI renders) wired into the default `make` goal (`verified-build`), so every build is gated on the frontend actually rendering. Closes the v78 (blank screen) + v79 (Babel-8 ESM-import) class structurally. Detail: `docs/guides/packaging.md`. **Prior shipped & live:** v81 Communications-tab matched-only (`query_email_activity` gates on `EXISTS(email_investor_links)`; unmatched email captured but never shown; `docs/guides/email.md`); v80 admin-only email-activity panel (`GET /api/email/activity`); v78 retired `lp_profiles`/LP Tracker + repointed Dashboard "Total Committed" onto the grid (graveyard-excluded). **Digest fully live:** capture (DWD) → propose→approve; Gmail-DWD→SMTP transport; daily Phase-B digest (`digest_builder.py` + always-on `digest_scheduler.py` reading a DB policy + `send-now`); **daily auto-send is now ENABLED** (Grant turned it on in Settings → Admin, 2026-06-16). Detail: `docs/guides/email.md`.
+- **Tests:** **27/27 backend green** (`python3 backend/run_tests.py`), `py_compile` clean; frontend render-smoke gates the default `make` build.
-- **Live since v74 (2026-06-13):** login works; `/assets/` traversal 404s (plain + URL-encoded), root health 200. On boot, `ensure_thesis_v2_promoted` makes the v2.0 reserve-asset spine the working *approved* spine (node-level, reversible). Security/privacy hardening (path-traversal close, outreach NER backstop, get-by-id soft-delete) shipped in v74 — detail in `EVALUATION.md`.
+- **Debt (P2, not deploy-blocking; full list `EVALUATION.md`):** reports-subsystem soft-delete sweep (`handle_pipeline_report` + report aggregates still count tombstoned rows) + report-endpoint tests; `?limit=abc` crashes the request thread; auth regression test for the 3 v79-gated GETs (`/api/users`, `/api/email/status`, `/api/email/accounts`); scrub-gateway TLS verify off; hardcoded Spark/Qdrant IPs + **oversized StartOS package icon** (fix before the next s9pk upload); the 5.4k-line `server.py` monolith.
-- **Tests (2026-06-17):** **27/27 backend tests green** via `python3 backend/run_tests.py`, `py_compile` clean. (+4 last session for the Matrix intake bot: `matrix_intake/test_parse.py`, `test_proposals.py`, `test_crm_client.py`, and `test_intake_endpoints.py` — the last boots the real server against a temp DB and covers `/api/intake/match`, the create→match no-duplicate contract, and `source="matrix_intake"` provenance.) **This session (v86 fuzzy + conversational pass) added cases to those same files** — `test_intake_endpoints.py`: fuzzy `candidates` (near-spelling, legal-suffix-at-1.0, one-char email typo, exact→no-candidates, nothing-close→empty); `test_proposals.py`: the disambiguation grammar + `attach_to_candidate`/`promote_to_new`/`same_fields`; `test_parse.py`: `revise` merge + email-integrity-from-instruction + match-id preservation; `test_crm_client.py`: the `{match, candidates}` shape + no-query-skips-network. `test_email_activity_panel.py` now covers the **typed facet + org/contact resolution** (the dropdown fix), the **date-range filter**, the **detail view** (full body / recipients / attachments / soft-delete), and the **content-search route** (hydrate / drop-tombstoned / 503 / admin) with retrieval stubbed; `test_digest_builder.py` adds the **window resolver** + **`send_digest_window`** (no-cursor-touch) cases. Frontend **render smoke check** (`cd start9/0.4 && make render-smoke`) still gates the default `make` build. The 2 stale thesis tests stay fixed (seed structure in `docs/guides/thesis.md`).
+- **Open / risks:** the v2.0 reserve-asset spine is the *working* approved spine but **not a canonical `thesis_version`** (needs Grant + Jonathan dual sign-off; Appendix-A conviction incl. ~40% Strike stays Grant's working read, not fed to the engine); **Claude/Architect path still unverified live on the box**; the intake matcher reads only the grid blob (not classic `contacts`); doc drift — `crm-overview.md` + `EVALUATION.md` still call `lp_profiles` live (doc-auditor pass).
-- **Decided, not yet built (detail in `ROADMAP.md`):** Pipeline adoption + a grid flag that auto-loads flagged investors as opportunities; **NL→safe-query** feature (search item 3 — the larger, separate build); CRM as canonical thesis backbone with the signal-engine reading from it (reconciliation unwired); reply-all for Tier-B drafts (currently reply to the LP only). *(Done this session, v83: email search item 1 [activity query/panel gaps — typed facet fix + date range + full-body view] and item 2 [semantic content search] both shipped; daily-digest windowed preview→send.)*
+- **Next:** 1) **intake parse roster fix** (found in live-smoke: *"jonathan is chatting with wyoming"* → picked the teammate, not the prospect) — feed the parser the team roster + outreach frame (ROADMAP); cheap, offline-testable, fixes the gripe — and **re-smoke the Qwen `revise` leg** (still unexercised in-room); 2) **Pipeline adoption** — grid flag → auto-sync an `opportunities` row onto the Pipeline board (the agreed next major build; **design the grid↔pipeline link first** — ROADMAP "Adopt the Pipeline"); 3) **spark-control intake dashboard card** (separate session in the spark-control repo — handoff at `docs/handoffs/add-intake-bot-to-spark-control.md`), and longer-term **extract the bot to its own repo** (ROADMAP); 4) **NL→safe-query** (search item 3 — separate, larger build); 5) Grant + Jonathan freeze v2.0 canonical; 6) reply-all for Tier-B drafts; then clear the P2 debt above.
-- **Known debt (P2, not deploy-blocking):** **reports-subsystem soft-delete sweep** — `handle_pipeline_report` + remaining report/aggregate queries over opportunities/communications still count soft-deleted rows (v78 shrank this surface: the `lp_profiles`/lp-breakdown aggregates are gone and the dashboard "Total Committed" is now grid-sourced); needs a pass + report-endpoint tests. Also `?limit=abc` crashes the request thread (authenticated list path); scrub-gateway TLS verify off; `cryptography==42.0.5`; stale user-visible `start9/0.4/assets/ABOUT.md`; hardcoded Spark/Qdrant IPs in the s9pk; **StartOS package icon oversized/zoomed** (research the Start9 icon spec, source a base ten31 logo, produce a correctly sized icon **before the next s9pk upload**); the 5.4k-line `server.py` monolith. P3 batch + full list in `EVALUATION.md`. *(Resolved v82: front-end CDN/SRI risk — libs vendored + SRI-pinned — and the render smoke check is now scripted into the build.)*
-- **Doc drift to reconcile:** `crm-overview.md` + `EVALUATION.md` still describe `lp_profiles` as a live model in places — a doc-auditor pass should align them to "grid canonical, `lp_profiles` retired."
-- **Other gaps:** the v2.0 spine is the *working* spine but **not a canonical `thesis_version`** (needs Grant + Jonathan dual sign-off); Appendix-A conviction/exposure (incl. ~40% Strike) stay Grant's working read, not canonical, not fed to the engine. Live infra now exercised on the box (Gmail capture + schedulers up; local-Spark summarization confirmed via the digest preview; Qdrant via Communications content-search); **Claude/Architect path still unverified live on the box.**
-- **Next:** 1) **Pipeline adoption** — grid flag → auto-create/sync an `opportunities` row so flagged investors load into the Pipeline board (the agreed next major build; design the grid↔pipeline link first — see ROADMAP "Adopt the Pipeline"); 2) **intake-bot service polish** — DONE as a docker-compose container (`restart: unless-stopped`, survives reboot, 2026-06-17); remaining: a **spark-control dashboard card** (separate session in the spark-control repo — handoff at `docs/handoffs/add-intake-bot-to-spark-control.md`) and, longer-term, **extracting the bot to its own repo** (ROADMAP — the right blast-radius boundary); 3) **intake parse fix (found in live-smoke 2026-06-17)** — the v86 disambiguation shortlist + grammar worked in-room, but the parse mis-identified the investor when the message named an internal teammate (*"jonathan is chatting with wyoming"* → picked jonathan, not Wyoming); near-term fix = feed the parser the team roster + outreach frame (ROADMAP); the Qwen `revise` leg is still unsmoked; 4) **reports-subsystem soft-delete sweep** + report-endpoint tests; 5) `?limit=abc` crash; 6) **auth regression test** for the 3 v79-gated GET endpoints (`/api/users`, `/api/email/status`, `/api/email/accounts`); 7) **NL→safe-query** (search item 3 — separate, larger); 8) Grant + Jonathan freeze v2.0 canonical; 9) reply-all for Tier-B drafts.

docs/guides/matrix-intake.md

@@ -149,6 +149,10 @@ rows ≥ `min_score` (0.62), ranked, capped at 5:
 - **Not yet a spark-control dashboard card.** The container is managed via `docker`/SSH today; a
   managed card (Update/Restart/Stop/Logs tile, like `matrix-bridge`) is a separate spark-control
   task — see `docs/handoffs/add-intake-bot-to-spark-control.md`.
+- **Gotcha — the repo-root `.dockerignore` is SHARED** with the s9pk build (`start9/0.4/Dockerfile`,
+  same repo-root context). Don't add bot-only exclusions (e.g. `frontend/`, `docs/`) to it — you'd
+  break the CRM image build, which needs them. It already excludes the security-critical bits
+  (`data/`, `.env`), which is all the bot build needs.
 - **Server-side endpoints ship in the s9pk, not the bot.** `GET /api/intake/match` and the
   `source` provenance on `log-communication` live in `backend/server.py`, so they reach the box
   only via an **s9pk build + install** — a bot restart won't deliver them. (Missed in v83: the