From b2690c43428cf7c66ef288f59220880223fdf91a Mon Sep 17 00:00:00 2001 From: Keysat Date: Thu, 18 Jun 2026 12:32:06 -0500 Subject: [PATCH] Redact whole review threads on decision (replies too) The bot was granted a redact/mod power level in the review room, so it can now clear a resolved thread entirely, not just the card: redact_thread redacts the card root then scans recent history for its m.thread replies (the human's yes/no + any bot messages) and redacts those too, so decided threads drop out of the threads view, not only the main timeline. Drops the in-thread confirmation on a successful decision (the thread clearing is the ack; a confirmation would keep the thread alive). redact_resolved.py extended to also clear replies of already- resolved threads for the one-time backfill. Bot-only; no s9pk change. --- AGENTS.md | 2 +- backend/matrix_intake/bot.py | 53 ++++++++++++++++++------ backend/matrix_intake/redact_resolved.py | 29 +++++++------ 3 files changed, 59 insertions(+), 25 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 76df105..0125551 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -105,7 +105,7 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude _Phase 0 + Phase 1 built; **box and repo at v0.1.0:90** (v89 + v90 installed & verified live 2026-06-18 — `installed-version` = 0.1.0:90, server up on :8080, no errors; the StartOS version-graph traversal logs an inert down-to-39-then-up-to-90 because the per-version `up`/`down` hooks are no-ops — the real SQLite migrations run in-app at startup). **The fundraising grid + email capture is the canonical system of record** (2026-06-16) — vestigial classic-CRM surfaces get pruned/repurposed. Deploy/feature history lives in git log + `start9/0.4/startos/versions/`; longer-term backlog + debt in `ROADMAP.md` / `EVALUATION.md`._ -- **Email-proposal review over Matrix + a `bot` role — DEPLOYED & LIVE 2026-06-18 (box v0.1.0:91), smoke-tested OK.** **Post-smoke refinements (v0.1.0:91 + bot):** (1) every review card/reply is framed with `-----------------------` dash rules so threads don't bleed together on mobile (`email_proposals.frame`); (2) on a conclusive decision (either surface) the bot **redacts the card** (`client.room_redact`) so the room clears down to only undecided items — redacting its *own* card needs no power; wiping the human's reply for a fully-empty thread needs a `redact`/mod power level (not set, optional extension); (3) the proposed note now NAMES who emailed whom — "{teammate} emailed {investor}" / "{sender} emailed the team" — instead of a bare "Sent/Received", and outbound detection now also matches our corporate domain (public providers excluded) so a teammate's non-enrolled `@ten31.xyz` mail no longer reads as "Received". Feature 1 (the inline source-email popup) is live on the box. The Matrix leg is live: bot CRM user flipped to `bot` (via the v90 Settings → Admin dropdown), `MATRIX_EMAIL_REVIEW_ROOM=!CImVJWFmzNxPcCcrZl:matrix.gilliam.ai` set on the Spark `.env`, bot joins the room on startup, polls every 20s. **End-to-end proven on deploy:** the whole pending backlog (N=16) posted as review cards to the room and was marked posted on the box (no errors); the bot now idles (to_post empty) holding 16 open threads. **Deploy gotchas learned:** (a) order matters — the bot caches its JWT (role embedded) at login and only re-logins on 401, not 403, so the role must be `bot` *before* the bot process starts; (b) "invited" ≠ "joined" — the bot must `client.join` the room (added in `2998706`) or `room_send` fails M_FORBIDDEN; (c) on first run the bot posts a card per *pending* proposal — fine here at N=16, but a large backlog would flood + get homeserver-429'd (a `since`-floor on `to_post` is the upgrade path if that ever bites). **Smoke-test still pending (manual):** reply `yes`/`no`/edit **in-thread** on a card → grid note + thread closes; approve one on the web → the bot announces + closes its thread (≤20s); confirm a Matrix decision clears the web card (≤25s). Two asks on the email-capture "proposed grid notes": (1) **inline source-email popup** — each proposed-note card on the Email Capture page gets a **View email** toggle that lazily fetches the existing **`GET /api/email/detail`** and renders from/to/cc/date/subject + scrollable body inline, so you can judge the note against the email (frontend-only, reuses the Communications detail pattern). (2) **CRM→Matrix review bridge** — the CRM (box, no matrix-nio) can't push, so the **intake bot (Spark) PULLS**: new **`GET /api/intake/email-proposals`** returns three work-lists (`to_post`/`open`/`to_close`); the bot posts a review card (metadata + snippet + draft note) to a **dedicated review room** (`MATRIX_EMAIL_REVIEW_ROOM`), records the thread root (`POST .../{id}/matrix`), and relays in-thread **yes / no / NL-edit** (`POST .../{id}/decide`, NL-revise via local Qwen). **Bidirectional sync:** decide on the web → bot announces + closes the thread; decide on Matrix → the web panel's ~25s poll clears the card. State is CRM-side in **`email_proposal_matrix`** (1:1 side row, email-integration migration **`0003`**, additive + idempotent `CREATE TABLE IF NOT EXISTS` — the email runner re-runs every boot via `executescript`, so no `ALTER`), so it survives a bot restart. New **`bot` role** (authenticated, never admin — `require_bot_or_admin`) gates the email-proposal endpoints; the bot's CRM user must be flipped `member→bot` (one-time, kept out of the invite UI). **Deploy split:** endpoints + migration + role + frontend → **s9pk (v89/v90, installed)**; poll loop + review-room handling → **Spark git pull + restart (pending)**. Tests: `backend/test_email_proposal_matrix.py` + `backend/matrix_intake/test_email_proposals.py`; **30/30 suite green**, render-smoke green, migration verified twice on a copy of `data/crm.db` + applied clean on the box. **Next (see the two manual steps above): flip the bot user → `bot`; then Spark deploy with the review room; then live-smoke the web↔Matrix round-trip** (the popup is already verifiable on the box). Guide: `docs/guides/matrix-intake.md` "Email-activity proposal review". +- **Email-proposal review over Matrix + a `bot` role — DEPLOYED & LIVE 2026-06-18 (box v0.1.0:91), smoke-tested OK.** **Post-smoke refinements (v0.1.0:91 + bot):** (1) every review card/reply is framed with `-----------------------` dash rules so threads don't bleed together on mobile (`email_proposals.frame`); (2) on a conclusive decision (either surface) the bot **redacts the whole thread** (`redact_thread`: the card + a backward room scan for its `m.thread` replies) so a resolved thread clears from both the timeline and the threads view — the bot was granted a `redact`/mod power level in the room (needed to redact the human's reply); no in-thread confirmation is posted on success (the thread vanishing is the ack — a confirmation would keep the thread alive). One-time backfill of pre-existing threads via `backend/matrix_intake/redact_resolved.py` (dry-run default; `--apply`); (3) the proposed note now NAMES who emailed whom — "{teammate} emailed {investor}" / "{sender} emailed the team" — instead of a bare "Sent/Received", and outbound detection now also matches our corporate domain (public providers excluded) so a teammate's non-enrolled `@ten31.xyz` mail no longer reads as "Received". Feature 1 (the inline source-email popup) is live on the box. The Matrix leg is live: bot CRM user flipped to `bot` (via the v90 Settings → Admin dropdown), `MATRIX_EMAIL_REVIEW_ROOM=!CImVJWFmzNxPcCcrZl:matrix.gilliam.ai` set on the Spark `.env`, bot joins the room on startup, polls every 20s. **End-to-end proven on deploy:** the whole pending backlog (N=16) posted as review cards to the room and was marked posted on the box (no errors); the bot now idles (to_post empty) holding 16 open threads. **Deploy gotchas learned:** (a) order matters — the bot caches its JWT (role embedded) at login and only re-logins on 401, not 403, so the role must be `bot` *before* the bot process starts; (b) "invited" ≠ "joined" — the bot must `client.join` the room (added in `2998706`) or `room_send` fails M_FORBIDDEN; (c) on first run the bot posts a card per *pending* proposal — fine here at N=16, but a large backlog would flood + get homeserver-429'd (a `since`-floor on `to_post` is the upgrade path if that ever bites). **Smoke-test still pending (manual):** reply `yes`/`no`/edit **in-thread** on a card → grid note + thread closes; approve one on the web → the bot announces + closes its thread (≤20s); confirm a Matrix decision clears the web card (≤25s). Two asks on the email-capture "proposed grid notes": (1) **inline source-email popup** — each proposed-note card on the Email Capture page gets a **View email** toggle that lazily fetches the existing **`GET /api/email/detail`** and renders from/to/cc/date/subject + scrollable body inline, so you can judge the note against the email (frontend-only, reuses the Communications detail pattern). (2) **CRM→Matrix review bridge** — the CRM (box, no matrix-nio) can't push, so the **intake bot (Spark) PULLS**: new **`GET /api/intake/email-proposals`** returns three work-lists (`to_post`/`open`/`to_close`); the bot posts a review card (metadata + snippet + draft note) to a **dedicated review room** (`MATRIX_EMAIL_REVIEW_ROOM`), records the thread root (`POST .../{id}/matrix`), and relays in-thread **yes / no / NL-edit** (`POST .../{id}/decide`, NL-revise via local Qwen). **Bidirectional sync:** decide on the web → bot announces + closes the thread; decide on Matrix → the web panel's ~25s poll clears the card. State is CRM-side in **`email_proposal_matrix`** (1:1 side row, email-integration migration **`0003`**, additive + idempotent `CREATE TABLE IF NOT EXISTS` — the email runner re-runs every boot via `executescript`, so no `ALTER`), so it survives a bot restart. New **`bot` role** (authenticated, never admin — `require_bot_or_admin`) gates the email-proposal endpoints; the bot's CRM user must be flipped `member→bot` (one-time, kept out of the invite UI). **Deploy split:** endpoints + migration + role + frontend → **s9pk (v89/v90, installed)**; poll loop + review-room handling → **Spark git pull + restart (pending)**. Tests: `backend/test_email_proposal_matrix.py` + `backend/matrix_intake/test_email_proposals.py`; **30/30 suite green**, render-smoke green, migration verified twice on a copy of `data/crm.db` + applied clean on the box. **Next (see the two manual steps above): flip the bot user → `bot`; then Spark deploy with the review room; then live-smoke the web↔Matrix round-trip** (the popup is already verifiable on the box). Guide: `docs/guides/matrix-intake.md` "Email-activity proposal review". - **Adopt the Pipeline — grid drives the deal board — DEPLOYED & live-smoked 2026-06-18 (v0.1.0:88; the full +Pipeline → board → advance-stage → remove round-trip is verified on the box). v88 (frontend-only): retired the Pipeline page's "+ New Opportunity" button + its create-by-contact modal** — opportunities are now born **only** from a grid investor row (matches how the team works; the board is view + stage-management; button replaced with a muted "Add deals from the Fundraising Grid" hint). An **"Add to Pipeline"** row action on the fundraising grid opens a seed modal (primary contact / target fund / expected amount / stage / probability) and creates a durably-linked `opportunities` row via the new **`opportunities.fundraising_investor_id`** (migration 0005, additive + reversible). **Grid owns the link + seed; the board owns stage/probability/owner** — a grid save never reseeds a live opp (`POST /api/fundraising/pipeline/link` is idempotent, one live opp/investor). Contact is **reused from the grid's synced `fundraising_contacts.contact_id`** (the `POST /api/contacts` side-door is gone); grid `lead`→owner. Two **read-only** grid columns (Pipeline action + Pipeline Stage) are **injected on read** from the live opp and **stripped on write** (never persisted, never dirty the autosave). **Remove from pipeline** (`POST .../unlink`) **soft-deletes the opp; the grid row stays fully intact**; deleting an investor from the grid archives its orphaned opp (`reconcile_grid_pipeline_links`, after `sync_fundraising_relational`). **Folded in:** the standing P2 soft-delete leak in `handle_pipeline_report` + dashboard pipeline aggregates (archived opps no longer counted). Tests: `backend/test_grid_pipeline_link.py`; 28/28 suite green, render-smoke green; migration verified on a copy of `data/crm.db` and **applied clean on the box**. **Next: live-smoke on the box — add an investor to the pipeline, confirm it lands on the board, advance a stage, and remove (opp archived, grid row intact).** Detail + locked decisions in `ROADMAP.md` "Adopt the Pipeline". diff --git a/backend/matrix_intake/bot.py b/backend/matrix_intake/bot.py index 5df75bb..08f4c7a 100644 --- a/backend/matrix_intake/bot.py +++ b/backend/matrix_intake/bot.py @@ -11,7 +11,7 @@ Lifts matrix-bridge's prime-then-listen + threaded-reply plumbing. Config: repo """ import asyncio -from nio import AsyncClient, MatrixRoom, RoomMessageText +from nio import AsyncClient, MatrixRoom, MessageDirection, RoomMessageText import crm_client import email_proposals @@ -27,6 +27,7 @@ UNCLEAR_HELP = ( ) EMAIL_POLL_SEC = 20 # how often the bot polls the CRM for new/decided email-activity proposals +MAX_THREAD_SCAN_PAGES = 8 # how far back to scan for a resolved thread's replies before redacting async def main(): @@ -163,14 +164,42 @@ async def main(): await say(room_id, "I didn't catch that.\n\n" + proposals.render_disambiguation(proposal), root) async def redact_card(event_id): - """Remove a decided card from the room so only undecided ones remain. Redacting our OWN - card needs no special power; in Element a redacted message drops out of the timeline. (To - also wipe the human's yes/no reply for a fully-empty thread, give the bot a redact/mod - power level — not required for this.)""" + """Redact one event (best-effort). Redacting our OWN message needs no special power; + redacting someone else's reply needs the bot to hold a redact/mod power level.""" try: await client.room_redact(review_room, event_id, reason="proposal resolved") except Exception as exc: - print(f"matrix-intake: could not redact card {event_id}: {exc}", flush=True) + print(f"matrix-intake: could not redact {event_id}: {exc}", flush=True) + + async def redact_thread(root): + """Clear a resolved thread: redact the card AND every reply under it, so the thread drops + out of the threads view (not just the main timeline). The card is ours (always redactable); + the human's yes/no reply needs the bot's redact/mod power — if it lacks power that redact + just no-ops and the reply lingers. Finds replies by scanning recent room history for + m.thread events pointing at this root (the triggering reply is already synced, so a + backward scan from the current token includes it).""" + await redact_card(root) + token = getattr(client, "next_batch", None) + if not token: + return + try: + scanned = 0 + for _ in range(MAX_THREAD_SCAN_PAGES): + resp = await client.room_messages(review_room, start=token, + direction=MessageDirection.back, limit=100) + chunk = getattr(resp, "chunk", None) + if not chunk: + break + for ev in chunk: + rel = ((getattr(ev, "source", None) or {}).get("content", {}) or {}).get("m.relates_to") or {} + if rel.get("rel_type") == "m.thread" and rel.get("event_id") == root: + await redact_card(ev.event_id) + token = getattr(resp, "end", None) + scanned += len(chunk) + if not token or scanned > 1000: + break + except Exception as exc: + print(f"matrix-intake: thread reply cleanup failed for {root}: {exc}", flush=True) async def handle_email_reply(room_id, root, text): """An in-thread reply to a CRM-drafted email-proposal card: yes commits, no dismisses, and @@ -190,8 +219,9 @@ async def main(): email_threads[root] = item # restore for retry await say(room_id, email_proposals.frame(f"⚠️ couldn't add it ({str(exc)[:200]}). Reply **yes** to retry, **no** to dismiss."), root) return - await say(room_id, email_proposals.frame(f"✅ Added to the grid for **{item.get('investor_name') or 'the investor'}**."), root) - await redact_card(root) + # Success → clear the whole thread (card + replies). No confirmation: the thread + # vanishing is the acknowledgment, and a confirmation reply would keep it alive. + await redact_thread(root) elif decision == "reject": email_threads.pop(root, None) try: @@ -200,8 +230,7 @@ async def main(): email_threads[root] = item await say(room_id, email_proposals.frame(f"⚠️ couldn't dismiss it ({str(exc)[:200]}). Try again."), root) return - await say(room_id, email_proposals.frame("🗑️ Dismissed — nothing added to the grid."), root) - await redact_card(root) + await redact_thread(root) else: try: new_note = await asyncio.to_thread(email_proposals.revise_note, item.get("note") or "", text) @@ -244,12 +273,12 @@ async def main(): "note": it.get("proposed_note") or ""} except Exception as exc: print(f"matrix-intake: failed to post email proposal {it.get('id')}: {exc}", flush=True) - for it in lists["to_close"]: # decided on the web → remove the card, then close + for it in lists["to_close"]: # decided on the web → clear the thread, then close ev = it.get("event_id") if not ev: continue try: - await redact_card(ev) + await redact_thread(ev) await asyncio.to_thread(crm_client.mark_email_proposal_closed, it["id"]) email_threads.pop(ev, None) except Exception as exc: diff --git a/backend/matrix_intake/redact_resolved.py b/backend/matrix_intake/redact_resolved.py index 851d95b..da88797 100644 --- a/backend/matrix_intake/redact_resolved.py +++ b/backend/matrix_intake/redact_resolved.py @@ -39,7 +39,8 @@ async def main(apply): sync = await client.sync(timeout=10000, full_state=False) token = sync.next_batch - cards = {} # event_id -> snippet (dedup across pages) + cards = {} # root event_id -> snippet (still-identifiable card bodies) + replies = {} # reply event_id -> (thread_root, snippet) for _ in range(MAX_PAGES): resp = await client.room_messages(review_room, start=token, direction=MessageDirection.back, limit=100) @@ -47,24 +48,28 @@ async def main(apply): if not chunk: break for ev in chunk: - if getattr(ev, "sender", None) != mx["user_id"]: - continue - body = getattr(ev, "body", "") or "" - if CARD_MARKER in body: - cards[ev.event_id] = body.replace("\n", " ")[:70] + body = (getattr(ev, "body", "") or "").replace("\n", " ") + rel = ((getattr(ev, "source", None) or {}).get("content", {}) or {}).get("m.relates_to") or {} + if rel.get("rel_type") == "m.thread" and rel.get("event_id"): + replies[ev.event_id] = (rel["event_id"], body[:50]) # a threaded reply (card already redacted) + elif getattr(ev, "sender", None) == mx["user_id"] and CARD_MARKER in body: + cards[ev.event_id] = body[:70] # an un-redacted card root token = getattr(resp, "end", None) if not token: break - to_redact = [(eid, snip) for eid, snip in cards.items() if eid not in open_ids] - print(f"bot cards found: {len(cards)}; resolved (to redact): {len(to_redact)}") - for eid, snip in to_redact: - print(("APPLY redact " if apply else "WOULD redact ") + eid + " :: " + snip) + # Redact card roots that aren't still pending, AND any reply whose thread isn't still pending. + targets = [(eid, "card :: " + snip) for eid, snip in cards.items() if eid not in open_ids] + targets += [(eid, "reply :: " + snip) for eid, (root, snip) in replies.items() if root not in open_ids] + print(f"resolved cards: {sum(1 for e,_ in cards.items() if e not in open_ids)}; " + f"thread replies to clear: {sum(1 for _,(r,_) in replies.items() if r not in open_ids)}") + for eid, label in targets: + print(("APPLY redact " if apply else "WOULD redact ") + eid + " :: " + label) if apply: - r = await client.room_redact(review_room, eid, reason="retroactive cleanup of resolved cards") + r = await client.room_redact(review_room, eid, reason="retroactive cleanup of resolved review threads") if not hasattr(r, "event_id"): print(f" ! redact failed: {r}") - print(("done — redacted " if apply else "dry run — would redact ") + f"{len(to_redact)} card(s).") + print(("done — redacted " if apply else "dry run — would redact ") + f"{len(targets)} event(s).") finally: await client.close()