diff --git a/.env.example b/.env.example index 6b96022..6503dff 100644 --- a/.env.example +++ b/.env.example @@ -57,6 +57,13 @@ MATRIX_USER=@intake-bot: MATRIX_ACCESS_TOKEN= MATRIX_DEVICE_ID=ten31-intake-bot MATRIX_INTAKE_ROOM=!: +# Dedicated room for reviewing CRM-drafted email-activity proposals (the proposed grid notes the +# Email Capture panel shows). The bot posts a review card per pending proposal here and relays the +# in-thread yes/no/edit back to the CRM, in sync with the web panel. Separate from the intake room +# so high-volume email proposals don't drown the conversational intake. Leave empty to disable the +# whole email-review poll loop. The bot must be a member of this room. Needs the server side in the +# s9pk (≥ v0.1.0:89) and the bot's CRM user set to role 'bot' (see docs/guides/matrix-intake.md). +MATRIX_EMAIL_REVIEW_ROOM=!: # CRM write-back: the bot logs in as a DEDICATED service user (admin-created CRM user; # the CRM has no service-key path, so it uses normal Bearer-JWT auth). CRM_API_BASE=http://127.0.0.1:8080 diff --git a/AGENTS.md b/AGENTS.md index f3a2231..d1b1c38 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -103,7 +103,9 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude ## Current state -_Phase 0 + Phase 1 built; **box and repo at v0.1.0:88** (deployed & verified live 2026-06-18 — chain …86→88 clean, `0005_grid_pipeline_link.sql` applied on the box, server up; the full Pipeline **+Pipeline → board → advance-stage → remove** round-trip is live-smoked on the box). **The fundraising grid + email capture is the canonical system of record** (2026-06-16) — vestigial classic-CRM surfaces get pruned/repurposed. Deploy/feature history lives in git log + `start9/0.4/startos/versions/`; longer-term backlog + debt in `ROADMAP.md` / `EVALUATION.md`._ +_Phase 0 + Phase 1 built; **box at v0.1.0:88, repo at v0.1.0:89 (built, NOT yet deployed)**. v88 deployed & verified live 2026-06-18 — chain …86→88 clean, `0005_grid_pipeline_link.sql` applied on the box, server up; the full Pipeline **+Pipeline → board → advance-stage → remove** round-trip is live-smoked on the box. **The fundraising grid + email capture is the canonical system of record** (2026-06-16) — vestigial classic-CRM surfaces get pruned/repurposed. Deploy/feature history lives in git log + `start9/0.4/startos/versions/`; longer-term backlog + debt in `ROADMAP.md` / `EVALUATION.md`._ + +- **Email-proposal review over Matrix + a `bot` role — BUILT v0.1.0:89, NOT yet deployed (2026-06-18).** Two asks on the email-capture "proposed grid notes": (1) **inline source-email popup** — each proposed-note card on the Email Capture page gets a **View email** toggle that lazily fetches the existing **`GET /api/email/detail`** and renders from/to/cc/date/subject + scrollable body inline, so you can judge the note against the email (frontend-only, reuses the Communications detail pattern). (2) **CRM→Matrix review bridge** — the CRM (box, no matrix-nio) can't push, so the **intake bot (Spark) PULLS**: new **`GET /api/intake/email-proposals`** returns three work-lists (`to_post`/`open`/`to_close`); the bot posts a review card (metadata + snippet + draft note) to a **dedicated review room** (`MATRIX_EMAIL_REVIEW_ROOM`), records the thread root (`POST .../{id}/matrix`), and relays in-thread **yes / no / NL-edit** (`POST .../{id}/decide`, NL-revise via local Qwen). **Bidirectional sync:** decide on the web → bot announces + closes the thread; decide on Matrix → the web panel's ~25s poll clears the card. State is CRM-side in **`email_proposal_matrix`** (1:1 side row, email-integration migration **`0003`**, additive + idempotent `CREATE TABLE IF NOT EXISTS` — the email runner re-runs every boot via `executescript`, so no `ALTER`), so it survives a bot restart. New **`bot` role** (authenticated, never admin — `require_bot_or_admin`) gates the email-proposal endpoints; the bot's CRM user must be flipped `member→bot` (one-time, kept out of the invite UI). **Deploy split:** endpoints + migration + role + frontend → **s9pk v89 build+install**; poll loop + review-room handling → **Spark git pull + restart**. Tests: `backend/test_email_proposal_matrix.py` + `backend/matrix_intake/test_email_proposals.py`; **30/30 suite green**, render-smoke green, migration verified twice on a copy of `data/crm.db`. **Next: deploy v89 to the box (auth required), set the bot user's role + the review room + invite the bot, then live-smoke both legs** (popup; web↔Matrix round-trip). Guide: `docs/guides/matrix-intake.md` "Email-activity proposal review". - **Adopt the Pipeline — grid drives the deal board — DEPLOYED & live-smoked 2026-06-18 (v0.1.0:88; the full +Pipeline → board → advance-stage → remove round-trip is verified on the box). v88 (frontend-only): retired the Pipeline page's "+ New Opportunity" button + its create-by-contact modal** — opportunities are now born **only** from a grid investor row (matches how the team works; the board is view + stage-management; button replaced with a muted "Add deals from the Fundraising Grid" hint). An **"Add to Pipeline"** row action on the fundraising grid opens a seed modal (primary contact / target fund / expected amount / stage / probability) and creates a durably-linked `opportunities` row via the new **`opportunities.fundraising_investor_id`** (migration 0005, additive + reversible). **Grid owns the link + seed; the board owns stage/probability/owner** — a grid save never reseeds a live opp (`POST /api/fundraising/pipeline/link` is idempotent, one live opp/investor). Contact is **reused from the grid's synced `fundraising_contacts.contact_id`** (the `POST /api/contacts` side-door is gone); grid `lead`→owner. Two **read-only** grid columns (Pipeline action + Pipeline Stage) are **injected on read** from the live opp and **stripped on write** (never persisted, never dirty the autosave). **Remove from pipeline** (`POST .../unlink`) **soft-deletes the opp; the grid row stays fully intact**; deleting an investor from the grid archives its orphaned opp (`reconcile_grid_pipeline_links`, after `sync_fundraising_relational`). **Folded in:** the standing P2 soft-delete leak in `handle_pipeline_report` + dashboard pipeline aggregates (archived opps no longer counted). Tests: `backend/test_grid_pipeline_link.py`; 28/28 suite green, render-smoke green; migration verified on a copy of `data/crm.db` and **applied clean on the box**. **Next: live-smoke on the box — add an investor to the pipeline, confirm it lands on the board, advance a stage, and remove (opp archived, grid row intact).** Detail + locked decisions in `ROADMAP.md` "Adopt the Pipeline". diff --git a/backend/email_integration/migrations/0003_email_proposal_matrix.sql b/backend/email_integration/migrations/0003_email_proposal_matrix.sql new file mode 100644 index 0000000..6be3272 --- /dev/null +++ b/backend/email_integration/migrations/0003_email_proposal_matrix.sql @@ -0,0 +1,30 @@ +-- ============================================================================ +-- email_proposal_matrix — Matrix-review state for an email_activity_proposal, +-- kept 1:1 with the proposal (proposal_id PK). The CRM runs on the box and has +-- no matrix-nio, so it cannot post to Matrix itself: the intake bot (on the Spark) +-- PULLS pending proposals, posts a review card to the dedicated Matrix review room, +-- and writes the thread-root event_id back here. Persisting it CRM-side (not just in +-- the bot's memory) keeps both surfaces in sync and survives a bot restart. +-- +-- A SIDE TABLE rather than new columns on email_activity_proposals because the +-- email-integration migration runner (email_integration/db.py:apply_migrations) +-- re-runs every .sql file on every boot via executescript with no ledger — so +-- CREATE TABLE IF NOT EXISTS is idempotent, whereas ALTER ... ADD COLUMN would throw +-- "duplicate column" on the second boot and abort startup. Reversal: DROP TABLE +-- (this runner has no .down.sql convention; cf. 0001/0002). +-- +-- posted_at — set once the bot has posted the review card (event_id = thread root). +-- closed_at — set when the thread is resolved: either the bot decided in-thread, OR +-- the bot announced a web-side decision. A posted+decided proposal with +-- closed_at NULL is exactly the bot's signal to post "decided on the web" +-- into the thread and then close it. +-- ============================================================================ +CREATE TABLE IF NOT EXISTS email_proposal_matrix ( + proposal_id TEXT PRIMARY KEY, + event_id TEXT, -- Matrix thread-root event id of the posted review card + posted_at TEXT, + closed_at TEXT, + created_at TEXT DEFAULT (datetime('now')), + FOREIGN KEY(proposal_id) REFERENCES email_activity_proposals(id) ON DELETE CASCADE +); +CREATE INDEX IF NOT EXISTS idx_email_proposal_matrix_event ON email_proposal_matrix(event_id); diff --git a/backend/matrix_intake/bot.py b/backend/matrix_intake/bot.py index 741bade..bd07918 100644 --- a/backend/matrix_intake/bot.py +++ b/backend/matrix_intake/bot.py @@ -14,6 +14,7 @@ import asyncio from nio import AsyncClient, MatrixRoom, RoomMessageText import crm_client +import email_proposals import matrix_io import parse import proposals @@ -25,6 +26,8 @@ UNCLEAR_HELP = ( "or a note like `Note for Acme Capital: wants the Q3 deck, follow up next week`." ) +EMAIL_POLL_SEC = 20 # how often the bot polls the CRM for new/decided email-activity proposals + async def main(): mx = settings.matrix_settings() @@ -37,6 +40,8 @@ async def main(): roster = settings.team_roster() # frames the parse: teammates do outreach, aren't prospects if roster: print(f"matrix-intake: team roster loaded ({len(roster)} names)", flush=True) + review_room = settings.email_review_room() # CRM-drafted email proposals (empty → feature off) + email_threads = {} # Matrix thread-root event_id -> {id, investor_name, note} for an email proposal async def handle_intake(room_id, root, text): # A bare yes/no/approve typed in the MAIN timeline (not inside a proposal's thread) is @@ -157,15 +162,103 @@ async def main(): store.put(root, proposal) await say(room_id, "I didn't catch that.\n\n" + proposals.render_disambiguation(proposal), root) + async def handle_email_reply(room_id, root, text): + """An in-thread reply to a CRM-drafted email-proposal card: yes commits, no dismisses, and + anything else is a natural-language revision of the note (re-drafted by local Qwen; the + human still approves the revised note, so the draft→approve gate holds).""" + item = email_threads.get(root) + if item is None: + return # a threaded reply we don't own (or already resolved) + decision = email_proposals.interpret(text) + if decision == "approve": + # Claim before the await (double-approve guard, like the intake commit path). + email_threads.pop(root, None) + try: + await asyncio.to_thread(crm_client.decide_email_proposal, item["id"], "approve", item.get("note")) + except Exception as exc: + email_threads[root] = item # restore for retry + await say(room_id, f"⚠️ couldn't add it ({str(exc)[:200]}). Reply **yes** to retry, **no** to dismiss.", root) + return + await say(room_id, f"✅ Added to the grid for **{item.get('investor_name') or 'the investor'}**.", root) + elif decision == "reject": + email_threads.pop(root, None) + try: + await asyncio.to_thread(crm_client.decide_email_proposal, item["id"], "dismiss") + except Exception as exc: + email_threads[root] = item + await say(room_id, f"⚠️ couldn't dismiss it ({str(exc)[:200]}). Try again.", root) + return + await say(room_id, "🗑️ Dismissed — nothing added to the grid.", root) + else: + try: + new_note = await asyncio.to_thread(email_proposals.revise_note, item.get("note") or "", text) + except Exception as exc: + await say(room_id, f"⚠️ couldn't revise that ({str(exc)[:200]}). Reply **yes** to add as-is, " + "**no** to dismiss, or rephrase.", root) + return + if not new_note: + await say(room_id, "I didn't catch a change. Reply **yes** to add the note as-is, **no** to " + "dismiss, or tell me how to change it.", root) + return + item["note"] = new_note + email_threads[root] = item + await say(room_id, f"✏️ Updated draft note:\n\n{new_note}\n\nReply **yes** to add it, **no** to " + "dismiss, or refine again.", root) + + async def poll_email_proposals(): + """Poll the CRM for email-activity proposals: post a review card for each new one, rebuild + the reply-routing map from already-posted threads (so replies still route after a restart), + and announce+close any decided on the web. One failing cycle logs and retries next tick.""" + while True: + try: + lists = await asyncio.to_thread(crm_client.list_email_proposals) + for it in lists["open"]: # rebuild routing for threads posted before (e.g. a restart) + ev = it.get("event_id") + if ev and ev not in email_threads: + email_threads[ev] = {"id": it["id"], "investor_name": it.get("investor_name"), + "note": it.get("proposed_note") or ""} + for it in lists["to_post"]: + try: + resp = await client.room_send( + review_room, "m.room.message", + matrix_io.thread_content(email_proposals.render_card(it), None)) + ev = getattr(resp, "event_id", None) + if not ev: + print(f"matrix-intake: card send returned no event_id for {it['id']}", flush=True) + continue + await asyncio.to_thread(crm_client.mark_email_proposal_posted, it["id"], ev) + email_threads[ev] = {"id": it["id"], "investor_name": it.get("investor_name"), + "note": it.get("proposed_note") or ""} + except Exception as exc: + print(f"matrix-intake: failed to post email proposal {it.get('id')}: {exc}", flush=True) + for it in lists["to_close"]: # decided on the web → announce in-thread, then close + ev = it.get("event_id") + if not ev: + continue + try: + await say(review_room, email_proposals.closure_line(it.get("status")), ev) + await asyncio.to_thread(crm_client.mark_email_proposal_closed, it["id"]) + email_threads.pop(ev, None) + except Exception as exc: + print(f"matrix-intake: failed to close email proposal {it.get('id')}: {exc}", flush=True) + except Exception as exc: + print(f"matrix-intake: email-proposal poll error: {exc}", flush=True) + await asyncio.sleep(EMAIL_POLL_SEC) + async def on_message(room: MatrixRoom, event: RoomMessageText): if event.sender == mx["user_id"]: return # never react to our own messages (we post in-thread — this prevents loops) - if room.room_id != intake_room: - return text = (event.body or "").strip() if not text: return root = matrix_io.thread_root_of(event) + # Email-proposal review room: only a threaded reply to a card we posted is actionable. + if review_room and room.room_id == review_room: + if root and root in email_threads: + await handle_email_reply(room.room_id, root, text) + return + if room.room_id != intake_room: + return if root and store.has(root): await handle_reply(room.room_id, root, text) elif root: @@ -180,8 +273,12 @@ async def main(): client.add_event_callback(on_message, RoomMessageText) who = await client.whoami() print(f"matrix-intake: listening as {who.user_id} in room {intake_room}", flush=True) + tasks = [asyncio.create_task(client.sync_forever(timeout=30000))] + if review_room: + tasks.append(asyncio.create_task(poll_email_proposals())) + print(f"matrix-intake: reviewing email proposals in room {review_room} (every {EMAIL_POLL_SEC}s)", flush=True) try: - await client.sync_forever(timeout=30000) + await asyncio.gather(*tasks) finally: await client.close() diff --git a/backend/matrix_intake/crm_client.py b/backend/matrix_intake/crm_client.py index 9557781..e79d70d 100644 --- a/backend/matrix_intake/crm_client.py +++ b/backend/matrix_intake/crm_client.py @@ -98,6 +98,47 @@ def match(proposal): return {"match": match_out, "candidates": candidates} +def list_email_proposals(): + """Pull the email-activity review work-lists for the poll loop: {to_post, open, to_close}. + to_post = pending, un-posted (post a card); open = posted, awaiting a decision (rebuild the + reply-routing map after a restart); to_close = decided on the web (announce in-thread + close).""" + status, data = _authed("GET", "/api/intake/email-proposals") + if status != 200: + raise RuntimeError(f"email-proposals list failed ({status}): {data.get('error') or data}") + payload = data.get("data") or {} + return {k: (payload.get(k) or []) for k in ("to_post", "open", "to_close")} + + +def mark_email_proposal_posted(proposal_id, event_id): + """Record the Matrix thread-root event id so the proposal's review state survives a restart.""" + status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/matrix", + {"event_id": event_id}) + if status != 200: + raise RuntimeError(f"mark posted failed ({status}): {data.get('error') or data}") + return data.get("data") or {} + + +def mark_email_proposal_closed(proposal_id): + """Mark the review thread resolved after announcing a web-side decision in it.""" + status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/matrix", + {"closed": True}) + if status != 200: + raise RuntimeError(f"mark closed failed ({status}): {data.get('error') or data}") + return data.get("data") or {} + + +def decide_email_proposal(proposal_id, decision, note=None): + """Relay an in-thread approve/dismiss (with the possibly-revised note) to the CRM. The server + appends the note to the grid on approve, tags source='matrix', and closes the thread.""" + body = {"decision": decision} + if note is not None: + body["note"] = note + status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/decide", body) + if status not in (200, 201): + raise RuntimeError(f"email-proposal decide failed ({status}): {data.get('error') or data}") + return data.get("data") or {} + + def build_commit_payload(proposal): """Pure: map a proposal to the /api/fundraising/log-communication request body. diff --git a/backend/matrix_intake/email_proposals.py b/backend/matrix_intake/email_proposals.py new file mode 100644 index 0000000..f47cbe6 --- /dev/null +++ b/backend/matrix_intake/email_proposals.py @@ -0,0 +1,85 @@ +"""Email-activity proposal review over Matrix — the CRM→Matrix leg of the email-capture flow. + +The CRM (on the box) drafts a proposed grid note per newly-matched email (local model, no Claude) +and queues it for human review. The CRM is stdlib-only and can't post to Matrix itself, so this +bot PULLS the pending proposals (crm_client.list_email_proposals), posts a review card to the +dedicated review room, and relays the human's in-thread reply back to the CRM. Same draft→approve +discipline as the intake bot: nothing is appended to the grid until a human approves — here OR on +the web Email Capture panel, the two surfaces kept in sync via the CRM's email_proposal_matrix row. + +This module is the PURE logic (card rendering, reply grammar, note revision) so it's unit-tested +offline; the async poll/post/reply wiring lives in bot.py (network + Matrix, live-smoke only). +""" +import spark + +_YES = {"yes", "y", "approve", "approved", "ok", "confirm", "go", "add", "👍", "✅"} +_NO = {"no", "n", "cancel", "discard", "reject", "skip", "stop", "👎", "❌"} + +_SNIPPET_MAX = 400 # email snippet shown on the card; the full body is in the web popup + + +def _truncate(s, n): + s = (s or "").strip() + return s if len(s) <= n else s[:n].rstrip() + "…" + + +def render_card(item): + """The review card posted to the Matrix review room: who/when + a short email snippet + the + drafted note. Deliberately compact for mobile — the full scrollable body is in the web Email + Capture popup (this is the metadata+snippet+note choice).""" + name = item.get("investor_name") or "Unknown investor" + direction = "Sent" if item.get("direction") == "sent" else "Received" + frm = item.get("from_name") or item.get("from_email") or "?" + lines = [f"📧 Proposed **grid note** for **{name}** ({direction})"] + if item.get("email_subject"): + lines.append(f"· Subject: {item['email_subject']}") + if item.get("email_date"): + lines.append(f"· Date: {item['email_date']}") + lines.append(f"· From: {frm}") + snippet = _truncate(item.get("snippet"), _SNIPPET_MAX) + if snippet: + lines.append(f"· Email: {snippet}") + lines.append("") + lines.append(f"📝 Draft note: {item.get('proposed_note') or '(empty)'}") + lines.append("") + lines.append("Reply **yes** to add it to the grid, **no** to dismiss, or just tell me how to " + "change the note (e.g. *say we discussed the Q3 raise*).") + return "\n".join(lines) + + +def closure_line(status): + """Posted in-thread when a proposal was decided on the WEB while its Matrix thread was open.""" + verb = "approved ✅ and added to the grid" if status == "approved" else "dismissed 🗑️" + return f"This was {verb} on the web — nothing more to do here. Thread closed." + + +def interpret(text): + """Classify an in-thread reply: 'approve' | 'reject' | 'revise' (anything else → revise the note).""" + t = (text or "").strip().lower() + if t in _YES: + return "approve" + if t in _NO: + return "reject" + return "revise" + + +REVISE_SYSTEM = ( + "You revise a single CRM note from a short instruction a venture-fund team member typed. " + "You are given the CURRENT note and an INSTRUCTION. Apply the instruction and reply with " + "ONLY a JSON object of the form {\"note\": \"\"}. Keep it to one or two " + "factual sentences, no preamble. Output JSON only." +) + + +def revise_note(note, instruction, parse_fn=spark.parse_json): + """Re-draft the note via local Qwen from a free-form instruction (no Claude, no scrub — same + local-only basis as the intake parse). Returns the new note text, or None if the model gave + nothing usable / unchanged, in which case the caller re-prompts. `parse_fn` is injectable for + tests.""" + prompt = "CURRENT:\n" + (note or "") + "\n\nINSTRUCTION:\n" + (instruction or "").strip() + raw = parse_fn(prompt, system=REVISE_SYSTEM, max_tokens=400) or {} + new = raw.get("note") if isinstance(raw, dict) else None + new = (new or "").strip() + if not new or new == (note or "").strip(): + return None + return new diff --git a/backend/matrix_intake/settings.py b/backend/matrix_intake/settings.py index c070c94..ed1aff4 100644 --- a/backend/matrix_intake/settings.py +++ b/backend/matrix_intake/settings.py @@ -61,3 +61,10 @@ def crm_settings(): # unset/empty just means no roster framing, i.e. the prior behavior. def team_roster(): return [n.strip() for n in os.environ.get("INTAKE_TEAM_ROSTER", "").split(",") if n.strip()] + + +# Dedicated room for reviewing CRM-drafted email-activity proposals (the CRM→Matrix push leg). +# Separate from the intake room so high-volume email proposals don't drown the conversational +# intake flow. Unset/empty disables the whole email-review poll loop (the bot just does intake). +def email_review_room(): + return os.environ.get("MATRIX_EMAIL_REVIEW_ROOM", "").strip() diff --git a/backend/matrix_intake/test_email_proposals.py b/backend/matrix_intake/test_email_proposals.py new file mode 100644 index 0000000..fde9274 --- /dev/null +++ b/backend/matrix_intake/test_email_proposals.py @@ -0,0 +1,72 @@ +"""Offline tests for the email-proposal review logic (card render, reply grammar, note revision). +The network/Matrix wiring lives in bot.py (live-smoke only); this covers the pure functions.""" +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import email_proposals # noqa: E402 + +ITEM = { + "id": "p1", "investor_name": "Acme Capital", "direction": "received", + "from_name": "Jane Doe", "from_email": "jane@acme.com", + "email_subject": "Re: Fund III", "email_date": "2026-06-02", + "snippet": "thanks for the deck — one question on terms", "proposed_note": "✉ Received: asked about terms", +} + + +def test_interpret_yes_no_else(): + assert email_proposals.interpret("yes") == "approve" + assert email_proposals.interpret(" Y ") == "approve" + assert email_proposals.interpret("✅") == "approve" + assert email_proposals.interpret("no") == "reject" + assert email_proposals.interpret("skip") == "reject" + # anything that isn't a clear yes/no is treated as a revision instruction + assert email_proposals.interpret("say we discussed the Q3 raise") == "revise" + + +def test_render_card_has_context_note_and_actions(): + card = email_proposals.render_card(ITEM) + assert "Acme Capital" in card and "Received" in card + assert "Jane Doe" in card + assert "Re: Fund III" in card and "2026-06-02" in card + assert "thanks for the deck" in card + assert "✉ Received: asked about terms" in card + assert "yes" in card.lower() and "no" in card.lower() + + +def test_render_card_sent_direction(): + assert "(Sent)" in email_proposals.render_card(dict(ITEM, direction="sent")) + + +def test_render_card_truncates_long_snippet(): + card = email_proposals.render_card(dict(ITEM, snippet="x" * 1000)) + assert "…" in card and len(card) < 1000 + + +def test_revise_note_applies_model_output(): + out = email_proposals.revise_note( + "old note", "make it about the Q3 raise", + parse_fn=lambda prompt, system=None, max_tokens=400: {"note": "Discussed the Q3 raise."}) + assert out == "Discussed the Q3 raise." + + +def test_revise_note_noop_or_empty_returns_none(): + # model echoes the same note unchanged -> None so the caller re-prompts (not "Updated") + assert email_proposals.revise_note("same", "x", parse_fn=lambda *a, **k: {"note": "same"}) is None + # model returns nothing usable -> None + assert email_proposals.revise_note("n", "y", parse_fn=lambda *a, **k: {}) is None + assert email_proposals.revise_note("n", "y", parse_fn=lambda *a, **k: None) is None + + +def test_closure_line_reflects_status(): + assert "approved" in email_proposals.closure_line("approved").lower() + assert "dismiss" in email_proposals.closure_line("dismissed").lower() + + +if __name__ == "__main__": + fns = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)] + for fn in fns: + fn() + print(f"ok {fn.__name__}") + print(f"\n{len(fns)} passed") diff --git a/backend/server.py b/backend/server.py index acba154..e595f3f 100644 --- a/backend/server.py +++ b/backend/server.py @@ -619,6 +619,14 @@ def parse_iso_utc(ts): def require_admin(user): return bool(user and user.get('role') == 'admin') +def require_bot_or_admin(user): + """Gate for agent/bot-facing endpoints: a dedicated 'bot' service account OR an admin + (admins keep parity for debugging/curl). The 'bot' role is authenticated-but-not-admin — + it never passes require_admin, so a bot credential cannot reach user-management, security, + or settings. Reach (which endpoints) is controlled here; autonomy (acting without a human) + stays governed by the per-feature draft->approve gate, independent of role.""" + return bool(user and user.get('role') in ('admin', 'bot')) + def log_audit(conn, user_id, entity_type, entity_id, action, changes=None): conn.execute( "INSERT INTO audit_log (id, user_id, entity_type, entity_id, action, changes) VALUES (?, ?, ?, ?, ?, ?)", @@ -2068,6 +2076,9 @@ class CRMHandler(BaseHTTPRequestHandler): # Matrix intake bot — new-vs-existing lookup for its in-thread proposal if path == '/api/intake/match': return self.handle_intake_match(user, params) + # Matrix review bot — email-activity proposal work-lists (to_post/open/to_close) + if path == '/api/intake/email-proposals': + return self.handle_list_bot_email_proposals(user) # Users if path == '/api/users': @@ -2187,6 +2198,10 @@ class CRMHandler(BaseHTTPRequestHandler): return self.handle_decide_activity_proposal(user, path.split('/')[-2], 'approve', body) if re.match(r'^/api/activity/proposals/[^/]+/dismiss$', path): return self.handle_decide_activity_proposal(user, path.split('/')[-2], 'dismiss', body) + if re.match(r'^/api/intake/email-proposals/[^/]+/matrix$', path): + return self.handle_mark_email_proposal_matrix(user, path.split('/')[-2], body) + if re.match(r'^/api/intake/email-proposals/[^/]+/decide$', path): + return self.handle_decide_email_proposal_matrix(user, path.split('/')[-2], body) if re.match(r'^/api/thesis/nodes/[^/]+/choose$', path): return self.handle_choose_variant(user, path.split('/')[-2]) if re.match(r'^/api/thesis/lines/[^/]+/approve$', path): @@ -3964,6 +3979,59 @@ class CRMHandler(BaseHTTPRequestHandler): return self.send_error_json(res["error"], code) return self.send_json({"data": res}) + # ─── Matrix review-bot bridge for email-activity proposals (bot-or-admin) ─── + def handle_list_bot_email_proposals(self, user): + """The bot's poll endpoint: {to_post, open, to_close}. Bot-or-admin (the proposals + carry LP email content, so this stays off the member tier).""" + if not require_bot_or_admin(user): + return self.send_error_json("Bot or admin required", 403) + conn = get_db() + try: + return self.send_json({"data": list_bot_email_proposals(conn)}) + finally: + conn.close() + + def handle_mark_email_proposal_matrix(self, user, proposal_id, body): + """Record Matrix thread state: {event_id} marks the card posted; {closed:true} marks + the thread resolved after the bot announced a web-side decision.""" + if not require_bot_or_admin(user): + return self.send_error_json("Bot or admin required", 403) + body = body or {} + conn = get_db() + try: + if body.get("closed"): + res = mark_proposal_matrix_closed(conn, proposal_id) + else: + event_id = str(body.get("event_id") or "").strip() + if not event_id: + return self.send_error_json("event_id or closed is required") + res = mark_proposal_matrix_posted(conn, proposal_id, event_id) + finally: + conn.close() + if res.get("error"): + return self.send_error_json(res["error"], 404 if res["error"] == "not_found" else 400) + return self.send_json({"data": res}) + + def handle_decide_email_proposal_matrix(self, user, proposal_id, body): + """In-thread Matrix decision relayed by the bot: approve/dismiss (+ optional edited note), + tagged source='matrix' and closing the thread in the same transaction.""" + if not require_bot_or_admin(user): + return self.send_error_json("Bot or admin required", 403) + body = body or {} + decision = str(body.get("decision") or "").strip() + if decision not in ("approve", "dismiss"): + return self.send_error_json("decision must be approve or dismiss") + conn = get_db() + try: + res = decide_email_activity_proposal(conn, proposal_id, decision, user['user_id'], + body.get('note'), source="matrix", close_matrix=True) + finally: + conn.close() + if res.get("error"): + code = {"not_found": 404, "already_decided": 409}.get(res["error"], 400) + return self.send_error_json(res["error"], code) + return self.send_json({"data": res}) + # ─── UI-triggered index jobs + entity-merge review (Phase 1) ─── def handle_index_job(self, user, kind): if not require_admin(user): @@ -4357,8 +4425,10 @@ class CRMHandler(BaseHTTPRequestHandler): return self.send_error_json("password must be at least 8 characters") role = body.get('role', 'member') - if role not in ('admin', 'member'): - return self.send_error_json("role must be admin or member") + # 'bot' is a deliberately-provisioned agent service account (kept out of the invite UI's + # member/admin dropdown) — authenticated but never an admin. See require_bot_or_admin. + if role not in ('admin', 'member', 'bot'): + return self.send_error_json("role must be admin, member, or bot") conn = get_db() existing = conn.execute( @@ -4417,9 +4487,9 @@ class CRMHandler(BaseHTTPRequestHandler): if 'role' in body: role = str(body.get('role')) - if role not in ('admin', 'member'): + if role not in ('admin', 'member', 'bot'): conn.close() - return self.send_error_json("role must be admin or member") + return self.send_error_json("role must be admin, member, or bot") sets.append("role = ?") args.append(role) @@ -5728,8 +5798,14 @@ def list_email_activity_proposals(conn, status="pending", limit=200): return [] -def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_note=None): - """Approve (optionally with an edited note -> append to grid) or dismiss a proposal.""" +def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_note=None, + source="crm_ui", close_matrix=False): + """Approve (optionally with an edited note -> append to grid) or dismiss a proposal. + + `source` records the channel in the audit log ('crm_ui' for the web panel, 'matrix' for an + in-thread approval relayed by the review bot). `close_matrix` is set by the bot when the + decision was made in-thread: it stamps the Matrix side row closed in the same transaction so + the web->Matrix close path (list_bot_email_proposals.to_close) won't re-announce it.""" p = conn.execute("SELECT * FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone() if not p: return {"error": "not_found"} @@ -5747,15 +5823,85 @@ def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_ action, result = "email.activity_dismissed", {"status": "dismissed"} else: return {"error": "bad_decision"} + if close_matrix: + _mark_proposal_matrix(conn, proposal_id, closed_at=now()) conn.execute( "INSERT INTO interaction_log (id, ts, actor_type, actor_id, action, target_type, target_id, payload, source, created_at) " "VALUES (?,?,?,?,?,?,?,?,?,?)", (generate_id(), now(), "human", user_id, action, "fundraising_investor", p["investor_id"], - json.dumps({"proposal_id": proposal_id}), "crm_ui", now())) + json.dumps({"proposal_id": proposal_id}), source, now())) conn.commit() return result +# ─── Matrix review-bot bridge for email-activity proposals (Features 2/3) ────── +# The CRM (box, stdlib, no matrix-nio) can't post to Matrix, so the intake bot (Spark) PULLS +# pending proposals via list_bot_email_proposals, posts a review card to the dedicated review +# room, and writes the thread-root event_id back here. State lives CRM-side (email_proposal_matrix) +# so both surfaces stay in sync and it survives a bot restart. All queries degrade to empty when +# the email integration tables are absent (OperationalError), mirroring list_email_activity_proposals. +_BOT_PROPOSAL_COLS = ( + "SELECT p.id, p.investor_name, p.direction, p.summary, p.proposed_note, " + "p.email_subject, p.email_date, e.from_name, e.from_email, e.snippet, " + "m.event_id AS event_id, p.status AS status " + "FROM email_activity_proposals p " + "LEFT JOIN email_proposal_matrix m ON m.proposal_id = p.id " + "LEFT JOIN emails e ON e.id = p.email_id ") + + +def list_bot_email_proposals(conn, limit=100): + """The three work-lists the Matrix review bot polls: + to_post — pending, not yet posted to Matrix -> bot posts a review card. + open — pending, posted, not closed -> live threads; the bot rebuilds its + event_id->proposal routing map from these after a restart. + to_close — decided on the WEB while a thread is open -> bot announces it in-thread, closes. + Each item carries the card content (investor, direction, subject, date, from, snippet, note).""" + try: + to_post = [dict(r) for r in conn.execute( + _BOT_PROPOSAL_COLS + "WHERE p.status='pending' AND (m.proposal_id IS NULL OR m.posted_at IS NULL) " + "ORDER BY p.email_date ASC, p.created_at ASC LIMIT ?", (limit,))] + open_threads = [dict(r) for r in conn.execute( + _BOT_PROPOSAL_COLS + "WHERE p.status='pending' AND m.posted_at IS NOT NULL AND m.closed_at IS NULL " + "ORDER BY p.email_date ASC, p.created_at ASC LIMIT ?", (limit,))] + to_close = [dict(r) for r in conn.execute( + _BOT_PROPOSAL_COLS + "WHERE p.status!='pending' AND m.posted_at IS NOT NULL AND m.closed_at IS NULL " + "ORDER BY p.decided_at ASC LIMIT ?", (limit,))] + except sqlite3.OperationalError: + return {"to_post": [], "open": [], "to_close": []} + return {"to_post": to_post, "open": open_threads, "to_close": to_close} + + +def _mark_proposal_matrix(conn, proposal_id, *, event_id=None, posted_at=None, closed_at=None): + """Idempotent upsert of the 1:1 Matrix side row. Only the passed fields are written.""" + cols, vals, sets = ["proposal_id"], [proposal_id], [] + for name, val in (("event_id", event_id), ("posted_at", posted_at), ("closed_at", closed_at)): + if val is not None: + cols.append(name); vals.append(val); sets.append(f"{name}=excluded.{name}") + placeholders = ",".join("?" for _ in cols) + sql = f"INSERT INTO email_proposal_matrix ({','.join(cols)}) VALUES ({placeholders})" + if sets: + sql += " ON CONFLICT(proposal_id) DO UPDATE SET " + ",".join(sets) + conn.execute(sql, vals) + + +def mark_proposal_matrix_posted(conn, proposal_id, event_id): + """Record that the bot posted a review card (thread root = event_id).""" + if not conn.execute("SELECT 1 FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone(): + return {"error": "not_found"} + _mark_proposal_matrix(conn, proposal_id, event_id=event_id, posted_at=now()) + conn.commit() + return {"ok": True} + + +def mark_proposal_matrix_closed(conn, proposal_id): + """Mark the Matrix review thread resolved (the bot announced a web-side decision).""" + if not conn.execute("SELECT 1 FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone(): + return {"error": "not_found"} + _mark_proposal_matrix(conn, proposal_id, closed_at=now()) + conn.commit() + return {"ok": True} + + # ─── Main Entry Point ──────────────────────────────────────────────────────── def main(): diff --git a/backend/test_email_proposal_matrix.py b/backend/test_email_proposal_matrix.py new file mode 100644 index 0000000..5b817db --- /dev/null +++ b/backend/test_email_proposal_matrix.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +"""Test the Matrix review-bot bridge for email-activity proposals (Features 2/3): +the bot work-lists (to_post / open / to_close), the Matrix side-row mark helpers, and an +in-thread (source='matrix') decision that closes the thread — plus the bot-or-admin role gate. +Synthetic data only (guardrail #9). The local model is stubbed. +Run: cd backend && python3 test_email_proposal_matrix.py +""" +import json +import os +import sqlite3 +import sys +import tempfile + +os.environ["CRM_DB_PATH"] = os.path.join(tempfile.mkdtemp(), "crm.db") +os.environ.setdefault("CRM_DATA_DIR", os.path.dirname(os.environ["CRM_DB_PATH"])) +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import server # noqa: E402 + +server._summarize_email_gist = lambda subject, body: "fundraising update; proposed a call" + +FAILS = [] + + +def check(cond, msg): + print((" PASS " if cond else " FAIL ") + msg) + if not cond: + FAILS.append(msg) + + +def setup(): + conn = sqlite3.connect(os.environ["CRM_DB_PATH"]) + conn.row_factory = sqlite3.Row + conn.executescript(""" + CREATE TABLE app_settings (key TEXT PRIMARY KEY, value_json TEXT, updated_at TEXT); + CREATE TABLE email_accounts (id TEXT, email_address TEXT, sync_enabled INT DEFAULT 1, sync_status TEXT, backfill_complete INT); + CREATE TABLE emails (id TEXT PRIMARY KEY, subject TEXT, body_text TEXT, snippet TEXT, from_name TEXT, from_email TEXT, sent_at TEXT, is_matched INT, match_status TEXT); + CREATE TABLE email_investor_links (id TEXT, email_id TEXT, fundraising_investor_id TEXT, organization_id TEXT, contact_id TEXT, match_confidence REAL); + CREATE TABLE email_activity_proposals (id TEXT PRIMARY KEY, email_id TEXT UNIQUE, investor_id TEXT, investor_name TEXT, + direction TEXT, summary TEXT, proposed_note TEXT, email_subject TEXT, email_date TEXT, status TEXT DEFAULT 'pending', + decided_by TEXT, decided_at TEXT, final_note TEXT, created_at TEXT); + CREATE TABLE email_proposal_matrix (proposal_id TEXT PRIMARY KEY, event_id TEXT, posted_at TEXT, closed_at TEXT, created_at TEXT); + CREATE TABLE users (id TEXT PRIMARY KEY, username TEXT); + CREATE TABLE fundraising_investors (id TEXT PRIMARY KEY, investor_name TEXT, notes TEXT); + CREATE TABLE fundraising_state (id TEXT PRIMARY KEY, grid_json TEXT, views_json TEXT, version INT, + updated_by TEXT REFERENCES users(id), updated_at TEXT); + CREATE TABLE interaction_log (id TEXT PRIMARY KEY, ts TEXT, actor_type TEXT, actor_id TEXT, action TEXT, target_type TEXT, target_id TEXT, payload TEXT, source TEXT, created_at TEXT); + """) + conn.execute("INSERT INTO users (id,username) VALUES ('user-1','grant')") + conn.execute("INSERT INTO app_settings VALUES ('email_activity_since', ?, ?)", (json.dumps("2026-01-01T00:00:00"), "x")) + conn.execute("INSERT INTO email_accounts (id,email_address) VALUES ('a','grant@ten31.xyz')") + conn.execute("INSERT INTO fundraising_investors (id,investor_name,notes) VALUES ('inv1','Harbor & Vine','existing note')") + grid = {"columns": [], "rows": [{"id": "inv1", "investor_name": "Harbor & Vine", "notes": "existing note"}]} + conn.execute("INSERT INTO fundraising_state (id,grid_json,views_json,version) VALUES ('main',?,?,1)", (json.dumps(grid), "[]")) + conn.executemany("INSERT INTO emails (id,subject,body_text,snippet,from_name,from_email,sent_at,is_matched,match_status) VALUES (?,?,?,?,?,?,?,1,'matched')", [ + ("e1", "Fund III", "Here is the update", "the quarterly update is attached", "Grant", "grant@ten31.xyz", "2026-06-01T10:00:00"), + ("e2", "Re: Fund III", "Thanks, a question", "thanks — one question on terms", "LP Contact", "lp@harborvine.example", "2026-06-02T10:00:00"), + ]) + conn.executemany("INSERT INTO email_investor_links (id,email_id,fundraising_investor_id,match_confidence) VALUES (?,?, 'inv1', 1.0)", + [("l1", "e1"), ("l2", "e2")]) + conn.commit() + conn.close() + + +def main(): + setup() + + # role gate: bot passes the agent gate but is NOT an admin; member passes neither. + check(server.require_bot_or_admin({"role": "bot"}), "bot passes require_bot_or_admin") + check(server.require_bot_or_admin({"role": "admin"}), "admin passes require_bot_or_admin") + check(not server.require_bot_or_admin({"role": "member"}), "member does NOT pass require_bot_or_admin") + check(not server.require_admin({"role": "bot"}), "bot is NOT an admin (no user-mgmt/settings reach)") + + check(server.propose_email_activity_notes().get("proposed") == 2, "drafts 2 proposals") + conn = server.get_db() + props = server.list_email_activity_proposals(conn, status="pending") + by_email = {p["email_id"]: p for p in props} + p_a, p_b = by_email["e1"], by_email["e2"] + + # Both are pending + un-posted → both in to_post; card carries from/snippet/note context. + lists = server.list_bot_email_proposals(conn) + check(len(lists["to_post"]) == 2 and not lists["open"] and not lists["to_close"], "both proposals queued to_post") + card = next(it for it in lists["to_post"] if it["id"] == p_a["id"]) + check(card.get("from_name") == "Grant" and "quarterly update" in (card.get("snippet") or ""), "card carries from_name + snippet") + check("✉" in (card.get("proposed_note") or ""), "card carries the drafted note") + + # Post p_a to Matrix → it leaves to_post and becomes an open thread (event id recorded). + server.mark_proposal_matrix_posted(conn, p_a["id"], "evtA") + lists = server.list_bot_email_proposals(conn) + check(len(lists["to_post"]) == 1 and lists["to_post"][0]["id"] == p_b["id"], "posting p_a leaves only p_b to_post") + check(len(lists["open"]) == 1 and lists["open"][0]["id"] == p_a["id"] and lists["open"][0]["event_id"] == "evtA", + "posted p_a is an open thread carrying its event id") + + # Decide p_a IN-THREAD on Matrix (approve + close in one transaction). + r = server.decide_email_activity_proposal(conn, p_a["id"], "approve", "user-1", source="matrix", close_matrix=True) + check(r.get("status") == "approved" and r.get("placed_in_grid") is True, "matrix approve appends to the grid") + lists = server.list_bot_email_proposals(conn) + check(not any(it["id"] == p_a["id"] for it in lists["open"] + lists["to_close"]), + "matrix-decided proposal is closed (not re-announced via to_close)") + src = conn.execute("SELECT source FROM interaction_log WHERE action='email.activity_approved'").fetchone()["source"] + check(src == "matrix", "matrix decision is audited source='matrix'") + + # Web-decide path: post p_b, then dismiss it on the WEB (default source, no close) → the bot + # must see it in to_close so it can announce the web decision in-thread, then close. + server.mark_proposal_matrix_posted(conn, p_b["id"], "evtB") + server.decide_email_activity_proposal(conn, p_b["id"], "dismiss", "user-1") # web path: source crm_ui, no close + lists = server.list_bot_email_proposals(conn) + check(len(lists["to_close"]) == 1 and lists["to_close"][0]["id"] == p_b["id"] and lists["to_close"][0]["status"] == "dismissed", + "web-decided open thread surfaces in to_close") + src2 = conn.execute("SELECT source FROM interaction_log WHERE action='email.activity_dismissed'").fetchone()["source"] + check(src2 == "crm_ui", "web decision is audited source='crm_ui'") + + server.mark_proposal_matrix_closed(conn, p_b["id"]) + lists = server.list_bot_email_proposals(conn) + check(not lists["to_close"] and not lists["open"], "closing the thread clears the work-lists") + + # Marking a non-existent proposal is a clean not_found, not a crash. + check(server.mark_proposal_matrix_posted(conn, "nope", "evtX").get("error") == "not_found", "mark posted on unknown id -> not_found") + conn.close() + + if FAILS: + print(f"\nFAILED ({len(FAILS)})") + for f in FAILS: + print(" - " + f) + sys.exit(1) + print("\nALL PASS (email-proposal Matrix bridge)") + + +if __name__ == "__main__": + main() diff --git a/docs/guides/email.md b/docs/guides/email.md index 30bf6a1..86c641b 100644 --- a/docs/guides/email.md +++ b/docs/guides/email.md @@ -12,7 +12,7 @@ Read this before editing Gmail capture or draft creation. ## What it does - `backend/email_integration/` captures Gmail via **domain-wide delegation** (`credentials.py`, `matcher.py`, `parser.py`, `db.py`, `sync.py`, `scheduler.py`, `routes.py`) and creates Tier-B in-thread drafts (`compose.py`). It has its own `migrations/`. -- Captured email becomes CRM activity through a **propose → approve** flow — nothing lands on a contact record until a human approves the proposal. +- Captured email becomes CRM activity through a **propose → approve** flow — nothing lands on a contact record until a human approves the proposal. The proposed grid notes show on the **Email Capture** page (admin-only): each card has a **View email** toggle that fetches `GET /api/email/detail?id=` and shows the source email inline (from/to/cc/date/subject + scrollable body) so you can judge the note against it. The same proposals can also be reviewed/approved/edited from a **dedicated Matrix room**, kept in sync with this panel (decide on either surface; the other reflects it) — that CRM→Matrix bridge lives in the **review bot**, see `docs/guides/matrix-intake.md`. The proposal model itself (`email_activity_proposals` + the `propose_email_activity_notes` drafter + the decide path) lives in `backend/server.py`, not this package. ## Hard rule diff --git a/docs/guides/matrix-intake.md b/docs/guides/matrix-intake.md index 7163b27..ee3c21c 100644 --- a/docs/guides/matrix-intake.md +++ b/docs/guides/matrix-intake.md @@ -90,6 +90,42 @@ rows ≥ `min_score` (0.62), ranked, capped at 5: only the shortlist, never the whole LP list — intentionally NOT built in this pass, because the deterministic filter already surfaces every duplicate the human then resolves. +## Email-activity proposal review (the CRM→Matrix bridge, v0.1.0:89) + +A second, separate flow runs alongside intake: reviewing the **proposed grid notes** the CRM +drafts from newly-matched email (`server.propose_email_activity_notes`, surfaced on the web Email +Capture panel). The bot lets the team approve/dismiss/edit those on mobile, kept **in sync** with +the web panel. The CRM (box, stdlib, no matrix-nio) can't post to Matrix, so the bot **pulls**. + +- **Dedicated room** (`MATRIX_EMAIL_REVIEW_ROOM`, see *Config*) — separate from the intake room + so high-volume email proposals don't drown the conversational intake. Unset → the whole leg is + off (the bot just does intake). The bot must be a **member** of this room. +- **Poll loop** (`bot.poll_email_proposals`, every `EMAIL_POLL_SEC`=20s) calls `crm_client. + list_email_proposals` → `GET /api/intake/email-proposals`, which returns three work-lists: + - **to_post** — pending, not yet posted → the bot posts a review card (metadata + a short email + **snippet** + the drafted note; the full body is the web popup's job, kept compact for mobile), + then records the thread-root event id via `POST .../{id}/matrix {event_id}`. + - **open** — pending, posted, not closed → the bot rebuilds its `event_id → proposal` routing map + from these on **every poll**, so replies still route **after a bot restart** (unlike intake's + in-memory-only store — the state lives CRM-side in `email_proposal_matrix`). + - **to_close** — decided on the **web** while a thread was open → the bot posts a "decided on the + web — thread closed" line and `POST .../{id}/matrix {closed:true}`. +- **In-thread replies** (`bot.handle_email_reply`, `email_proposals.interpret`): `yes` → + `POST .../{id}/decide {decision:"approve", note}` (appends the note to the grid, source='matrix', + closes the thread atomically); `no` → dismiss; **anything else → NL revision of the note** via + local Qwen (`email_proposals.revise_note`, no Claude/scrub) — re-rendered for re-approval, so the + draft→approve gate holds. A no-op/empty revision re-prompts instead of saying "Updated". +- **Two surfaces, one source of truth.** Decide on the web → the bot announces + closes the thread; + decide on Matrix → the web panel polls `/api/activity/proposals` (~25s) and the card clears. + `email_proposal_matrix` (1:1 side row, migration `0003`) carries `event_id`/`posted_at`/`closed_at`; + a matrix decision sets `closed_at` in the same txn so it's never re-announced via `to_close`. +- **Pure logic is `email_proposals.py`** (card render, reply grammar, note revision) — unit-tested + offline in `test_email_proposals.py`; the async poll/post wiring is in `bot.py` (live-smoke only). +- **Known minors (low-likelihood, ~5-person team):** if the CRM is unreachable *between* posting a + card and recording its event id, the next poll re-posts a duplicate card (the orphan's replies + won't route — re-send/decide the recorded one). A mid-revise bot restart loses the in-memory + revised note (rebuilt from `open` = the original `proposed_note`; still a valid proposal). + ## Rules / gotchas - **Module-name collision:** the intake config module is `settings.py`, **not** `config.py`, @@ -161,7 +197,11 @@ rows ≥ `min_score` (0.62), ranked, capped at 5: - **Server-side endpoints ship in the s9pk, not the bot.** `GET /api/intake/match` and the `source` provenance on `log-communication` live in `backend/server.py`, so they reach the box only via an **s9pk build + install** — a bot restart won't deliver them. (Missed in v83: the - box 404'd `/api/intake/match` until **v0.1.0:84**.) + box 404'd `/api/intake/match` until **v0.1.0:84**.) **Same split for the email-review bridge + (v0.1.0:89):** the `/api/intake/email-proposals*` endpoints + the `email_proposal_matrix` + migration (`0003`) + the `bot` role ship in the **s9pk**; the poll loop + review-room handling + ship on the **Spark** (git pull + restart). A bot restart against a pre-v89 box returns nothing + useful (404/empty), so install the s9pk first, then set the bot user's role + the review room. - **`CRM_API_BASE` is the box over the LAN, not localhost** (bot on the Spark, CRM on the box). `https://immense-voyage.local` (443) is the **StartOS dashboard**, not the CRM — the CRM has its own interface address (the URL you open in a browser); container port 8080 isn't @@ -174,6 +214,19 @@ All in `.env` (names in `.env.example`): `MATRIX_HOMESERVER`, `MATRIX_USER`, `CRM_BOT_USERNAME`, `CRM_BOT_PASSWORD`, `CRM_API_VERIFY_TLS`. Spark settings are inherited from the ingest client (`SPARK_CONTROL_URL`, `CRM_CHAT_MODEL`). +- **`MATRIX_EMAIL_REVIEW_ROOM`** (optional) — the dedicated room for the email-activity proposal + review leg (above). Unset/empty disables that leg entirely (the bot does intake only). The bot + must be invited to + joined in this room. Read once at startup, like the room/roster. +- **Bot CRM user needs role `bot`.** The email-proposal endpoints (`/api/intake/email-proposals*`) + are gated to `require_bot_or_admin` because they expose LP email content (the proposals are + admin-only on the web). The `bot` role is **authenticated-but-not-admin** — it passes these + endpoints + the auth-only ones the bot already uses (login, `/api/intake/match`, + `log-communication`), but **never** `require_admin` (no user-management/settings/security reach). + One-time flip of the existing service account (kept out of the invite UI's member/admin dropdown + — provision deliberately): an admin `PATCH /api/users/ {"role":"bot"}`, or on the box + `UPDATE users SET role='bot' WHERE username='';`. Role controls *reach*; the + draft→approve gate (a human still approves every write) controls *autonomy* — two separate axes. + - **`INTAKE_TEAM_ROSTER`** (optional, comma-separated) — Ten31 team-member names that frame the parse (see *Flow* step 1). Use the **first names as actually typed in the room** ("Grant, Jonathan, …"). Read once at startup by `settings.team_roster()`, so **a roster change needs a diff --git a/frontend/index.html b/frontend/index.html index 97b86f6..7287034 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -10037,6 +10037,8 @@ const [proposals, setProposals] = useState([]); const [edits, setEdits] = useState({}); const [deciding, setDeciding] = useState(null); + const [openEmail, setOpenEmail] = useState(null); // proposal id whose source email is expanded + const [emailCache, setEmailCache] = useState({}); // email_id -> {loading, data, error} const load = useCallback(async () => { let s; @@ -10086,6 +10088,21 @@ return () => { cancelled = true; clearInterval(iv); }; }, [backfilling, load]); + // Steady-state poll of just the proposals so a decision made on Matrix (approve/dismiss + // in the review room) clears its card here without a manual reload — the mirror of the + // bot announcing a web-side decision in-thread. Admin-only (only admins see proposals). + const refreshProposals = useCallback(async () => { + try { + const pr = await api('/api/activity/proposals', {}, token); + setProposals(Array.isArray(pr?.proposals) ? pr.proposals : []); + } catch (_) { /* admin-only / transient — leave the current list */ } + }, [token]); + useEffect(() => { + if (!isAdmin) return undefined; + const iv = setInterval(() => { refreshProposals(); }, 25000); + return () => clearInterval(iv); + }, [isAdmin, refreshProposals]); + const runAction = async (key, endpoint, successMsg, confirmMsg, body) => { if (busy) return; if (confirmMsg && !window.confirm(confirmMsg)) return; @@ -10120,6 +10137,51 @@ } }; + // Click a proposal to see the email it was drafted from (from/to/cc/date/subject + + // scrollable body) so you can judge whether the note is right. Lazily fetched + + // cached per email; reuses the admin-only /api/email/detail used by Communications. + const toggleEmail = async (p) => { + if (openEmail === p.id) { setOpenEmail(null); return; } + setOpenEmail(p.id); + const eid = p.email_id; + if (!eid || emailCache[eid]) return; + setEmailCache((c) => ({ ...c, [eid]: { loading: true } })); + try { + const res = await api(`/api/email/detail?id=${encodeURIComponent(eid)}`, {}, token); + setEmailCache((c) => ({ ...c, [eid]: { loading: false, data: res } })); + } catch (err) { + setEmailCache((c) => ({ ...c, [eid]: { loading: false, error: getErrorMessage(err, 'Failed to load email') } })); + } + }; + + const renderProposalEmail = (p) => { + const det = emailCache[p.email_id]; + if (!det) return null; + if (det.loading) return
; + if (det.error) return
{det.error}
; + const d = det.data || {}; + const rcpt = (kind) => (d.recipients || []).filter((r) => r.kind === kind) + .map((r) => r.display_name ? `${r.display_name} <${r.address}>` : r.address).join(', '); + const to = rcpt('to'), cc = rcpt('cc'); + const from = d.from_name ? `${d.from_name} <${d.from_email || ''}>` : (d.from_email || '—'); + const lbl = { fontSize: '11px', color: '#8ea2b7' }; + return ( +
+
From: {from}
+ {to &&
To: {to}
} + {cc &&
Cc: {cc}
} +
Date: {d.sent_at ? new Date(d.sent_at).toLocaleString() : '—'}
+
Subject: {d.subject || '(no subject)'}
+ {(d.attachments || []).length > 0 && ( +
Attachments: {d.attachments.map((a) => a.filename).join(', ')}
+ )} + 
+                            {d.body_text || (d.has_html ? '(HTML-only email — open in Gmail to view formatting)' : '(no body captured)')}
+
+
+ ); + }; + if (loading) return
; if (error) return
{error}
; if (!status) return
No data
; @@ -10204,7 +10266,16 @@ {deciding === p.id ? 'Adding…' : 'Approve & add to grid'} + {p.email_id && ( + + )} + {openEmail === p.id && renderProposalEmail(p)} ))} diff --git a/start9/0.4/startos/utils.ts b/start9/0.4/startos/utils.ts index 183b3e7..434e3b7 100644 --- a/start9/0.4/startos/utils.ts +++ b/start9/0.4/startos/utils.ts @@ -53,8 +53,9 @@ export const PACKAGE_TITLE = 'Ten31 Database' // * 0.1.0:85 (cosmetic: drop the redundant "[note]" tag from the fundraising-grid note line — now "YYYY-MM-DD Contact: summary"; informative comm types [call, meeting, …] keep their "[type]" tag; shared by the Matrix intake bot + grid-UI logging; no schema change) // * 0.1.0:86 (Matrix intake fuzzy matching: GET /api/intake/match now returns ranked `candidates` [fuzzy near-matches — deterministic difflib name similarity + token overlap + email edit-distance ≤ 2, legal-suffix-aware] alongside the exact `match`, so the bot can surface near-duplicates ["Charlie"/"Charles", "Acme Capital"/"Acme Capital LLC", a one-char email typo] for human confirmation instead of silently creating a second investor; the bot-side disambiguation + conversational-edit UX ships on the Spark, not the s9pk; code-only, no schema change) // * 0.1.0:87 (Adopt the Pipeline — grid drives the deal board: new "Add to Pipeline" row action creates+links an opportunity via opportunities.fundraising_investor_id [migration 0005, additive], reusing the grid's synced contact [no POST /api/contacts side-door] and mapping the grid lead→owner; idempotent [one live opp/investor, re-link never reseeds board-owned stage/probability]; read-only Pipeline + Pipeline Stage grid columns derived live from the linked opp; "Remove from Pipeline" soft-deletes the opp [grid row untouched]; deleting a grid investor archives its orphaned opp; folds in the soft-delete fix for the pipeline report + dashboard aggregates [archived opps no longer counted]) -// * Current: 0.1.0:88 (frontend-only: retire the Pipeline page's "+ New Opportunity" button + its create-by-contact modal — opportunities are now born only from a fundraising-grid investor row ["+ Pipeline"], so the board is a view + stage-management surface; replaced the button with a muted "Add deals from the Fundraising Grid" hint; removed the now-dead handler/state + the page's unused /api/contacts fetch) -export const PACKAGE_VERSION = '0.1.0:88' +// * 0.1.0:88 (frontend-only: retire the Pipeline page's "+ New Opportunity" button + its create-by-contact modal — opportunities are now born only from a fundraising-grid investor row ["+ Pipeline"], so the board is a view + stage-management surface; replaced the button with a muted "Add deals from the Fundraising Grid" hint; removed the now-dead handler/state + the page's unused /api/contacts fetch) +// * Current: 0.1.0:89 (email-proposal review over Matrix + a dedicated agent role: Email Capture's proposed grid notes gain a click-to-view inline popup of the source email [from/to/cc/date/subject/scrollable body, via the existing GET /api/email/detail]; and a CRM→Matrix review bridge — the intake bot [Spark] pulls pending proposals, posts a review card to a dedicated review room [MATRIX_EMAIL_REVIEW_ROOM], and relays in-thread yes/no/NL-edit back to the CRM, with web panel ↔ Matrix kept in sync [decide on either surface; the other reflects it]. New side table email_proposal_matrix [email-integration migration 0003, additive + idempotent] holds per-proposal Matrix thread state; new bot-or-admin endpoints GET /api/intake/email-proposals + .../{id}/matrix + .../{id}/decide, gated by a new 'bot' role [authenticated, never admin]. Bot poll loop + review-room handling ship on the Spark, not the s9pk) +export const PACKAGE_VERSION = '0.1.0:89' export const DATA_MOUNT_PATH = '/data' export const WEB_PORT = 8080 diff --git a/start9/0.4/startos/versions/index.ts b/start9/0.4/startos/versions/index.ts index b04cc19..0de93f5 100644 --- a/start9/0.4/startos/versions/index.ts +++ b/start9/0.4/startos/versions/index.ts @@ -49,8 +49,9 @@ import { v_0_1_0_85 } from './v0.1.0.85' import { v_0_1_0_86 } from './v0.1.0.86' import { v_0_1_0_87 } from './v0.1.0.87' import { v_0_1_0_88 } from './v0.1.0.88' +import { v_0_1_0_89 } from './v0.1.0.89' export const versionGraph = VersionGraph.of({ - current: v_0_1_0_88, - other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73, v_0_1_0_74, v_0_1_0_75, v_0_1_0_76, v_0_1_0_77, v_0_1_0_78, v_0_1_0_79, v_0_1_0_80, v_0_1_0_81, v_0_1_0_82, v_0_1_0_83, v_0_1_0_84, v_0_1_0_85, v_0_1_0_86, v_0_1_0_87], + current: v_0_1_0_89, + other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73, v_0_1_0_74, v_0_1_0_75, v_0_1_0_76, v_0_1_0_77, v_0_1_0_78, v_0_1_0_79, v_0_1_0_80, v_0_1_0_81, v_0_1_0_82, v_0_1_0_83, v_0_1_0_84, v_0_1_0_85, v_0_1_0_86, v_0_1_0_87, v_0_1_0_88], }) diff --git a/start9/0.4/startos/versions/v0.1.0.89.ts b/start9/0.4/startos/versions/v0.1.0.89.ts new file mode 100644 index 0000000..914f2f8 --- /dev/null +++ b/start9/0.4/startos/versions/v0.1.0.89.ts @@ -0,0 +1,24 @@ +import { VersionInfo } from '@start9labs/start-sdk' + +// Email-proposal review over Matrix + a dedicated agent role. The CRM-drafted "proposed grid +// notes" (Email Capture panel) gain (1) a click-to-view inline popup of the source email +// (from/to/cc/date/subject/scrollable body, via the existing /api/email/detail) so a reviewer +// can judge the note against the email, and (2) a CRM→Matrix review bridge: the intake bot +// (Spark) pulls pending proposals, posts a review card to a dedicated review room, and relays +// the human's in-thread yes/no/edit back to the CRM — with the web panel and Matrix kept in +// sync (decide on either; the other surface reflects it). New side table email_proposal_matrix +// (email-integration migration 0003, additive + idempotent — CREATE TABLE IF NOT EXISTS) holds +// the per-proposal Matrix thread state. New bot-or-admin endpoints under /api/intake/ +// email-proposals (list/mark/decide), gated by a new 'bot' role (authenticated, never admin). +// The bot's poll loop + review-room handling ship on the Spark (git pull + restart), not here. +export const v_0_1_0_89 = VersionInfo.of({ + version: '0.1.0:89', + releaseNotes: { + en_US: [ + 'Email Capture: click a proposed grid note to see the source email inline', + '(from/to/cc/date/subject + body) before approving, and review/approve/dismiss/edit', + 'proposals from a dedicated Matrix room on mobile — decisions sync both ways.', + ].join(' '), + }, + migrations: { up: async () => {}, down: async () => {} }, +})