Email-proposal review over Matrix + a bot role (v0.1.0:89)

The email-capture "proposed grid notes" gain two review surfaces:

1. Inline source email — each proposed-note card on the Email Capture page
   gets a "View email" toggle that lazily fetches the existing
   GET /api/email/detail and shows from/to/cc/date/subject + scrollable body,
   so a reviewer can judge the note against the email it was drafted from.

2. CRM->Matrix review bridge — the CRM (box, stdlib, no matrix-nio) can't post
   to Matrix, so the intake bot (Spark) PULLS: GET /api/intake/email-proposals
   returns to_post/open/to_close work-lists; the bot posts a review card
   (metadata + snippet + draft note) to a dedicated review room
   (MATRIX_EMAIL_REVIEW_ROOM) and relays in-thread yes / no / NL-edit
   (POST .../{id}/decide, note revised via local Qwen). Decisions sync both
   ways: web decide -> bot announces + closes the thread; Matrix decide -> the
   web panel's ~25s poll clears the card. State lives CRM-side in the new
   email_proposal_matrix side row (email-integration migration 0003, additive
   + idempotent CREATE TABLE IF NOT EXISTS), so it survives a bot restart.

Adds a 'bot' role (authenticated, never admin; require_bot_or_admin) to gate
the email-proposal endpoints rather than handing the bot full admin — the
principled base for the coming agentic capabilities. Role controls reach;
the draft->approve gate still controls autonomy (a human approves every write).

Deploy split: endpoints + migration + role + frontend ship in the s9pk; the
bot poll loop + review-room handling ship on the Spark. The bot's CRM user
must be flipped member->bot and joined to the review room (one-time).

Tests: backend/test_email_proposal_matrix.py + matrix_intake/test_email_proposals.py
(30/30 suite green, render-smoke green, migration verified twice on a DB copy).
This commit is contained in:
Keysat
2026-06-18 09:51:41 -05:00
parent 41def0f014
commit 5faa5ae4d6
16 changed files with 783 additions and 17 deletions
+7
View File
@@ -57,6 +57,13 @@ MATRIX_USER=@intake-bot:<homeserver>
MATRIX_ACCESS_TOKEN=
MATRIX_DEVICE_ID=ten31-intake-bot
MATRIX_INTAKE_ROOM=!<roomid>:<homeserver>
# Dedicated room for reviewing CRM-drafted email-activity proposals (the proposed grid notes the
# Email Capture panel shows). The bot posts a review card per pending proposal here and relays the
# in-thread yes/no/edit back to the CRM, in sync with the web panel. Separate from the intake room
# so high-volume email proposals don't drown the conversational intake. Leave empty to disable the
# whole email-review poll loop. The bot must be a member of this room. Needs the server side in the
# s9pk (≥ v0.1.0:89) and the bot's CRM user set to role 'bot' (see docs/guides/matrix-intake.md).
MATRIX_EMAIL_REVIEW_ROOM=!<roomid>:<homeserver>
# CRM write-back: the bot logs in as a DEDICATED service user (admin-created CRM user;
# the CRM has no service-key path, so it uses normal Bearer-JWT auth).
CRM_API_BASE=http://127.0.0.1:8080
+3 -1
View File
@@ -103,7 +103,9 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude
## Current state
_Phase 0 + Phase 1 built; **box and repo at v0.1.0:88** (deployed & verified live 2026-06-18 — chain …86→88 clean, `0005_grid_pipeline_link.sql` applied on the box, server up; the full Pipeline **+Pipeline → board → advance-stage → remove** round-trip is live-smoked on the box). **The fundraising grid + email capture is the canonical system of record** (2026-06-16) — vestigial classic-CRM surfaces get pruned/repurposed. Deploy/feature history lives in git log + `start9/0.4/startos/versions/`; longer-term backlog + debt in `ROADMAP.md` / `EVALUATION.md`._
_Phase 0 + Phase 1 built; **box at v0.1.0:88, repo at v0.1.0:89 (built, NOT yet deployed)**. v88 deployed & verified live 2026-06-18 — chain …86→88 clean, `0005_grid_pipeline_link.sql` applied on the box, server up; the full Pipeline **+Pipeline → board → advance-stage → remove** round-trip is live-smoked on the box. **The fundraising grid + email capture is the canonical system of record** (2026-06-16) — vestigial classic-CRM surfaces get pruned/repurposed. Deploy/feature history lives in git log + `start9/0.4/startos/versions/`; longer-term backlog + debt in `ROADMAP.md` / `EVALUATION.md`._
- **Email-proposal review over Matrix + a `bot` role — BUILT v0.1.0:89, NOT yet deployed (2026-06-18).** Two asks on the email-capture "proposed grid notes": (1) **inline source-email popup** — each proposed-note card on the Email Capture page gets a **View email** toggle that lazily fetches the existing **`GET /api/email/detail`** and renders from/to/cc/date/subject + scrollable body inline, so you can judge the note against the email (frontend-only, reuses the Communications detail pattern). (2) **CRM→Matrix review bridge** — the CRM (box, no matrix-nio) can't push, so the **intake bot (Spark) PULLS**: new **`GET /api/intake/email-proposals`** returns three work-lists (`to_post`/`open`/`to_close`); the bot posts a review card (metadata + snippet + draft note) to a **dedicated review room** (`MATRIX_EMAIL_REVIEW_ROOM`), records the thread root (`POST .../{id}/matrix`), and relays in-thread **yes / no / NL-edit** (`POST .../{id}/decide`, NL-revise via local Qwen). **Bidirectional sync:** decide on the web → bot announces + closes the thread; decide on Matrix → the web panel's ~25s poll clears the card. State is CRM-side in **`email_proposal_matrix`** (1:1 side row, email-integration migration **`0003`**, additive + idempotent `CREATE TABLE IF NOT EXISTS` — the email runner re-runs every boot via `executescript`, so no `ALTER`), so it survives a bot restart. New **`bot` role** (authenticated, never admin — `require_bot_or_admin`) gates the email-proposal endpoints; the bot's CRM user must be flipped `member→bot` (one-time, kept out of the invite UI). **Deploy split:** endpoints + migration + role + frontend → **s9pk v89 build+install**; poll loop + review-room handling → **Spark git pull + restart**. Tests: `backend/test_email_proposal_matrix.py` + `backend/matrix_intake/test_email_proposals.py`; **30/30 suite green**, render-smoke green, migration verified twice on a copy of `data/crm.db`. **Next: deploy v89 to the box (auth required), set the bot user's role + the review room + invite the bot, then live-smoke both legs** (popup; web↔Matrix round-trip). Guide: `docs/guides/matrix-intake.md` "Email-activity proposal review".
- **Adopt the Pipeline — grid drives the deal board — DEPLOYED & live-smoked 2026-06-18 (v0.1.0:88; the full +Pipeline → board → advance-stage → remove round-trip is verified on the box). v88 (frontend-only): retired the Pipeline page's "+ New Opportunity" button + its create-by-contact modal** — opportunities are now born **only** from a grid investor row (matches how the team works; the board is view + stage-management; button replaced with a muted "Add deals from the Fundraising Grid" hint). An **"Add to Pipeline"** row action on the fundraising grid opens a seed modal (primary contact / target fund / expected amount / stage / probability) and creates a durably-linked `opportunities` row via the new **`opportunities.fundraising_investor_id`** (migration 0005, additive + reversible). **Grid owns the link + seed; the board owns stage/probability/owner** — a grid save never reseeds a live opp (`POST /api/fundraising/pipeline/link` is idempotent, one live opp/investor). Contact is **reused from the grid's synced `fundraising_contacts.contact_id`** (the `POST /api/contacts` side-door is gone); grid `lead`→owner. Two **read-only** grid columns (Pipeline action + Pipeline Stage) are **injected on read** from the live opp and **stripped on write** (never persisted, never dirty the autosave). **Remove from pipeline** (`POST .../unlink`) **soft-deletes the opp; the grid row stays fully intact**; deleting an investor from the grid archives its orphaned opp (`reconcile_grid_pipeline_links`, after `sync_fundraising_relational`). **Folded in:** the standing P2 soft-delete leak in `handle_pipeline_report` + dashboard pipeline aggregates (archived opps no longer counted). Tests: `backend/test_grid_pipeline_link.py`; 28/28 suite green, render-smoke green; migration verified on a copy of `data/crm.db` and **applied clean on the box**. **Next: live-smoke on the box — add an investor to the pipeline, confirm it lands on the board, advance a stage, and remove (opp archived, grid row intact).** Detail + locked decisions in `ROADMAP.md` "Adopt the Pipeline".
@@ -0,0 +1,30 @@
-- ============================================================================
-- email_proposal_matrix — Matrix-review state for an email_activity_proposal,
-- kept 1:1 with the proposal (proposal_id PK). The CRM runs on the box and has
-- no matrix-nio, so it cannot post to Matrix itself: the intake bot (on the Spark)
-- PULLS pending proposals, posts a review card to the dedicated Matrix review room,
-- and writes the thread-root event_id back here. Persisting it CRM-side (not just in
-- the bot's memory) keeps both surfaces in sync and survives a bot restart.
--
-- A SIDE TABLE rather than new columns on email_activity_proposals because the
-- email-integration migration runner (email_integration/db.py:apply_migrations)
-- re-runs every .sql file on every boot via executescript with no ledger — so
-- CREATE TABLE IF NOT EXISTS is idempotent, whereas ALTER ... ADD COLUMN would throw
-- "duplicate column" on the second boot and abort startup. Reversal: DROP TABLE
-- (this runner has no .down.sql convention; cf. 0001/0002).
--
-- posted_at — set once the bot has posted the review card (event_id = thread root).
-- closed_at — set when the thread is resolved: either the bot decided in-thread, OR
-- the bot announced a web-side decision. A posted+decided proposal with
-- closed_at NULL is exactly the bot's signal to post "decided on the web"
-- into the thread and then close it.
-- ============================================================================
CREATE TABLE IF NOT EXISTS email_proposal_matrix (
proposal_id TEXT PRIMARY KEY,
event_id TEXT, -- Matrix thread-root event id of the posted review card
posted_at TEXT,
closed_at TEXT,
created_at TEXT DEFAULT (datetime('now')),
FOREIGN KEY(proposal_id) REFERENCES email_activity_proposals(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_email_proposal_matrix_event ON email_proposal_matrix(event_id);
+100 -3
View File
@@ -14,6 +14,7 @@ import asyncio
from nio import AsyncClient, MatrixRoom, RoomMessageText
import crm_client
import email_proposals
import matrix_io
import parse
import proposals
@@ -25,6 +26,8 @@ UNCLEAR_HELP = (
"or a note like `Note for Acme Capital: wants the Q3 deck, follow up next week`."
)
EMAIL_POLL_SEC = 20 # how often the bot polls the CRM for new/decided email-activity proposals
async def main():
mx = settings.matrix_settings()
@@ -37,6 +40,8 @@ async def main():
roster = settings.team_roster() # frames the parse: teammates do outreach, aren't prospects
if roster:
print(f"matrix-intake: team roster loaded ({len(roster)} names)", flush=True)
review_room = settings.email_review_room() # CRM-drafted email proposals (empty → feature off)
email_threads = {} # Matrix thread-root event_id -> {id, investor_name, note} for an email proposal
async def handle_intake(room_id, root, text):
# A bare yes/no/approve typed in the MAIN timeline (not inside a proposal's thread) is
@@ -157,15 +162,103 @@ async def main():
store.put(root, proposal)
await say(room_id, "I didn't catch that.\n\n" + proposals.render_disambiguation(proposal), root)
async def handle_email_reply(room_id, root, text):
"""An in-thread reply to a CRM-drafted email-proposal card: yes commits, no dismisses, and
anything else is a natural-language revision of the note (re-drafted by local Qwen; the
human still approves the revised note, so the draft→approve gate holds)."""
item = email_threads.get(root)
if item is None:
return # a threaded reply we don't own (or already resolved)
decision = email_proposals.interpret(text)
if decision == "approve":
# Claim before the await (double-approve guard, like the intake commit path).
email_threads.pop(root, None)
try:
await asyncio.to_thread(crm_client.decide_email_proposal, item["id"], "approve", item.get("note"))
except Exception as exc:
email_threads[root] = item # restore for retry
await say(room_id, f"⚠️ couldn't add it ({str(exc)[:200]}). Reply **yes** to retry, **no** to dismiss.", root)
return
await say(room_id, f"✅ Added to the grid for **{item.get('investor_name') or 'the investor'}**.", root)
elif decision == "reject":
email_threads.pop(root, None)
try:
await asyncio.to_thread(crm_client.decide_email_proposal, item["id"], "dismiss")
except Exception as exc:
email_threads[root] = item
await say(room_id, f"⚠️ couldn't dismiss it ({str(exc)[:200]}). Try again.", root)
return
await say(room_id, "🗑️ Dismissed — nothing added to the grid.", root)
else:
try:
new_note = await asyncio.to_thread(email_proposals.revise_note, item.get("note") or "", text)
except Exception as exc:
await say(room_id, f"⚠️ couldn't revise that ({str(exc)[:200]}). Reply **yes** to add as-is, "
"**no** to dismiss, or rephrase.", root)
return
if not new_note:
await say(room_id, "I didn't catch a change. Reply **yes** to add the note as-is, **no** to "
"dismiss, or tell me how to change it.", root)
return
item["note"] = new_note
email_threads[root] = item
await say(room_id, f"✏️ Updated draft note:\n\n{new_note}\n\nReply **yes** to add it, **no** to "
"dismiss, or refine again.", root)
async def poll_email_proposals():
"""Poll the CRM for email-activity proposals: post a review card for each new one, rebuild
the reply-routing map from already-posted threads (so replies still route after a restart),
and announce+close any decided on the web. One failing cycle logs and retries next tick."""
while True:
try:
lists = await asyncio.to_thread(crm_client.list_email_proposals)
for it in lists["open"]: # rebuild routing for threads posted before (e.g. a restart)
ev = it.get("event_id")
if ev and ev not in email_threads:
email_threads[ev] = {"id": it["id"], "investor_name": it.get("investor_name"),
"note": it.get("proposed_note") or ""}
for it in lists["to_post"]:
try:
resp = await client.room_send(
review_room, "m.room.message",
matrix_io.thread_content(email_proposals.render_card(it), None))
ev = getattr(resp, "event_id", None)
if not ev:
print(f"matrix-intake: card send returned no event_id for {it['id']}", flush=True)
continue
await asyncio.to_thread(crm_client.mark_email_proposal_posted, it["id"], ev)
email_threads[ev] = {"id": it["id"], "investor_name": it.get("investor_name"),
"note": it.get("proposed_note") or ""}
except Exception as exc:
print(f"matrix-intake: failed to post email proposal {it.get('id')}: {exc}", flush=True)
for it in lists["to_close"]: # decided on the web → announce in-thread, then close
ev = it.get("event_id")
if not ev:
continue
try:
await say(review_room, email_proposals.closure_line(it.get("status")), ev)
await asyncio.to_thread(crm_client.mark_email_proposal_closed, it["id"])
email_threads.pop(ev, None)
except Exception as exc:
print(f"matrix-intake: failed to close email proposal {it.get('id')}: {exc}", flush=True)
except Exception as exc:
print(f"matrix-intake: email-proposal poll error: {exc}", flush=True)
await asyncio.sleep(EMAIL_POLL_SEC)
async def on_message(room: MatrixRoom, event: RoomMessageText):
if event.sender == mx["user_id"]:
return # never react to our own messages (we post in-thread — this prevents loops)
if room.room_id != intake_room:
return
text = (event.body or "").strip()
if not text:
return
root = matrix_io.thread_root_of(event)
# Email-proposal review room: only a threaded reply to a card we posted is actionable.
if review_room and room.room_id == review_room:
if root and root in email_threads:
await handle_email_reply(room.room_id, root, text)
return
if room.room_id != intake_room:
return
if root and store.has(root):
await handle_reply(room.room_id, root, text)
elif root:
@@ -180,8 +273,12 @@ async def main():
client.add_event_callback(on_message, RoomMessageText)
who = await client.whoami()
print(f"matrix-intake: listening as {who.user_id} in room {intake_room}", flush=True)
tasks = [asyncio.create_task(client.sync_forever(timeout=30000))]
if review_room:
tasks.append(asyncio.create_task(poll_email_proposals()))
print(f"matrix-intake: reviewing email proposals in room {review_room} (every {EMAIL_POLL_SEC}s)", flush=True)
try:
await client.sync_forever(timeout=30000)
await asyncio.gather(*tasks)
finally:
await client.close()
+41
View File
@@ -98,6 +98,47 @@ def match(proposal):
return {"match": match_out, "candidates": candidates}
def list_email_proposals():
"""Pull the email-activity review work-lists for the poll loop: {to_post, open, to_close}.
to_post = pending, un-posted (post a card); open = posted, awaiting a decision (rebuild the
reply-routing map after a restart); to_close = decided on the web (announce in-thread + close)."""
status, data = _authed("GET", "/api/intake/email-proposals")
if status != 200:
raise RuntimeError(f"email-proposals list failed ({status}): {data.get('error') or data}")
payload = data.get("data") or {}
return {k: (payload.get(k) or []) for k in ("to_post", "open", "to_close")}
def mark_email_proposal_posted(proposal_id, event_id):
"""Record the Matrix thread-root event id so the proposal's review state survives a restart."""
status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/matrix",
{"event_id": event_id})
if status != 200:
raise RuntimeError(f"mark posted failed ({status}): {data.get('error') or data}")
return data.get("data") or {}
def mark_email_proposal_closed(proposal_id):
"""Mark the review thread resolved after announcing a web-side decision in it."""
status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/matrix",
{"closed": True})
if status != 200:
raise RuntimeError(f"mark closed failed ({status}): {data.get('error') or data}")
return data.get("data") or {}
def decide_email_proposal(proposal_id, decision, note=None):
"""Relay an in-thread approve/dismiss (with the possibly-revised note) to the CRM. The server
appends the note to the grid on approve, tags source='matrix', and closes the thread."""
body = {"decision": decision}
if note is not None:
body["note"] = note
status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/decide", body)
if status not in (200, 201):
raise RuntimeError(f"email-proposal decide failed ({status}): {data.get('error') or data}")
return data.get("data") or {}
def build_commit_payload(proposal):
"""Pure: map a proposal to the /api/fundraising/log-communication request body.
+85
View File
@@ -0,0 +1,85 @@
"""Email-activity proposal review over Matrix — the CRM→Matrix leg of the email-capture flow.
The CRM (on the box) drafts a proposed grid note per newly-matched email (local model, no Claude)
and queues it for human review. The CRM is stdlib-only and can't post to Matrix itself, so this
bot PULLS the pending proposals (crm_client.list_email_proposals), posts a review card to the
dedicated review room, and relays the human's in-thread reply back to the CRM. Same draft→approve
discipline as the intake bot: nothing is appended to the grid until a human approves — here OR on
the web Email Capture panel, the two surfaces kept in sync via the CRM's email_proposal_matrix row.
This module is the PURE logic (card rendering, reply grammar, note revision) so it's unit-tested
offline; the async poll/post/reply wiring lives in bot.py (network + Matrix, live-smoke only).
"""
import spark
_YES = {"yes", "y", "approve", "approved", "ok", "confirm", "go", "add", "👍", ""}
_NO = {"no", "n", "cancel", "discard", "reject", "skip", "stop", "👎", ""}
_SNIPPET_MAX = 400 # email snippet shown on the card; the full body is in the web popup
def _truncate(s, n):
s = (s or "").strip()
return s if len(s) <= n else s[:n].rstrip() + ""
def render_card(item):
"""The review card posted to the Matrix review room: who/when + a short email snippet + the
drafted note. Deliberately compact for mobile — the full scrollable body is in the web Email
Capture popup (this is the metadata+snippet+note choice)."""
name = item.get("investor_name") or "Unknown investor"
direction = "Sent" if item.get("direction") == "sent" else "Received"
frm = item.get("from_name") or item.get("from_email") or "?"
lines = [f"📧 Proposed **grid note** for **{name}** ({direction})"]
if item.get("email_subject"):
lines.append(f"· Subject: {item['email_subject']}")
if item.get("email_date"):
lines.append(f"· Date: {item['email_date']}")
lines.append(f"· From: {frm}")
snippet = _truncate(item.get("snippet"), _SNIPPET_MAX)
if snippet:
lines.append(f"· Email: {snippet}")
lines.append("")
lines.append(f"📝 Draft note: {item.get('proposed_note') or '(empty)'}")
lines.append("")
lines.append("Reply **yes** to add it to the grid, **no** to dismiss, or just tell me how to "
"change the note (e.g. *say we discussed the Q3 raise*).")
return "\n".join(lines)
def closure_line(status):
"""Posted in-thread when a proposal was decided on the WEB while its Matrix thread was open."""
verb = "approved ✅ and added to the grid" if status == "approved" else "dismissed 🗑️"
return f"This was {verb} on the web — nothing more to do here. Thread closed."
def interpret(text):
"""Classify an in-thread reply: 'approve' | 'reject' | 'revise' (anything else → revise the note)."""
t = (text or "").strip().lower()
if t in _YES:
return "approve"
if t in _NO:
return "reject"
return "revise"
REVISE_SYSTEM = (
"You revise a single CRM note from a short instruction a venture-fund team member typed. "
"You are given the CURRENT note and an INSTRUCTION. Apply the instruction and reply with "
"ONLY a JSON object of the form {\"note\": \"<the full revised note>\"}. Keep it to one or two "
"factual sentences, no preamble. Output JSON only."
)
def revise_note(note, instruction, parse_fn=spark.parse_json):
"""Re-draft the note via local Qwen from a free-form instruction (no Claude, no scrub — same
local-only basis as the intake parse). Returns the new note text, or None if the model gave
nothing usable / unchanged, in which case the caller re-prompts. `parse_fn` is injectable for
tests."""
prompt = "CURRENT:\n" + (note or "") + "\n\nINSTRUCTION:\n" + (instruction or "").strip()
raw = parse_fn(prompt, system=REVISE_SYSTEM, max_tokens=400) or {}
new = raw.get("note") if isinstance(raw, dict) else None
new = (new or "").strip()
if not new or new == (note or "").strip():
return None
return new
+7
View File
@@ -61,3 +61,10 @@ def crm_settings():
# unset/empty just means no roster framing, i.e. the prior behavior.
def team_roster():
return [n.strip() for n in os.environ.get("INTAKE_TEAM_ROSTER", "").split(",") if n.strip()]
# Dedicated room for reviewing CRM-drafted email-activity proposals (the CRM→Matrix push leg).
# Separate from the intake room so high-volume email proposals don't drown the conversational
# intake flow. Unset/empty disables the whole email-review poll loop (the bot just does intake).
def email_review_room():
return os.environ.get("MATRIX_EMAIL_REVIEW_ROOM", "").strip()
@@ -0,0 +1,72 @@
"""Offline tests for the email-proposal review logic (card render, reply grammar, note revision).
The network/Matrix wiring lives in bot.py (live-smoke only); this covers the pure functions."""
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import email_proposals # noqa: E402
ITEM = {
"id": "p1", "investor_name": "Acme Capital", "direction": "received",
"from_name": "Jane Doe", "from_email": "jane@acme.com",
"email_subject": "Re: Fund III", "email_date": "2026-06-02",
"snippet": "thanks for the deck — one question on terms", "proposed_note": "✉ Received: asked about terms",
}
def test_interpret_yes_no_else():
assert email_proposals.interpret("yes") == "approve"
assert email_proposals.interpret(" Y ") == "approve"
assert email_proposals.interpret("") == "approve"
assert email_proposals.interpret("no") == "reject"
assert email_proposals.interpret("skip") == "reject"
# anything that isn't a clear yes/no is treated as a revision instruction
assert email_proposals.interpret("say we discussed the Q3 raise") == "revise"
def test_render_card_has_context_note_and_actions():
card = email_proposals.render_card(ITEM)
assert "Acme Capital" in card and "Received" in card
assert "Jane Doe" in card
assert "Re: Fund III" in card and "2026-06-02" in card
assert "thanks for the deck" in card
assert "✉ Received: asked about terms" in card
assert "yes" in card.lower() and "no" in card.lower()
def test_render_card_sent_direction():
assert "(Sent)" in email_proposals.render_card(dict(ITEM, direction="sent"))
def test_render_card_truncates_long_snippet():
card = email_proposals.render_card(dict(ITEM, snippet="x" * 1000))
assert "" in card and len(card) < 1000
def test_revise_note_applies_model_output():
out = email_proposals.revise_note(
"old note", "make it about the Q3 raise",
parse_fn=lambda prompt, system=None, max_tokens=400: {"note": "Discussed the Q3 raise."})
assert out == "Discussed the Q3 raise."
def test_revise_note_noop_or_empty_returns_none():
# model echoes the same note unchanged -> None so the caller re-prompts (not "Updated")
assert email_proposals.revise_note("same", "x", parse_fn=lambda *a, **k: {"note": "same"}) is None
# model returns nothing usable -> None
assert email_proposals.revise_note("n", "y", parse_fn=lambda *a, **k: {}) is None
assert email_proposals.revise_note("n", "y", parse_fn=lambda *a, **k: None) is None
def test_closure_line_reflects_status():
assert "approved" in email_proposals.closure_line("approved").lower()
assert "dismiss" in email_proposals.closure_line("dismissed").lower()
if __name__ == "__main__":
fns = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
for fn in fns:
fn()
print(f"ok {fn.__name__}")
print(f"\n{len(fns)} passed")
+153 -7
View File
@@ -619,6 +619,14 @@ def parse_iso_utc(ts):
def require_admin(user):
return bool(user and user.get('role') == 'admin')
def require_bot_or_admin(user):
"""Gate for agent/bot-facing endpoints: a dedicated 'bot' service account OR an admin
(admins keep parity for debugging/curl). The 'bot' role is authenticated-but-not-admin
it never passes require_admin, so a bot credential cannot reach user-management, security,
or settings. Reach (which endpoints) is controlled here; autonomy (acting without a human)
stays governed by the per-feature draft->approve gate, independent of role."""
return bool(user and user.get('role') in ('admin', 'bot'))
def log_audit(conn, user_id, entity_type, entity_id, action, changes=None):
conn.execute(
"INSERT INTO audit_log (id, user_id, entity_type, entity_id, action, changes) VALUES (?, ?, ?, ?, ?, ?)",
@@ -2068,6 +2076,9 @@ class CRMHandler(BaseHTTPRequestHandler):
# Matrix intake bot — new-vs-existing lookup for its in-thread proposal
if path == '/api/intake/match':
return self.handle_intake_match(user, params)
# Matrix review bot — email-activity proposal work-lists (to_post/open/to_close)
if path == '/api/intake/email-proposals':
return self.handle_list_bot_email_proposals(user)
# Users
if path == '/api/users':
@@ -2187,6 +2198,10 @@ class CRMHandler(BaseHTTPRequestHandler):
return self.handle_decide_activity_proposal(user, path.split('/')[-2], 'approve', body)
if re.match(r'^/api/activity/proposals/[^/]+/dismiss$', path):
return self.handle_decide_activity_proposal(user, path.split('/')[-2], 'dismiss', body)
if re.match(r'^/api/intake/email-proposals/[^/]+/matrix$', path):
return self.handle_mark_email_proposal_matrix(user, path.split('/')[-2], body)
if re.match(r'^/api/intake/email-proposals/[^/]+/decide$', path):
return self.handle_decide_email_proposal_matrix(user, path.split('/')[-2], body)
if re.match(r'^/api/thesis/nodes/[^/]+/choose$', path):
return self.handle_choose_variant(user, path.split('/')[-2])
if re.match(r'^/api/thesis/lines/[^/]+/approve$', path):
@@ -3964,6 +3979,59 @@ class CRMHandler(BaseHTTPRequestHandler):
return self.send_error_json(res["error"], code)
return self.send_json({"data": res})
# ─── Matrix review-bot bridge for email-activity proposals (bot-or-admin) ───
def handle_list_bot_email_proposals(self, user):
"""The bot's poll endpoint: {to_post, open, to_close}. Bot-or-admin (the proposals
carry LP email content, so this stays off the member tier)."""
if not require_bot_or_admin(user):
return self.send_error_json("Bot or admin required", 403)
conn = get_db()
try:
return self.send_json({"data": list_bot_email_proposals(conn)})
finally:
conn.close()
def handle_mark_email_proposal_matrix(self, user, proposal_id, body):
"""Record Matrix thread state: {event_id} marks the card posted; {closed:true} marks
the thread resolved after the bot announced a web-side decision."""
if not require_bot_or_admin(user):
return self.send_error_json("Bot or admin required", 403)
body = body or {}
conn = get_db()
try:
if body.get("closed"):
res = mark_proposal_matrix_closed(conn, proposal_id)
else:
event_id = str(body.get("event_id") or "").strip()
if not event_id:
return self.send_error_json("event_id or closed is required")
res = mark_proposal_matrix_posted(conn, proposal_id, event_id)
finally:
conn.close()
if res.get("error"):
return self.send_error_json(res["error"], 404 if res["error"] == "not_found" else 400)
return self.send_json({"data": res})
def handle_decide_email_proposal_matrix(self, user, proposal_id, body):
"""In-thread Matrix decision relayed by the bot: approve/dismiss (+ optional edited note),
tagged source='matrix' and closing the thread in the same transaction."""
if not require_bot_or_admin(user):
return self.send_error_json("Bot or admin required", 403)
body = body or {}
decision = str(body.get("decision") or "").strip()
if decision not in ("approve", "dismiss"):
return self.send_error_json("decision must be approve or dismiss")
conn = get_db()
try:
res = decide_email_activity_proposal(conn, proposal_id, decision, user['user_id'],
body.get('note'), source="matrix", close_matrix=True)
finally:
conn.close()
if res.get("error"):
code = {"not_found": 404, "already_decided": 409}.get(res["error"], 400)
return self.send_error_json(res["error"], code)
return self.send_json({"data": res})
# ─── UI-triggered index jobs + entity-merge review (Phase 1) ───
def handle_index_job(self, user, kind):
if not require_admin(user):
@@ -4357,8 +4425,10 @@ class CRMHandler(BaseHTTPRequestHandler):
return self.send_error_json("password must be at least 8 characters")
role = body.get('role', 'member')
if role not in ('admin', 'member'):
return self.send_error_json("role must be admin or member")
# 'bot' is a deliberately-provisioned agent service account (kept out of the invite UI's
# member/admin dropdown) — authenticated but never an admin. See require_bot_or_admin.
if role not in ('admin', 'member', 'bot'):
return self.send_error_json("role must be admin, member, or bot")
conn = get_db()
existing = conn.execute(
@@ -4417,9 +4487,9 @@ class CRMHandler(BaseHTTPRequestHandler):
if 'role' in body:
role = str(body.get('role'))
if role not in ('admin', 'member'):
if role not in ('admin', 'member', 'bot'):
conn.close()
return self.send_error_json("role must be admin or member")
return self.send_error_json("role must be admin, member, or bot")
sets.append("role = ?")
args.append(role)
@@ -5728,8 +5798,14 @@ def list_email_activity_proposals(conn, status="pending", limit=200):
return []
def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_note=None):
"""Approve (optionally with an edited note -> append to grid) or dismiss a proposal."""
def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_note=None,
source="crm_ui", close_matrix=False):
"""Approve (optionally with an edited note -> append to grid) or dismiss a proposal.
`source` records the channel in the audit log ('crm_ui' for the web panel, 'matrix' for an
in-thread approval relayed by the review bot). `close_matrix` is set by the bot when the
decision was made in-thread: it stamps the Matrix side row closed in the same transaction so
the web->Matrix close path (list_bot_email_proposals.to_close) won't re-announce it."""
p = conn.execute("SELECT * FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone()
if not p:
return {"error": "not_found"}
@@ -5747,15 +5823,85 @@ def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_
action, result = "email.activity_dismissed", {"status": "dismissed"}
else:
return {"error": "bad_decision"}
if close_matrix:
_mark_proposal_matrix(conn, proposal_id, closed_at=now())
conn.execute(
"INSERT INTO interaction_log (id, ts, actor_type, actor_id, action, target_type, target_id, payload, source, created_at) "
"VALUES (?,?,?,?,?,?,?,?,?,?)",
(generate_id(), now(), "human", user_id, action, "fundraising_investor", p["investor_id"],
json.dumps({"proposal_id": proposal_id}), "crm_ui", now()))
json.dumps({"proposal_id": proposal_id}), source, now()))
conn.commit()
return result
# ─── Matrix review-bot bridge for email-activity proposals (Features 2/3) ──────
# The CRM (box, stdlib, no matrix-nio) can't post to Matrix, so the intake bot (Spark) PULLS
# pending proposals via list_bot_email_proposals, posts a review card to the dedicated review
# room, and writes the thread-root event_id back here. State lives CRM-side (email_proposal_matrix)
# so both surfaces stay in sync and it survives a bot restart. All queries degrade to empty when
# the email integration tables are absent (OperationalError), mirroring list_email_activity_proposals.
_BOT_PROPOSAL_COLS = (
"SELECT p.id, p.investor_name, p.direction, p.summary, p.proposed_note, "
"p.email_subject, p.email_date, e.from_name, e.from_email, e.snippet, "
"m.event_id AS event_id, p.status AS status "
"FROM email_activity_proposals p "
"LEFT JOIN email_proposal_matrix m ON m.proposal_id = p.id "
"LEFT JOIN emails e ON e.id = p.email_id ")
def list_bot_email_proposals(conn, limit=100):
"""The three work-lists the Matrix review bot polls:
to_post pending, not yet posted to Matrix -> bot posts a review card.
open pending, posted, not closed -> live threads; the bot rebuilds its
event_id->proposal routing map from these after a restart.
to_close decided on the WEB while a thread is open -> bot announces it in-thread, closes.
Each item carries the card content (investor, direction, subject, date, from, snippet, note)."""
try:
to_post = [dict(r) for r in conn.execute(
_BOT_PROPOSAL_COLS + "WHERE p.status='pending' AND (m.proposal_id IS NULL OR m.posted_at IS NULL) "
"ORDER BY p.email_date ASC, p.created_at ASC LIMIT ?", (limit,))]
open_threads = [dict(r) for r in conn.execute(
_BOT_PROPOSAL_COLS + "WHERE p.status='pending' AND m.posted_at IS NOT NULL AND m.closed_at IS NULL "
"ORDER BY p.email_date ASC, p.created_at ASC LIMIT ?", (limit,))]
to_close = [dict(r) for r in conn.execute(
_BOT_PROPOSAL_COLS + "WHERE p.status!='pending' AND m.posted_at IS NOT NULL AND m.closed_at IS NULL "
"ORDER BY p.decided_at ASC LIMIT ?", (limit,))]
except sqlite3.OperationalError:
return {"to_post": [], "open": [], "to_close": []}
return {"to_post": to_post, "open": open_threads, "to_close": to_close}
def _mark_proposal_matrix(conn, proposal_id, *, event_id=None, posted_at=None, closed_at=None):
"""Idempotent upsert of the 1:1 Matrix side row. Only the passed fields are written."""
cols, vals, sets = ["proposal_id"], [proposal_id], []
for name, val in (("event_id", event_id), ("posted_at", posted_at), ("closed_at", closed_at)):
if val is not None:
cols.append(name); vals.append(val); sets.append(f"{name}=excluded.{name}")
placeholders = ",".join("?" for _ in cols)
sql = f"INSERT INTO email_proposal_matrix ({','.join(cols)}) VALUES ({placeholders})"
if sets:
sql += " ON CONFLICT(proposal_id) DO UPDATE SET " + ",".join(sets)
conn.execute(sql, vals)
def mark_proposal_matrix_posted(conn, proposal_id, event_id):
"""Record that the bot posted a review card (thread root = event_id)."""
if not conn.execute("SELECT 1 FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone():
return {"error": "not_found"}
_mark_proposal_matrix(conn, proposal_id, event_id=event_id, posted_at=now())
conn.commit()
return {"ok": True}
def mark_proposal_matrix_closed(conn, proposal_id):
"""Mark the Matrix review thread resolved (the bot announced a web-side decision)."""
if not conn.execute("SELECT 1 FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone():
return {"error": "not_found"}
_mark_proposal_matrix(conn, proposal_id, closed_at=now())
conn.commit()
return {"ok": True}
# ─── Main Entry Point ────────────────────────────────────────────────────────
def main():
+129
View File
@@ -0,0 +1,129 @@
#!/usr/bin/env python3
"""Test the Matrix review-bot bridge for email-activity proposals (Features 2/3):
the bot work-lists (to_post / open / to_close), the Matrix side-row mark helpers, and an
in-thread (source='matrix') decision that closes the thread — plus the bot-or-admin role gate.
Synthetic data only (guardrail #9). The local model is stubbed.
Run: cd backend && python3 test_email_proposal_matrix.py
"""
import json
import os
import sqlite3
import sys
import tempfile
os.environ["CRM_DB_PATH"] = os.path.join(tempfile.mkdtemp(), "crm.db")
os.environ.setdefault("CRM_DATA_DIR", os.path.dirname(os.environ["CRM_DB_PATH"]))
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import server # noqa: E402
server._summarize_email_gist = lambda subject, body: "fundraising update; proposed a call"
FAILS = []
def check(cond, msg):
print((" PASS " if cond else " FAIL ") + msg)
if not cond:
FAILS.append(msg)
def setup():
conn = sqlite3.connect(os.environ["CRM_DB_PATH"])
conn.row_factory = sqlite3.Row
conn.executescript("""
CREATE TABLE app_settings (key TEXT PRIMARY KEY, value_json TEXT, updated_at TEXT);
CREATE TABLE email_accounts (id TEXT, email_address TEXT, sync_enabled INT DEFAULT 1, sync_status TEXT, backfill_complete INT);
CREATE TABLE emails (id TEXT PRIMARY KEY, subject TEXT, body_text TEXT, snippet TEXT, from_name TEXT, from_email TEXT, sent_at TEXT, is_matched INT, match_status TEXT);
CREATE TABLE email_investor_links (id TEXT, email_id TEXT, fundraising_investor_id TEXT, organization_id TEXT, contact_id TEXT, match_confidence REAL);
CREATE TABLE email_activity_proposals (id TEXT PRIMARY KEY, email_id TEXT UNIQUE, investor_id TEXT, investor_name TEXT,
direction TEXT, summary TEXT, proposed_note TEXT, email_subject TEXT, email_date TEXT, status TEXT DEFAULT 'pending',
decided_by TEXT, decided_at TEXT, final_note TEXT, created_at TEXT);
CREATE TABLE email_proposal_matrix (proposal_id TEXT PRIMARY KEY, event_id TEXT, posted_at TEXT, closed_at TEXT, created_at TEXT);
CREATE TABLE users (id TEXT PRIMARY KEY, username TEXT);
CREATE TABLE fundraising_investors (id TEXT PRIMARY KEY, investor_name TEXT, notes TEXT);
CREATE TABLE fundraising_state (id TEXT PRIMARY KEY, grid_json TEXT, views_json TEXT, version INT,
updated_by TEXT REFERENCES users(id), updated_at TEXT);
CREATE TABLE interaction_log (id TEXT PRIMARY KEY, ts TEXT, actor_type TEXT, actor_id TEXT, action TEXT, target_type TEXT, target_id TEXT, payload TEXT, source TEXT, created_at TEXT);
""")
conn.execute("INSERT INTO users (id,username) VALUES ('user-1','grant')")
conn.execute("INSERT INTO app_settings VALUES ('email_activity_since', ?, ?)", (json.dumps("2026-01-01T00:00:00"), "x"))
conn.execute("INSERT INTO email_accounts (id,email_address) VALUES ('a','grant@ten31.xyz')")
conn.execute("INSERT INTO fundraising_investors (id,investor_name,notes) VALUES ('inv1','Harbor & Vine','existing note')")
grid = {"columns": [], "rows": [{"id": "inv1", "investor_name": "Harbor & Vine", "notes": "existing note"}]}
conn.execute("INSERT INTO fundraising_state (id,grid_json,views_json,version) VALUES ('main',?,?,1)", (json.dumps(grid), "[]"))
conn.executemany("INSERT INTO emails (id,subject,body_text,snippet,from_name,from_email,sent_at,is_matched,match_status) VALUES (?,?,?,?,?,?,?,1,'matched')", [
("e1", "Fund III", "Here is the update", "the quarterly update is attached", "Grant", "grant@ten31.xyz", "2026-06-01T10:00:00"),
("e2", "Re: Fund III", "Thanks, a question", "thanks — one question on terms", "LP Contact", "lp@harborvine.example", "2026-06-02T10:00:00"),
])
conn.executemany("INSERT INTO email_investor_links (id,email_id,fundraising_investor_id,match_confidence) VALUES (?,?, 'inv1', 1.0)",
[("l1", "e1"), ("l2", "e2")])
conn.commit()
conn.close()
def main():
setup()
# role gate: bot passes the agent gate but is NOT an admin; member passes neither.
check(server.require_bot_or_admin({"role": "bot"}), "bot passes require_bot_or_admin")
check(server.require_bot_or_admin({"role": "admin"}), "admin passes require_bot_or_admin")
check(not server.require_bot_or_admin({"role": "member"}), "member does NOT pass require_bot_or_admin")
check(not server.require_admin({"role": "bot"}), "bot is NOT an admin (no user-mgmt/settings reach)")
check(server.propose_email_activity_notes().get("proposed") == 2, "drafts 2 proposals")
conn = server.get_db()
props = server.list_email_activity_proposals(conn, status="pending")
by_email = {p["email_id"]: p for p in props}
p_a, p_b = by_email["e1"], by_email["e2"]
# Both are pending + un-posted → both in to_post; card carries from/snippet/note context.
lists = server.list_bot_email_proposals(conn)
check(len(lists["to_post"]) == 2 and not lists["open"] and not lists["to_close"], "both proposals queued to_post")
card = next(it for it in lists["to_post"] if it["id"] == p_a["id"])
check(card.get("from_name") == "Grant" and "quarterly update" in (card.get("snippet") or ""), "card carries from_name + snippet")
check("" in (card.get("proposed_note") or ""), "card carries the drafted note")
# Post p_a to Matrix → it leaves to_post and becomes an open thread (event id recorded).
server.mark_proposal_matrix_posted(conn, p_a["id"], "evtA")
lists = server.list_bot_email_proposals(conn)
check(len(lists["to_post"]) == 1 and lists["to_post"][0]["id"] == p_b["id"], "posting p_a leaves only p_b to_post")
check(len(lists["open"]) == 1 and lists["open"][0]["id"] == p_a["id"] and lists["open"][0]["event_id"] == "evtA",
"posted p_a is an open thread carrying its event id")
# Decide p_a IN-THREAD on Matrix (approve + close in one transaction).
r = server.decide_email_activity_proposal(conn, p_a["id"], "approve", "user-1", source="matrix", close_matrix=True)
check(r.get("status") == "approved" and r.get("placed_in_grid") is True, "matrix approve appends to the grid")
lists = server.list_bot_email_proposals(conn)
check(not any(it["id"] == p_a["id"] for it in lists["open"] + lists["to_close"]),
"matrix-decided proposal is closed (not re-announced via to_close)")
src = conn.execute("SELECT source FROM interaction_log WHERE action='email.activity_approved'").fetchone()["source"]
check(src == "matrix", "matrix decision is audited source='matrix'")
# Web-decide path: post p_b, then dismiss it on the WEB (default source, no close) → the bot
# must see it in to_close so it can announce the web decision in-thread, then close.
server.mark_proposal_matrix_posted(conn, p_b["id"], "evtB")
server.decide_email_activity_proposal(conn, p_b["id"], "dismiss", "user-1") # web path: source crm_ui, no close
lists = server.list_bot_email_proposals(conn)
check(len(lists["to_close"]) == 1 and lists["to_close"][0]["id"] == p_b["id"] and lists["to_close"][0]["status"] == "dismissed",
"web-decided open thread surfaces in to_close")
src2 = conn.execute("SELECT source FROM interaction_log WHERE action='email.activity_dismissed'").fetchone()["source"]
check(src2 == "crm_ui", "web decision is audited source='crm_ui'")
server.mark_proposal_matrix_closed(conn, p_b["id"])
lists = server.list_bot_email_proposals(conn)
check(not lists["to_close"] and not lists["open"], "closing the thread clears the work-lists")
# Marking a non-existent proposal is a clean not_found, not a crash.
check(server.mark_proposal_matrix_posted(conn, "nope", "evtX").get("error") == "not_found", "mark posted on unknown id -> not_found")
conn.close()
if FAILS:
print(f"\nFAILED ({len(FAILS)})")
for f in FAILS:
print(" - " + f)
sys.exit(1)
print("\nALL PASS (email-proposal Matrix bridge)")
if __name__ == "__main__":
main()
+1 -1
View File
@@ -12,7 +12,7 @@ Read this before editing Gmail capture or draft creation.
## What it does
- `backend/email_integration/` captures Gmail via **domain-wide delegation** (`credentials.py`, `matcher.py`, `parser.py`, `db.py`, `sync.py`, `scheduler.py`, `routes.py`) and creates Tier-B in-thread drafts (`compose.py`). It has its own `migrations/`.
- Captured email becomes CRM activity through a **propose → approve** flow — nothing lands on a contact record until a human approves the proposal.
- Captured email becomes CRM activity through a **propose → approve** flow — nothing lands on a contact record until a human approves the proposal. The proposed grid notes show on the **Email Capture** page (admin-only): each card has a **View email** toggle that fetches `GET /api/email/detail?id=` and shows the source email inline (from/to/cc/date/subject + scrollable body) so you can judge the note against it. The same proposals can also be reviewed/approved/edited from a **dedicated Matrix room**, kept in sync with this panel (decide on either surface; the other reflects it) — that CRM→Matrix bridge lives in the **review bot**, see `docs/guides/matrix-intake.md`. The proposal model itself (`email_activity_proposals` + the `propose_email_activity_notes` drafter + the decide path) lives in `backend/server.py`, not this package.
## Hard rule
+54 -1
View File
@@ -90,6 +90,42 @@ rows ≥ `min_score` (0.62), ranked, capped at 5:
only the shortlist, never the whole LP list — intentionally NOT built in this pass, because the
deterministic filter already surfaces every duplicate the human then resolves.
## Email-activity proposal review (the CRM→Matrix bridge, v0.1.0:89)
A second, separate flow runs alongside intake: reviewing the **proposed grid notes** the CRM
drafts from newly-matched email (`server.propose_email_activity_notes`, surfaced on the web Email
Capture panel). The bot lets the team approve/dismiss/edit those on mobile, kept **in sync** with
the web panel. The CRM (box, stdlib, no matrix-nio) can't post to Matrix, so the bot **pulls**.
- **Dedicated room** (`MATRIX_EMAIL_REVIEW_ROOM`, see *Config*) — separate from the intake room
so high-volume email proposals don't drown the conversational intake. Unset → the whole leg is
off (the bot just does intake). The bot must be a **member** of this room.
- **Poll loop** (`bot.poll_email_proposals`, every `EMAIL_POLL_SEC`=20s) calls `crm_client.
list_email_proposals` → `GET /api/intake/email-proposals`, which returns three work-lists:
- **to_post** — pending, not yet posted → the bot posts a review card (metadata + a short email
**snippet** + the drafted note; the full body is the web popup's job, kept compact for mobile),
then records the thread-root event id via `POST .../{id}/matrix {event_id}`.
- **open** — pending, posted, not closed → the bot rebuilds its `event_id → proposal` routing map
from these on **every poll**, so replies still route **after a bot restart** (unlike intake's
in-memory-only store — the state lives CRM-side in `email_proposal_matrix`).
- **to_close** — decided on the **web** while a thread was open → the bot posts a "decided on the
web — thread closed" line and `POST .../{id}/matrix {closed:true}`.
- **In-thread replies** (`bot.handle_email_reply`, `email_proposals.interpret`): `yes` →
`POST .../{id}/decide {decision:"approve", note}` (appends the note to the grid, source='matrix',
closes the thread atomically); `no` → dismiss; **anything else → NL revision of the note** via
local Qwen (`email_proposals.revise_note`, no Claude/scrub) — re-rendered for re-approval, so the
draft→approve gate holds. A no-op/empty revision re-prompts instead of saying "Updated".
- **Two surfaces, one source of truth.** Decide on the web → the bot announces + closes the thread;
decide on Matrix → the web panel polls `/api/activity/proposals` (~25s) and the card clears.
`email_proposal_matrix` (1:1 side row, migration `0003`) carries `event_id`/`posted_at`/`closed_at`;
a matrix decision sets `closed_at` in the same txn so it's never re-announced via `to_close`.
- **Pure logic is `email_proposals.py`** (card render, reply grammar, note revision) — unit-tested
offline in `test_email_proposals.py`; the async poll/post wiring is in `bot.py` (live-smoke only).
- **Known minors (low-likelihood, ~5-person team):** if the CRM is unreachable *between* posting a
card and recording its event id, the next poll re-posts a duplicate card (the orphan's replies
won't route — re-send/decide the recorded one). A mid-revise bot restart loses the in-memory
revised note (rebuilt from `open` = the original `proposed_note`; still a valid proposal).
## Rules / gotchas
- **Module-name collision:** the intake config module is `settings.py`, **not** `config.py`,
@@ -161,7 +197,11 @@ rows ≥ `min_score` (0.62), ranked, capped at 5:
- **Server-side endpoints ship in the s9pk, not the bot.** `GET /api/intake/match` and the
`source` provenance on `log-communication` live in `backend/server.py`, so they reach the box
only via an **s9pk build + install** — a bot restart won't deliver them. (Missed in v83: the
box 404'd `/api/intake/match` until **v0.1.0:84**.)
box 404'd `/api/intake/match` until **v0.1.0:84**.) **Same split for the email-review bridge
(v0.1.0:89):** the `/api/intake/email-proposals*` endpoints + the `email_proposal_matrix`
migration (`0003`) + the `bot` role ship in the **s9pk**; the poll loop + review-room handling
ship on the **Spark** (git pull + restart). A bot restart against a pre-v89 box returns nothing
useful (404/empty), so install the s9pk first, then set the bot user's role + the review room.
- **`CRM_API_BASE` is the box over the LAN, not localhost** (bot on the Spark, CRM on the box).
`https://immense-voyage.local` (443) is the **StartOS dashboard**, not the CRM — the CRM has
its own interface address (the URL you open in a browser); container port 8080 isn't
@@ -174,6 +214,19 @@ All in `.env` (names in `.env.example`): `MATRIX_HOMESERVER`, `MATRIX_USER`,
`CRM_BOT_USERNAME`, `CRM_BOT_PASSWORD`, `CRM_API_VERIFY_TLS`. Spark settings are inherited from
the ingest client (`SPARK_CONTROL_URL`, `CRM_CHAT_MODEL`).
- **`MATRIX_EMAIL_REVIEW_ROOM`** (optional) — the dedicated room for the email-activity proposal
review leg (above). Unset/empty disables that leg entirely (the bot does intake only). The bot
must be invited to + joined in this room. Read once at startup, like the room/roster.
- **Bot CRM user needs role `bot`.** The email-proposal endpoints (`/api/intake/email-proposals*`)
are gated to `require_bot_or_admin` because they expose LP email content (the proposals are
admin-only on the web). The `bot` role is **authenticated-but-not-admin** — it passes these
endpoints + the auth-only ones the bot already uses (login, `/api/intake/match`,
`log-communication`), but **never** `require_admin` (no user-management/settings/security reach).
One-time flip of the existing service account (kept out of the invite UI's member/admin dropdown
— provision deliberately): an admin `PATCH /api/users/<id> {"role":"bot"}`, or on the box
`UPDATE users SET role='bot' WHERE username='<CRM_BOT_USERNAME>';`. Role controls *reach*; the
draft→approve gate (a human still approves every write) controls *autonomy* — two separate axes.
- **`INTAKE_TEAM_ROSTER`** (optional, comma-separated) — Ten31 team-member names that frame the
parse (see *Flow* step 1). Use the **first names as actually typed in the room** ("Grant,
Jonathan, …"). Read once at startup by `settings.team_roster()`, so **a roster change needs a
+71
View File
@@ -10037,6 +10037,8 @@
const [proposals, setProposals] = useState([]);
const [edits, setEdits] = useState({});
const [deciding, setDeciding] = useState(null);
const [openEmail, setOpenEmail] = useState(null); // proposal id whose source email is expanded
const [emailCache, setEmailCache] = useState({}); // email_id -> {loading, data, error}
const load = useCallback(async () => {
let s;
@@ -10086,6 +10088,21 @@
return () => { cancelled = true; clearInterval(iv); };
}, [backfilling, load]);
// Steady-state poll of just the proposals so a decision made on Matrix (approve/dismiss
// in the review room) clears its card here without a manual reload — the mirror of the
// bot announcing a web-side decision in-thread. Admin-only (only admins see proposals).
const refreshProposals = useCallback(async () => {
try {
const pr = await api('/api/activity/proposals', {}, token);
setProposals(Array.isArray(pr?.proposals) ? pr.proposals : []);
} catch (_) { /* admin-only / transient — leave the current list */ }
}, [token]);
useEffect(() => {
if (!isAdmin) return undefined;
const iv = setInterval(() => { refreshProposals(); }, 25000);
return () => clearInterval(iv);
}, [isAdmin, refreshProposals]);
const runAction = async (key, endpoint, successMsg, confirmMsg, body) => {
if (busy) return;
if (confirmMsg && !window.confirm(confirmMsg)) return;
@@ -10120,6 +10137,51 @@
}
};
// Click a proposal to see the email it was drafted from (from/to/cc/date/subject +
// scrollable body) so you can judge whether the note is right. Lazily fetched +
// cached per email; reuses the admin-only /api/email/detail used by Communications.
const toggleEmail = async (p) => {
if (openEmail === p.id) { setOpenEmail(null); return; }
setOpenEmail(p.id);
const eid = p.email_id;
if (!eid || emailCache[eid]) return;
setEmailCache((c) => ({ ...c, [eid]: { loading: true } }));
try {
const res = await api(`/api/email/detail?id=${encodeURIComponent(eid)}`, {}, token);
setEmailCache((c) => ({ ...c, [eid]: { loading: false, data: res } }));
} catch (err) {
setEmailCache((c) => ({ ...c, [eid]: { loading: false, error: getErrorMessage(err, 'Failed to load email') } }));
}
};
const renderProposalEmail = (p) => {
const det = emailCache[p.email_id];
if (!det) return null;
if (det.loading) return <div style={{ marginTop: '8px' }}><SkeletonBlock lines={4} /></div>;
if (det.error) return <div className="toast error" style={{ position: 'static', marginTop: '8px' }}>{det.error}</div>;
const d = det.data || {};
const rcpt = (kind) => (d.recipients || []).filter((r) => r.kind === kind)
.map((r) => r.display_name ? `${r.display_name} <${r.address}>` : r.address).join(', ');
const to = rcpt('to'), cc = rcpt('cc');
const from = d.from_name ? `${d.from_name} <${d.from_email || ''}>` : (d.from_email || '—');
const lbl = { fontSize: '11px', color: '#8ea2b7' };
return (
<div style={{ marginTop: '8px', border: '1px solid #263548', borderRadius: '6px', background: '#0d1622', padding: '10px' }}>
<div style={lbl}><b>From:</b> {from}</div>
{to && <div style={lbl}><b>To:</b> {to}</div>}
{cc && <div style={lbl}><b>Cc:</b> {cc}</div>}
<div style={lbl}><b>Date:</b> {d.sent_at ? new Date(d.sent_at).toLocaleString() : '—'}</div>
<div style={lbl}><b>Subject:</b> {d.subject || '(no subject)'}</div>
{(d.attachments || []).length > 0 && (
<div style={{ ...lbl, marginTop: '2px' }}><b>Attachments:</b> {d.attachments.map((a) => a.filename).join(', ')}</div>
)}
<pre style={{ margin: '8px 0 0', maxHeight: '280px', overflowY: 'auto', whiteSpace: 'pre-wrap', wordBreak: 'break-word', fontSize: '12px', lineHeight: 1.5, color: '#cdd9e5', fontFamily: 'inherit' }}>
{d.body_text || (d.has_html ? '(HTML-only email — open in Gmail to view formatting)' : '(no body captured)')}
</pre>
</div>
);
};
if (loading) return <div style={{ padding: '20px' }}><SkeletonBlock lines={8} /></div>;
if (error) return <div className="toast error" style={{ position: 'static' }}>{error}</div>;
if (!status) return <div className="empty-state">No data</div>;
@@ -10204,7 +10266,16 @@
{deciding === p.id ? 'Adding…' : 'Approve & add to grid'}
</button>
<button onClick={() => decide(p, 'dismiss')} disabled={deciding === p.id}>Dismiss</button>
{p.email_id && (
<button
onClick={() => toggleEmail(p)}
style={{ background: 'transparent', border: '1px solid #263548', color: '#8ea2b7' }}
>
{openEmail === p.id ? 'Hide email' : 'View email'}
</button>
)}
</div>
{openEmail === p.id && renderProposalEmail(p)}
</div>
))}
</div>
+3 -2
View File
@@ -53,8 +53,9 @@ export const PACKAGE_TITLE = 'Ten31 Database'
// * 0.1.0:85 (cosmetic: drop the redundant "[note]" tag from the fundraising-grid note line — now "YYYY-MM-DD Contact: summary"; informative comm types [call, meeting, …] keep their "[type]" tag; shared by the Matrix intake bot + grid-UI logging; no schema change)
// * 0.1.0:86 (Matrix intake fuzzy matching: GET /api/intake/match now returns ranked `candidates` [fuzzy near-matches — deterministic difflib name similarity + token overlap + email edit-distance ≤ 2, legal-suffix-aware] alongside the exact `match`, so the bot can surface near-duplicates ["Charlie"/"Charles", "Acme Capital"/"Acme Capital LLC", a one-char email typo] for human confirmation instead of silently creating a second investor; the bot-side disambiguation + conversational-edit UX ships on the Spark, not the s9pk; code-only, no schema change)
// * 0.1.0:87 (Adopt the Pipeline — grid drives the deal board: new "Add to Pipeline" row action creates+links an opportunity via opportunities.fundraising_investor_id [migration 0005, additive], reusing the grid's synced contact [no POST /api/contacts side-door] and mapping the grid lead→owner; idempotent [one live opp/investor, re-link never reseeds board-owned stage/probability]; read-only Pipeline + Pipeline Stage grid columns derived live from the linked opp; "Remove from Pipeline" soft-deletes the opp [grid row untouched]; deleting a grid investor archives its orphaned opp; folds in the soft-delete fix for the pipeline report + dashboard aggregates [archived opps no longer counted])
// * Current: 0.1.0:88 (frontend-only: retire the Pipeline page's "+ New Opportunity" button + its create-by-contact modal — opportunities are now born only from a fundraising-grid investor row ["+ Pipeline"], so the board is a view + stage-management surface; replaced the button with a muted "Add deals from the Fundraising Grid" hint; removed the now-dead handler/state + the page's unused /api/contacts fetch)
export const PACKAGE_VERSION = '0.1.0:88'
// * 0.1.0:88 (frontend-only: retire the Pipeline page's "+ New Opportunity" button + its create-by-contact modal — opportunities are now born only from a fundraising-grid investor row ["+ Pipeline"], so the board is a view + stage-management surface; replaced the button with a muted "Add deals from the Fundraising Grid" hint; removed the now-dead handler/state + the page's unused /api/contacts fetch)
// * Current: 0.1.0:89 (email-proposal review over Matrix + a dedicated agent role: Email Capture's proposed grid notes gain a click-to-view inline popup of the source email [from/to/cc/date/subject/scrollable body, via the existing GET /api/email/detail]; and a CRM→Matrix review bridge — the intake bot [Spark] pulls pending proposals, posts a review card to a dedicated review room [MATRIX_EMAIL_REVIEW_ROOM], and relays in-thread yes/no/NL-edit back to the CRM, with web panel ↔ Matrix kept in sync [decide on either surface; the other reflects it]. New side table email_proposal_matrix [email-integration migration 0003, additive + idempotent] holds per-proposal Matrix thread state; new bot-or-admin endpoints GET /api/intake/email-proposals + .../{id}/matrix + .../{id}/decide, gated by a new 'bot' role [authenticated, never admin]. Bot poll loop + review-room handling ship on the Spark, not the s9pk)
export const PACKAGE_VERSION = '0.1.0:89'
export const DATA_MOUNT_PATH = '/data'
export const WEB_PORT = 8080
+3 -2
View File
@@ -49,8 +49,9 @@ import { v_0_1_0_85 } from './v0.1.0.85'
import { v_0_1_0_86 } from './v0.1.0.86'
import { v_0_1_0_87 } from './v0.1.0.87'
import { v_0_1_0_88 } from './v0.1.0.88'
import { v_0_1_0_89 } from './v0.1.0.89'
export const versionGraph = VersionGraph.of({
current: v_0_1_0_88,
other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73, v_0_1_0_74, v_0_1_0_75, v_0_1_0_76, v_0_1_0_77, v_0_1_0_78, v_0_1_0_79, v_0_1_0_80, v_0_1_0_81, v_0_1_0_82, v_0_1_0_83, v_0_1_0_84, v_0_1_0_85, v_0_1_0_86, v_0_1_0_87],
current: v_0_1_0_89,
other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73, v_0_1_0_74, v_0_1_0_75, v_0_1_0_76, v_0_1_0_77, v_0_1_0_78, v_0_1_0_79, v_0_1_0_80, v_0_1_0_81, v_0_1_0_82, v_0_1_0_83, v_0_1_0_84, v_0_1_0_85, v_0_1_0_86, v_0_1_0_87, v_0_1_0_88],
})
+24
View File
@@ -0,0 +1,24 @@
import { VersionInfo } from '@start9labs/start-sdk'
// Email-proposal review over Matrix + a dedicated agent role. The CRM-drafted "proposed grid
// notes" (Email Capture panel) gain (1) a click-to-view inline popup of the source email
// (from/to/cc/date/subject/scrollable body, via the existing /api/email/detail) so a reviewer
// can judge the note against the email, and (2) a CRM→Matrix review bridge: the intake bot
// (Spark) pulls pending proposals, posts a review card to a dedicated review room, and relays
// the human's in-thread yes/no/edit back to the CRM — with the web panel and Matrix kept in
// sync (decide on either; the other surface reflects it). New side table email_proposal_matrix
// (email-integration migration 0003, additive + idempotent — CREATE TABLE IF NOT EXISTS) holds
// the per-proposal Matrix thread state. New bot-or-admin endpoints under /api/intake/
// email-proposals (list/mark/decide), gated by a new 'bot' role (authenticated, never admin).
// The bot's poll loop + review-room handling ship on the Spark (git pull + restart), not here.
export const v_0_1_0_89 = VersionInfo.of({
version: '0.1.0:89',
releaseNotes: {
en_US: [
'Email Capture: click a proposed grid note to see the source email inline',
'(from/to/cc/date/subject + body) before approving, and review/approve/dismiss/edit',
'proposals from a dedicated Matrix room on mobile — decisions sync both ways.',
].join(' '),
},
migrations: { up: async () => {}, down: async () => {} },
})