Email-proposal review over Matrix + a bot role (v0.1.0:89)

The email-capture "proposed grid notes" gain two review surfaces:

1. Inline source email — each proposed-note card on the Email Capture page
   gets a "View email" toggle that lazily fetches the existing
   GET /api/email/detail and shows from/to/cc/date/subject + scrollable body,
   so a reviewer can judge the note against the email it was drafted from.

2. CRM->Matrix review bridge — the CRM (box, stdlib, no matrix-nio) can't post
   to Matrix, so the intake bot (Spark) PULLS: GET /api/intake/email-proposals
   returns to_post/open/to_close work-lists; the bot posts a review card
   (metadata + snippet + draft note) to a dedicated review room
   (MATRIX_EMAIL_REVIEW_ROOM) and relays in-thread yes / no / NL-edit
   (POST .../{id}/decide, note revised via local Qwen). Decisions sync both
   ways: web decide -> bot announces + closes the thread; Matrix decide -> the
   web panel's ~25s poll clears the card. State lives CRM-side in the new
   email_proposal_matrix side row (email-integration migration 0003, additive
   + idempotent CREATE TABLE IF NOT EXISTS), so it survives a bot restart.

Adds a 'bot' role (authenticated, never admin; require_bot_or_admin) to gate
the email-proposal endpoints rather than handing the bot full admin — the
principled base for the coming agentic capabilities. Role controls reach;
the draft->approve gate still controls autonomy (a human approves every write).

Deploy split: endpoints + migration + role + frontend ship in the s9pk; the
bot poll loop + review-room handling ship on the Spark. The bot's CRM user
must be flipped member->bot and joined to the review room (one-time).

Tests: backend/test_email_proposal_matrix.py + matrix_intake/test_email_proposals.py
(30/30 suite green, render-smoke green, migration verified twice on a DB copy).
This commit is contained in:
Keysat
2026-06-18 09:51:41 -05:00
parent 41def0f014
commit 5faa5ae4d6
16 changed files with 783 additions and 17 deletions
@@ -0,0 +1,30 @@
-- ============================================================================
-- email_proposal_matrix — Matrix-review state for an email_activity_proposal,
-- kept 1:1 with the proposal (proposal_id PK). The CRM runs on the box and has
-- no matrix-nio, so it cannot post to Matrix itself: the intake bot (on the Spark)
-- PULLS pending proposals, posts a review card to the dedicated Matrix review room,
-- and writes the thread-root event_id back here. Persisting it CRM-side (not just in
-- the bot's memory) keeps both surfaces in sync and survives a bot restart.
--
-- A SIDE TABLE rather than new columns on email_activity_proposals because the
-- email-integration migration runner (email_integration/db.py:apply_migrations)
-- re-runs every .sql file on every boot via executescript with no ledger — so
-- CREATE TABLE IF NOT EXISTS is idempotent, whereas ALTER ... ADD COLUMN would throw
-- "duplicate column" on the second boot and abort startup. Reversal: DROP TABLE
-- (this runner has no .down.sql convention; cf. 0001/0002).
--
-- posted_at — set once the bot has posted the review card (event_id = thread root).
-- closed_at — set when the thread is resolved: either the bot decided in-thread, OR
-- the bot announced a web-side decision. A posted+decided proposal with
-- closed_at NULL is exactly the bot's signal to post "decided on the web"
-- into the thread and then close it.
-- ============================================================================
CREATE TABLE IF NOT EXISTS email_proposal_matrix (
proposal_id TEXT PRIMARY KEY,
event_id TEXT, -- Matrix thread-root event id of the posted review card
posted_at TEXT,
closed_at TEXT,
created_at TEXT DEFAULT (datetime('now')),
FOREIGN KEY(proposal_id) REFERENCES email_activity_proposals(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_email_proposal_matrix_event ON email_proposal_matrix(event_id);
+100 -3
View File
@@ -14,6 +14,7 @@ import asyncio
from nio import AsyncClient, MatrixRoom, RoomMessageText
import crm_client
import email_proposals
import matrix_io
import parse
import proposals
@@ -25,6 +26,8 @@ UNCLEAR_HELP = (
"or a note like `Note for Acme Capital: wants the Q3 deck, follow up next week`."
)
EMAIL_POLL_SEC = 20 # how often the bot polls the CRM for new/decided email-activity proposals
async def main():
mx = settings.matrix_settings()
@@ -37,6 +40,8 @@ async def main():
roster = settings.team_roster() # frames the parse: teammates do outreach, aren't prospects
if roster:
print(f"matrix-intake: team roster loaded ({len(roster)} names)", flush=True)
review_room = settings.email_review_room() # CRM-drafted email proposals (empty → feature off)
email_threads = {} # Matrix thread-root event_id -> {id, investor_name, note} for an email proposal
async def handle_intake(room_id, root, text):
# A bare yes/no/approve typed in the MAIN timeline (not inside a proposal's thread) is
@@ -157,15 +162,103 @@ async def main():
store.put(root, proposal)
await say(room_id, "I didn't catch that.\n\n" + proposals.render_disambiguation(proposal), root)
async def handle_email_reply(room_id, root, text):
"""An in-thread reply to a CRM-drafted email-proposal card: yes commits, no dismisses, and
anything else is a natural-language revision of the note (re-drafted by local Qwen; the
human still approves the revised note, so the draft→approve gate holds)."""
item = email_threads.get(root)
if item is None:
return # a threaded reply we don't own (or already resolved)
decision = email_proposals.interpret(text)
if decision == "approve":
# Claim before the await (double-approve guard, like the intake commit path).
email_threads.pop(root, None)
try:
await asyncio.to_thread(crm_client.decide_email_proposal, item["id"], "approve", item.get("note"))
except Exception as exc:
email_threads[root] = item # restore for retry
await say(room_id, f"⚠️ couldn't add it ({str(exc)[:200]}). Reply **yes** to retry, **no** to dismiss.", root)
return
await say(room_id, f"✅ Added to the grid for **{item.get('investor_name') or 'the investor'}**.", root)
elif decision == "reject":
email_threads.pop(root, None)
try:
await asyncio.to_thread(crm_client.decide_email_proposal, item["id"], "dismiss")
except Exception as exc:
email_threads[root] = item
await say(room_id, f"⚠️ couldn't dismiss it ({str(exc)[:200]}). Try again.", root)
return
await say(room_id, "🗑️ Dismissed — nothing added to the grid.", root)
else:
try:
new_note = await asyncio.to_thread(email_proposals.revise_note, item.get("note") or "", text)
except Exception as exc:
await say(room_id, f"⚠️ couldn't revise that ({str(exc)[:200]}). Reply **yes** to add as-is, "
"**no** to dismiss, or rephrase.", root)
return
if not new_note:
await say(room_id, "I didn't catch a change. Reply **yes** to add the note as-is, **no** to "
"dismiss, or tell me how to change it.", root)
return
item["note"] = new_note
email_threads[root] = item
await say(room_id, f"✏️ Updated draft note:\n\n{new_note}\n\nReply **yes** to add it, **no** to "
"dismiss, or refine again.", root)
async def poll_email_proposals():
"""Poll the CRM for email-activity proposals: post a review card for each new one, rebuild
the reply-routing map from already-posted threads (so replies still route after a restart),
and announce+close any decided on the web. One failing cycle logs and retries next tick."""
while True:
try:
lists = await asyncio.to_thread(crm_client.list_email_proposals)
for it in lists["open"]: # rebuild routing for threads posted before (e.g. a restart)
ev = it.get("event_id")
if ev and ev not in email_threads:
email_threads[ev] = {"id": it["id"], "investor_name": it.get("investor_name"),
"note": it.get("proposed_note") or ""}
for it in lists["to_post"]:
try:
resp = await client.room_send(
review_room, "m.room.message",
matrix_io.thread_content(email_proposals.render_card(it), None))
ev = getattr(resp, "event_id", None)
if not ev:
print(f"matrix-intake: card send returned no event_id for {it['id']}", flush=True)
continue
await asyncio.to_thread(crm_client.mark_email_proposal_posted, it["id"], ev)
email_threads[ev] = {"id": it["id"], "investor_name": it.get("investor_name"),
"note": it.get("proposed_note") or ""}
except Exception as exc:
print(f"matrix-intake: failed to post email proposal {it.get('id')}: {exc}", flush=True)
for it in lists["to_close"]: # decided on the web → announce in-thread, then close
ev = it.get("event_id")
if not ev:
continue
try:
await say(review_room, email_proposals.closure_line(it.get("status")), ev)
await asyncio.to_thread(crm_client.mark_email_proposal_closed, it["id"])
email_threads.pop(ev, None)
except Exception as exc:
print(f"matrix-intake: failed to close email proposal {it.get('id')}: {exc}", flush=True)
except Exception as exc:
print(f"matrix-intake: email-proposal poll error: {exc}", flush=True)
await asyncio.sleep(EMAIL_POLL_SEC)
async def on_message(room: MatrixRoom, event: RoomMessageText):
if event.sender == mx["user_id"]:
return # never react to our own messages (we post in-thread — this prevents loops)
if room.room_id != intake_room:
return
text = (event.body or "").strip()
if not text:
return
root = matrix_io.thread_root_of(event)
# Email-proposal review room: only a threaded reply to a card we posted is actionable.
if review_room and room.room_id == review_room:
if root and root in email_threads:
await handle_email_reply(room.room_id, root, text)
return
if room.room_id != intake_room:
return
if root and store.has(root):
await handle_reply(room.room_id, root, text)
elif root:
@@ -180,8 +273,12 @@ async def main():
client.add_event_callback(on_message, RoomMessageText)
who = await client.whoami()
print(f"matrix-intake: listening as {who.user_id} in room {intake_room}", flush=True)
tasks = [asyncio.create_task(client.sync_forever(timeout=30000))]
if review_room:
tasks.append(asyncio.create_task(poll_email_proposals()))
print(f"matrix-intake: reviewing email proposals in room {review_room} (every {EMAIL_POLL_SEC}s)", flush=True)
try:
await client.sync_forever(timeout=30000)
await asyncio.gather(*tasks)
finally:
await client.close()
+41
View File
@@ -98,6 +98,47 @@ def match(proposal):
return {"match": match_out, "candidates": candidates}
def list_email_proposals():
"""Pull the email-activity review work-lists for the poll loop: {to_post, open, to_close}.
to_post = pending, un-posted (post a card); open = posted, awaiting a decision (rebuild the
reply-routing map after a restart); to_close = decided on the web (announce in-thread + close)."""
status, data = _authed("GET", "/api/intake/email-proposals")
if status != 200:
raise RuntimeError(f"email-proposals list failed ({status}): {data.get('error') or data}")
payload = data.get("data") or {}
return {k: (payload.get(k) or []) for k in ("to_post", "open", "to_close")}
def mark_email_proposal_posted(proposal_id, event_id):
"""Record the Matrix thread-root event id so the proposal's review state survives a restart."""
status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/matrix",
{"event_id": event_id})
if status != 200:
raise RuntimeError(f"mark posted failed ({status}): {data.get('error') or data}")
return data.get("data") or {}
def mark_email_proposal_closed(proposal_id):
"""Mark the review thread resolved after announcing a web-side decision in it."""
status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/matrix",
{"closed": True})
if status != 200:
raise RuntimeError(f"mark closed failed ({status}): {data.get('error') or data}")
return data.get("data") or {}
def decide_email_proposal(proposal_id, decision, note=None):
"""Relay an in-thread approve/dismiss (with the possibly-revised note) to the CRM. The server
appends the note to the grid on approve, tags source='matrix', and closes the thread."""
body = {"decision": decision}
if note is not None:
body["note"] = note
status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/decide", body)
if status not in (200, 201):
raise RuntimeError(f"email-proposal decide failed ({status}): {data.get('error') or data}")
return data.get("data") or {}
def build_commit_payload(proposal):
"""Pure: map a proposal to the /api/fundraising/log-communication request body.
+85
View File
@@ -0,0 +1,85 @@
"""Email-activity proposal review over Matrix — the CRM→Matrix leg of the email-capture flow.
The CRM (on the box) drafts a proposed grid note per newly-matched email (local model, no Claude)
and queues it for human review. The CRM is stdlib-only and can't post to Matrix itself, so this
bot PULLS the pending proposals (crm_client.list_email_proposals), posts a review card to the
dedicated review room, and relays the human's in-thread reply back to the CRM. Same draft→approve
discipline as the intake bot: nothing is appended to the grid until a human approves — here OR on
the web Email Capture panel, the two surfaces kept in sync via the CRM's email_proposal_matrix row.
This module is the PURE logic (card rendering, reply grammar, note revision) so it's unit-tested
offline; the async poll/post/reply wiring lives in bot.py (network + Matrix, live-smoke only).
"""
import spark
_YES = {"yes", "y", "approve", "approved", "ok", "confirm", "go", "add", "👍", ""}
_NO = {"no", "n", "cancel", "discard", "reject", "skip", "stop", "👎", ""}
_SNIPPET_MAX = 400 # email snippet shown on the card; the full body is in the web popup
def _truncate(s, n):
s = (s or "").strip()
return s if len(s) <= n else s[:n].rstrip() + ""
def render_card(item):
"""The review card posted to the Matrix review room: who/when + a short email snippet + the
drafted note. Deliberately compact for mobile — the full scrollable body is in the web Email
Capture popup (this is the metadata+snippet+note choice)."""
name = item.get("investor_name") or "Unknown investor"
direction = "Sent" if item.get("direction") == "sent" else "Received"
frm = item.get("from_name") or item.get("from_email") or "?"
lines = [f"📧 Proposed **grid note** for **{name}** ({direction})"]
if item.get("email_subject"):
lines.append(f"· Subject: {item['email_subject']}")
if item.get("email_date"):
lines.append(f"· Date: {item['email_date']}")
lines.append(f"· From: {frm}")
snippet = _truncate(item.get("snippet"), _SNIPPET_MAX)
if snippet:
lines.append(f"· Email: {snippet}")
lines.append("")
lines.append(f"📝 Draft note: {item.get('proposed_note') or '(empty)'}")
lines.append("")
lines.append("Reply **yes** to add it to the grid, **no** to dismiss, or just tell me how to "
"change the note (e.g. *say we discussed the Q3 raise*).")
return "\n".join(lines)
def closure_line(status):
"""Posted in-thread when a proposal was decided on the WEB while its Matrix thread was open."""
verb = "approved ✅ and added to the grid" if status == "approved" else "dismissed 🗑️"
return f"This was {verb} on the web — nothing more to do here. Thread closed."
def interpret(text):
"""Classify an in-thread reply: 'approve' | 'reject' | 'revise' (anything else → revise the note)."""
t = (text or "").strip().lower()
if t in _YES:
return "approve"
if t in _NO:
return "reject"
return "revise"
REVISE_SYSTEM = (
"You revise a single CRM note from a short instruction a venture-fund team member typed. "
"You are given the CURRENT note and an INSTRUCTION. Apply the instruction and reply with "
"ONLY a JSON object of the form {\"note\": \"<the full revised note>\"}. Keep it to one or two "
"factual sentences, no preamble. Output JSON only."
)
def revise_note(note, instruction, parse_fn=spark.parse_json):
"""Re-draft the note via local Qwen from a free-form instruction (no Claude, no scrub — same
local-only basis as the intake parse). Returns the new note text, or None if the model gave
nothing usable / unchanged, in which case the caller re-prompts. `parse_fn` is injectable for
tests."""
prompt = "CURRENT:\n" + (note or "") + "\n\nINSTRUCTION:\n" + (instruction or "").strip()
raw = parse_fn(prompt, system=REVISE_SYSTEM, max_tokens=400) or {}
new = raw.get("note") if isinstance(raw, dict) else None
new = (new or "").strip()
if not new or new == (note or "").strip():
return None
return new
+7
View File
@@ -61,3 +61,10 @@ def crm_settings():
# unset/empty just means no roster framing, i.e. the prior behavior.
def team_roster():
return [n.strip() for n in os.environ.get("INTAKE_TEAM_ROSTER", "").split(",") if n.strip()]
# Dedicated room for reviewing CRM-drafted email-activity proposals (the CRM→Matrix push leg).
# Separate from the intake room so high-volume email proposals don't drown the conversational
# intake flow. Unset/empty disables the whole email-review poll loop (the bot just does intake).
def email_review_room():
return os.environ.get("MATRIX_EMAIL_REVIEW_ROOM", "").strip()
@@ -0,0 +1,72 @@
"""Offline tests for the email-proposal review logic (card render, reply grammar, note revision).
The network/Matrix wiring lives in bot.py (live-smoke only); this covers the pure functions."""
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import email_proposals # noqa: E402
ITEM = {
"id": "p1", "investor_name": "Acme Capital", "direction": "received",
"from_name": "Jane Doe", "from_email": "jane@acme.com",
"email_subject": "Re: Fund III", "email_date": "2026-06-02",
"snippet": "thanks for the deck — one question on terms", "proposed_note": "✉ Received: asked about terms",
}
def test_interpret_yes_no_else():
assert email_proposals.interpret("yes") == "approve"
assert email_proposals.interpret(" Y ") == "approve"
assert email_proposals.interpret("") == "approve"
assert email_proposals.interpret("no") == "reject"
assert email_proposals.interpret("skip") == "reject"
# anything that isn't a clear yes/no is treated as a revision instruction
assert email_proposals.interpret("say we discussed the Q3 raise") == "revise"
def test_render_card_has_context_note_and_actions():
card = email_proposals.render_card(ITEM)
assert "Acme Capital" in card and "Received" in card
assert "Jane Doe" in card
assert "Re: Fund III" in card and "2026-06-02" in card
assert "thanks for the deck" in card
assert "✉ Received: asked about terms" in card
assert "yes" in card.lower() and "no" in card.lower()
def test_render_card_sent_direction():
assert "(Sent)" in email_proposals.render_card(dict(ITEM, direction="sent"))
def test_render_card_truncates_long_snippet():
card = email_proposals.render_card(dict(ITEM, snippet="x" * 1000))
assert "" in card and len(card) < 1000
def test_revise_note_applies_model_output():
out = email_proposals.revise_note(
"old note", "make it about the Q3 raise",
parse_fn=lambda prompt, system=None, max_tokens=400: {"note": "Discussed the Q3 raise."})
assert out == "Discussed the Q3 raise."
def test_revise_note_noop_or_empty_returns_none():
# model echoes the same note unchanged -> None so the caller re-prompts (not "Updated")
assert email_proposals.revise_note("same", "x", parse_fn=lambda *a, **k: {"note": "same"}) is None
# model returns nothing usable -> None
assert email_proposals.revise_note("n", "y", parse_fn=lambda *a, **k: {}) is None
assert email_proposals.revise_note("n", "y", parse_fn=lambda *a, **k: None) is None
def test_closure_line_reflects_status():
assert "approved" in email_proposals.closure_line("approved").lower()
assert "dismiss" in email_proposals.closure_line("dismissed").lower()
if __name__ == "__main__":
fns = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
for fn in fns:
fn()
print(f"ok {fn.__name__}")
print(f"\n{len(fns)} passed")
+153 -7
View File
@@ -619,6 +619,14 @@ def parse_iso_utc(ts):
def require_admin(user):
return bool(user and user.get('role') == 'admin')
def require_bot_or_admin(user):
"""Gate for agent/bot-facing endpoints: a dedicated 'bot' service account OR an admin
(admins keep parity for debugging/curl). The 'bot' role is authenticated-but-not-admin
it never passes require_admin, so a bot credential cannot reach user-management, security,
or settings. Reach (which endpoints) is controlled here; autonomy (acting without a human)
stays governed by the per-feature draft->approve gate, independent of role."""
return bool(user and user.get('role') in ('admin', 'bot'))
def log_audit(conn, user_id, entity_type, entity_id, action, changes=None):
conn.execute(
"INSERT INTO audit_log (id, user_id, entity_type, entity_id, action, changes) VALUES (?, ?, ?, ?, ?, ?)",
@@ -2068,6 +2076,9 @@ class CRMHandler(BaseHTTPRequestHandler):
# Matrix intake bot — new-vs-existing lookup for its in-thread proposal
if path == '/api/intake/match':
return self.handle_intake_match(user, params)
# Matrix review bot — email-activity proposal work-lists (to_post/open/to_close)
if path == '/api/intake/email-proposals':
return self.handle_list_bot_email_proposals(user)
# Users
if path == '/api/users':
@@ -2187,6 +2198,10 @@ class CRMHandler(BaseHTTPRequestHandler):
return self.handle_decide_activity_proposal(user, path.split('/')[-2], 'approve', body)
if re.match(r'^/api/activity/proposals/[^/]+/dismiss$', path):
return self.handle_decide_activity_proposal(user, path.split('/')[-2], 'dismiss', body)
if re.match(r'^/api/intake/email-proposals/[^/]+/matrix$', path):
return self.handle_mark_email_proposal_matrix(user, path.split('/')[-2], body)
if re.match(r'^/api/intake/email-proposals/[^/]+/decide$', path):
return self.handle_decide_email_proposal_matrix(user, path.split('/')[-2], body)
if re.match(r'^/api/thesis/nodes/[^/]+/choose$', path):
return self.handle_choose_variant(user, path.split('/')[-2])
if re.match(r'^/api/thesis/lines/[^/]+/approve$', path):
@@ -3964,6 +3979,59 @@ class CRMHandler(BaseHTTPRequestHandler):
return self.send_error_json(res["error"], code)
return self.send_json({"data": res})
# ─── Matrix review-bot bridge for email-activity proposals (bot-or-admin) ───
def handle_list_bot_email_proposals(self, user):
"""The bot's poll endpoint: {to_post, open, to_close}. Bot-or-admin (the proposals
carry LP email content, so this stays off the member tier)."""
if not require_bot_or_admin(user):
return self.send_error_json("Bot or admin required", 403)
conn = get_db()
try:
return self.send_json({"data": list_bot_email_proposals(conn)})
finally:
conn.close()
def handle_mark_email_proposal_matrix(self, user, proposal_id, body):
"""Record Matrix thread state: {event_id} marks the card posted; {closed:true} marks
the thread resolved after the bot announced a web-side decision."""
if not require_bot_or_admin(user):
return self.send_error_json("Bot or admin required", 403)
body = body or {}
conn = get_db()
try:
if body.get("closed"):
res = mark_proposal_matrix_closed(conn, proposal_id)
else:
event_id = str(body.get("event_id") or "").strip()
if not event_id:
return self.send_error_json("event_id or closed is required")
res = mark_proposal_matrix_posted(conn, proposal_id, event_id)
finally:
conn.close()
if res.get("error"):
return self.send_error_json(res["error"], 404 if res["error"] == "not_found" else 400)
return self.send_json({"data": res})
def handle_decide_email_proposal_matrix(self, user, proposal_id, body):
"""In-thread Matrix decision relayed by the bot: approve/dismiss (+ optional edited note),
tagged source='matrix' and closing the thread in the same transaction."""
if not require_bot_or_admin(user):
return self.send_error_json("Bot or admin required", 403)
body = body or {}
decision = str(body.get("decision") or "").strip()
if decision not in ("approve", "dismiss"):
return self.send_error_json("decision must be approve or dismiss")
conn = get_db()
try:
res = decide_email_activity_proposal(conn, proposal_id, decision, user['user_id'],
body.get('note'), source="matrix", close_matrix=True)
finally:
conn.close()
if res.get("error"):
code = {"not_found": 404, "already_decided": 409}.get(res["error"], 400)
return self.send_error_json(res["error"], code)
return self.send_json({"data": res})
# ─── UI-triggered index jobs + entity-merge review (Phase 1) ───
def handle_index_job(self, user, kind):
if not require_admin(user):
@@ -4357,8 +4425,10 @@ class CRMHandler(BaseHTTPRequestHandler):
return self.send_error_json("password must be at least 8 characters")
role = body.get('role', 'member')
if role not in ('admin', 'member'):
return self.send_error_json("role must be admin or member")
# 'bot' is a deliberately-provisioned agent service account (kept out of the invite UI's
# member/admin dropdown) — authenticated but never an admin. See require_bot_or_admin.
if role not in ('admin', 'member', 'bot'):
return self.send_error_json("role must be admin, member, or bot")
conn = get_db()
existing = conn.execute(
@@ -4417,9 +4487,9 @@ class CRMHandler(BaseHTTPRequestHandler):
if 'role' in body:
role = str(body.get('role'))
if role not in ('admin', 'member'):
if role not in ('admin', 'member', 'bot'):
conn.close()
return self.send_error_json("role must be admin or member")
return self.send_error_json("role must be admin, member, or bot")
sets.append("role = ?")
args.append(role)
@@ -5728,8 +5798,14 @@ def list_email_activity_proposals(conn, status="pending", limit=200):
return []
def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_note=None):
"""Approve (optionally with an edited note -> append to grid) or dismiss a proposal."""
def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_note=None,
source="crm_ui", close_matrix=False):
"""Approve (optionally with an edited note -> append to grid) or dismiss a proposal.
`source` records the channel in the audit log ('crm_ui' for the web panel, 'matrix' for an
in-thread approval relayed by the review bot). `close_matrix` is set by the bot when the
decision was made in-thread: it stamps the Matrix side row closed in the same transaction so
the web->Matrix close path (list_bot_email_proposals.to_close) won't re-announce it."""
p = conn.execute("SELECT * FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone()
if not p:
return {"error": "not_found"}
@@ -5747,15 +5823,85 @@ def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_
action, result = "email.activity_dismissed", {"status": "dismissed"}
else:
return {"error": "bad_decision"}
if close_matrix:
_mark_proposal_matrix(conn, proposal_id, closed_at=now())
conn.execute(
"INSERT INTO interaction_log (id, ts, actor_type, actor_id, action, target_type, target_id, payload, source, created_at) "
"VALUES (?,?,?,?,?,?,?,?,?,?)",
(generate_id(), now(), "human", user_id, action, "fundraising_investor", p["investor_id"],
json.dumps({"proposal_id": proposal_id}), "crm_ui", now()))
json.dumps({"proposal_id": proposal_id}), source, now()))
conn.commit()
return result
# ─── Matrix review-bot bridge for email-activity proposals (Features 2/3) ──────
# The CRM (box, stdlib, no matrix-nio) can't post to Matrix, so the intake bot (Spark) PULLS
# pending proposals via list_bot_email_proposals, posts a review card to the dedicated review
# room, and writes the thread-root event_id back here. State lives CRM-side (email_proposal_matrix)
# so both surfaces stay in sync and it survives a bot restart. All queries degrade to empty when
# the email integration tables are absent (OperationalError), mirroring list_email_activity_proposals.
_BOT_PROPOSAL_COLS = (
"SELECT p.id, p.investor_name, p.direction, p.summary, p.proposed_note, "
"p.email_subject, p.email_date, e.from_name, e.from_email, e.snippet, "
"m.event_id AS event_id, p.status AS status "
"FROM email_activity_proposals p "
"LEFT JOIN email_proposal_matrix m ON m.proposal_id = p.id "
"LEFT JOIN emails e ON e.id = p.email_id ")
def list_bot_email_proposals(conn, limit=100):
"""The three work-lists the Matrix review bot polls:
to_post pending, not yet posted to Matrix -> bot posts a review card.
open pending, posted, not closed -> live threads; the bot rebuilds its
event_id->proposal routing map from these after a restart.
to_close decided on the WEB while a thread is open -> bot announces it in-thread, closes.
Each item carries the card content (investor, direction, subject, date, from, snippet, note)."""
try:
to_post = [dict(r) for r in conn.execute(
_BOT_PROPOSAL_COLS + "WHERE p.status='pending' AND (m.proposal_id IS NULL OR m.posted_at IS NULL) "
"ORDER BY p.email_date ASC, p.created_at ASC LIMIT ?", (limit,))]
open_threads = [dict(r) for r in conn.execute(
_BOT_PROPOSAL_COLS + "WHERE p.status='pending' AND m.posted_at IS NOT NULL AND m.closed_at IS NULL "
"ORDER BY p.email_date ASC, p.created_at ASC LIMIT ?", (limit,))]
to_close = [dict(r) for r in conn.execute(
_BOT_PROPOSAL_COLS + "WHERE p.status!='pending' AND m.posted_at IS NOT NULL AND m.closed_at IS NULL "
"ORDER BY p.decided_at ASC LIMIT ?", (limit,))]
except sqlite3.OperationalError:
return {"to_post": [], "open": [], "to_close": []}
return {"to_post": to_post, "open": open_threads, "to_close": to_close}
def _mark_proposal_matrix(conn, proposal_id, *, event_id=None, posted_at=None, closed_at=None):
"""Idempotent upsert of the 1:1 Matrix side row. Only the passed fields are written."""
cols, vals, sets = ["proposal_id"], [proposal_id], []
for name, val in (("event_id", event_id), ("posted_at", posted_at), ("closed_at", closed_at)):
if val is not None:
cols.append(name); vals.append(val); sets.append(f"{name}=excluded.{name}")
placeholders = ",".join("?" for _ in cols)
sql = f"INSERT INTO email_proposal_matrix ({','.join(cols)}) VALUES ({placeholders})"
if sets:
sql += " ON CONFLICT(proposal_id) DO UPDATE SET " + ",".join(sets)
conn.execute(sql, vals)
def mark_proposal_matrix_posted(conn, proposal_id, event_id):
"""Record that the bot posted a review card (thread root = event_id)."""
if not conn.execute("SELECT 1 FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone():
return {"error": "not_found"}
_mark_proposal_matrix(conn, proposal_id, event_id=event_id, posted_at=now())
conn.commit()
return {"ok": True}
def mark_proposal_matrix_closed(conn, proposal_id):
"""Mark the Matrix review thread resolved (the bot announced a web-side decision)."""
if not conn.execute("SELECT 1 FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone():
return {"error": "not_found"}
_mark_proposal_matrix(conn, proposal_id, closed_at=now())
conn.commit()
return {"ok": True}
# ─── Main Entry Point ────────────────────────────────────────────────────────
def main():
+129
View File
@@ -0,0 +1,129 @@
#!/usr/bin/env python3
"""Test the Matrix review-bot bridge for email-activity proposals (Features 2/3):
the bot work-lists (to_post / open / to_close), the Matrix side-row mark helpers, and an
in-thread (source='matrix') decision that closes the thread — plus the bot-or-admin role gate.
Synthetic data only (guardrail #9). The local model is stubbed.
Run: cd backend && python3 test_email_proposal_matrix.py
"""
import json
import os
import sqlite3
import sys
import tempfile
os.environ["CRM_DB_PATH"] = os.path.join(tempfile.mkdtemp(), "crm.db")
os.environ.setdefault("CRM_DATA_DIR", os.path.dirname(os.environ["CRM_DB_PATH"]))
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import server # noqa: E402
server._summarize_email_gist = lambda subject, body: "fundraising update; proposed a call"
FAILS = []
def check(cond, msg):
print((" PASS " if cond else " FAIL ") + msg)
if not cond:
FAILS.append(msg)
def setup():
conn = sqlite3.connect(os.environ["CRM_DB_PATH"])
conn.row_factory = sqlite3.Row
conn.executescript("""
CREATE TABLE app_settings (key TEXT PRIMARY KEY, value_json TEXT, updated_at TEXT);
CREATE TABLE email_accounts (id TEXT, email_address TEXT, sync_enabled INT DEFAULT 1, sync_status TEXT, backfill_complete INT);
CREATE TABLE emails (id TEXT PRIMARY KEY, subject TEXT, body_text TEXT, snippet TEXT, from_name TEXT, from_email TEXT, sent_at TEXT, is_matched INT, match_status TEXT);
CREATE TABLE email_investor_links (id TEXT, email_id TEXT, fundraising_investor_id TEXT, organization_id TEXT, contact_id TEXT, match_confidence REAL);
CREATE TABLE email_activity_proposals (id TEXT PRIMARY KEY, email_id TEXT UNIQUE, investor_id TEXT, investor_name TEXT,
direction TEXT, summary TEXT, proposed_note TEXT, email_subject TEXT, email_date TEXT, status TEXT DEFAULT 'pending',
decided_by TEXT, decided_at TEXT, final_note TEXT, created_at TEXT);
CREATE TABLE email_proposal_matrix (proposal_id TEXT PRIMARY KEY, event_id TEXT, posted_at TEXT, closed_at TEXT, created_at TEXT);
CREATE TABLE users (id TEXT PRIMARY KEY, username TEXT);
CREATE TABLE fundraising_investors (id TEXT PRIMARY KEY, investor_name TEXT, notes TEXT);
CREATE TABLE fundraising_state (id TEXT PRIMARY KEY, grid_json TEXT, views_json TEXT, version INT,
updated_by TEXT REFERENCES users(id), updated_at TEXT);
CREATE TABLE interaction_log (id TEXT PRIMARY KEY, ts TEXT, actor_type TEXT, actor_id TEXT, action TEXT, target_type TEXT, target_id TEXT, payload TEXT, source TEXT, created_at TEXT);
""")
conn.execute("INSERT INTO users (id,username) VALUES ('user-1','grant')")
conn.execute("INSERT INTO app_settings VALUES ('email_activity_since', ?, ?)", (json.dumps("2026-01-01T00:00:00"), "x"))
conn.execute("INSERT INTO email_accounts (id,email_address) VALUES ('a','grant@ten31.xyz')")
conn.execute("INSERT INTO fundraising_investors (id,investor_name,notes) VALUES ('inv1','Harbor & Vine','existing note')")
grid = {"columns": [], "rows": [{"id": "inv1", "investor_name": "Harbor & Vine", "notes": "existing note"}]}
conn.execute("INSERT INTO fundraising_state (id,grid_json,views_json,version) VALUES ('main',?,?,1)", (json.dumps(grid), "[]"))
conn.executemany("INSERT INTO emails (id,subject,body_text,snippet,from_name,from_email,sent_at,is_matched,match_status) VALUES (?,?,?,?,?,?,?,1,'matched')", [
("e1", "Fund III", "Here is the update", "the quarterly update is attached", "Grant", "grant@ten31.xyz", "2026-06-01T10:00:00"),
("e2", "Re: Fund III", "Thanks, a question", "thanks — one question on terms", "LP Contact", "lp@harborvine.example", "2026-06-02T10:00:00"),
])
conn.executemany("INSERT INTO email_investor_links (id,email_id,fundraising_investor_id,match_confidence) VALUES (?,?, 'inv1', 1.0)",
[("l1", "e1"), ("l2", "e2")])
conn.commit()
conn.close()
def main():
setup()
# role gate: bot passes the agent gate but is NOT an admin; member passes neither.
check(server.require_bot_or_admin({"role": "bot"}), "bot passes require_bot_or_admin")
check(server.require_bot_or_admin({"role": "admin"}), "admin passes require_bot_or_admin")
check(not server.require_bot_or_admin({"role": "member"}), "member does NOT pass require_bot_or_admin")
check(not server.require_admin({"role": "bot"}), "bot is NOT an admin (no user-mgmt/settings reach)")
check(server.propose_email_activity_notes().get("proposed") == 2, "drafts 2 proposals")
conn = server.get_db()
props = server.list_email_activity_proposals(conn, status="pending")
by_email = {p["email_id"]: p for p in props}
p_a, p_b = by_email["e1"], by_email["e2"]
# Both are pending + un-posted → both in to_post; card carries from/snippet/note context.
lists = server.list_bot_email_proposals(conn)
check(len(lists["to_post"]) == 2 and not lists["open"] and not lists["to_close"], "both proposals queued to_post")
card = next(it for it in lists["to_post"] if it["id"] == p_a["id"])
check(card.get("from_name") == "Grant" and "quarterly update" in (card.get("snippet") or ""), "card carries from_name + snippet")
check("" in (card.get("proposed_note") or ""), "card carries the drafted note")
# Post p_a to Matrix → it leaves to_post and becomes an open thread (event id recorded).
server.mark_proposal_matrix_posted(conn, p_a["id"], "evtA")
lists = server.list_bot_email_proposals(conn)
check(len(lists["to_post"]) == 1 and lists["to_post"][0]["id"] == p_b["id"], "posting p_a leaves only p_b to_post")
check(len(lists["open"]) == 1 and lists["open"][0]["id"] == p_a["id"] and lists["open"][0]["event_id"] == "evtA",
"posted p_a is an open thread carrying its event id")
# Decide p_a IN-THREAD on Matrix (approve + close in one transaction).
r = server.decide_email_activity_proposal(conn, p_a["id"], "approve", "user-1", source="matrix", close_matrix=True)
check(r.get("status") == "approved" and r.get("placed_in_grid") is True, "matrix approve appends to the grid")
lists = server.list_bot_email_proposals(conn)
check(not any(it["id"] == p_a["id"] for it in lists["open"] + lists["to_close"]),
"matrix-decided proposal is closed (not re-announced via to_close)")
src = conn.execute("SELECT source FROM interaction_log WHERE action='email.activity_approved'").fetchone()["source"]
check(src == "matrix", "matrix decision is audited source='matrix'")
# Web-decide path: post p_b, then dismiss it on the WEB (default source, no close) → the bot
# must see it in to_close so it can announce the web decision in-thread, then close.
server.mark_proposal_matrix_posted(conn, p_b["id"], "evtB")
server.decide_email_activity_proposal(conn, p_b["id"], "dismiss", "user-1") # web path: source crm_ui, no close
lists = server.list_bot_email_proposals(conn)
check(len(lists["to_close"]) == 1 and lists["to_close"][0]["id"] == p_b["id"] and lists["to_close"][0]["status"] == "dismissed",
"web-decided open thread surfaces in to_close")
src2 = conn.execute("SELECT source FROM interaction_log WHERE action='email.activity_dismissed'").fetchone()["source"]
check(src2 == "crm_ui", "web decision is audited source='crm_ui'")
server.mark_proposal_matrix_closed(conn, p_b["id"])
lists = server.list_bot_email_proposals(conn)
check(not lists["to_close"] and not lists["open"], "closing the thread clears the work-lists")
# Marking a non-existent proposal is a clean not_found, not a crash.
check(server.mark_proposal_matrix_posted(conn, "nope", "evtX").get("error") == "not_found", "mark posted on unknown id -> not_found")
conn.close()
if FAILS:
print(f"\nFAILED ({len(FAILS)})")
for f in FAILS:
print(" - " + f)
sys.exit(1)
print("\nALL PASS (email-proposal Matrix bridge)")
if __name__ == "__main__":
main()