Email-proposal review over Matrix + a bot role (v0.1.0:89)
The email-capture "proposed grid notes" gain two review surfaces:
1. Inline source email — each proposed-note card on the Email Capture page
gets a "View email" toggle that lazily fetches the existing
GET /api/email/detail and shows from/to/cc/date/subject + scrollable body,
so a reviewer can judge the note against the email it was drafted from.
2. CRM->Matrix review bridge — the CRM (box, stdlib, no matrix-nio) can't post
to Matrix, so the intake bot (Spark) PULLS: GET /api/intake/email-proposals
returns to_post/open/to_close work-lists; the bot posts a review card
(metadata + snippet + draft note) to a dedicated review room
(MATRIX_EMAIL_REVIEW_ROOM) and relays in-thread yes / no / NL-edit
(POST .../{id}/decide, note revised via local Qwen). Decisions sync both
ways: web decide -> bot announces + closes the thread; Matrix decide -> the
web panel's ~25s poll clears the card. State lives CRM-side in the new
email_proposal_matrix side row (email-integration migration 0003, additive
+ idempotent CREATE TABLE IF NOT EXISTS), so it survives a bot restart.
Adds a 'bot' role (authenticated, never admin; require_bot_or_admin) to gate
the email-proposal endpoints rather than handing the bot full admin — the
principled base for the coming agentic capabilities. Role controls reach;
the draft->approve gate still controls autonomy (a human approves every write).
Deploy split: endpoints + migration + role + frontend ship in the s9pk; the
bot poll loop + review-room handling ship on the Spark. The bot's CRM user
must be flipped member->bot and joined to the review room (one-time).
Tests: backend/test_email_proposal_matrix.py + matrix_intake/test_email_proposals.py
(30/30 suite green, render-smoke green, migration verified twice on a DB copy).
This commit is contained in:
@@ -0,0 +1,30 @@
|
||||
-- ============================================================================
|
||||
-- email_proposal_matrix — Matrix-review state for an email_activity_proposal,
|
||||
-- kept 1:1 with the proposal (proposal_id PK). The CRM runs on the box and has
|
||||
-- no matrix-nio, so it cannot post to Matrix itself: the intake bot (on the Spark)
|
||||
-- PULLS pending proposals, posts a review card to the dedicated Matrix review room,
|
||||
-- and writes the thread-root event_id back here. Persisting it CRM-side (not just in
|
||||
-- the bot's memory) keeps both surfaces in sync and survives a bot restart.
|
||||
--
|
||||
-- A SIDE TABLE rather than new columns on email_activity_proposals because the
|
||||
-- email-integration migration runner (email_integration/db.py:apply_migrations)
|
||||
-- re-runs every .sql file on every boot via executescript with no ledger — so
|
||||
-- CREATE TABLE IF NOT EXISTS is idempotent, whereas ALTER ... ADD COLUMN would throw
|
||||
-- "duplicate column" on the second boot and abort startup. Reversal: DROP TABLE
|
||||
-- (this runner has no .down.sql convention; cf. 0001/0002).
|
||||
--
|
||||
-- posted_at — set once the bot has posted the review card (event_id = thread root).
|
||||
-- closed_at — set when the thread is resolved: either the bot decided in-thread, OR
|
||||
-- the bot announced a web-side decision. A posted+decided proposal with
|
||||
-- closed_at NULL is exactly the bot's signal to post "decided on the web"
|
||||
-- into the thread and then close it.
|
||||
-- ============================================================================
|
||||
CREATE TABLE IF NOT EXISTS email_proposal_matrix (
|
||||
proposal_id TEXT PRIMARY KEY,
|
||||
event_id TEXT, -- Matrix thread-root event id of the posted review card
|
||||
posted_at TEXT,
|
||||
closed_at TEXT,
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
FOREIGN KEY(proposal_id) REFERENCES email_activity_proposals(id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_email_proposal_matrix_event ON email_proposal_matrix(event_id);
|
||||
@@ -14,6 +14,7 @@ import asyncio
|
||||
from nio import AsyncClient, MatrixRoom, RoomMessageText
|
||||
|
||||
import crm_client
|
||||
import email_proposals
|
||||
import matrix_io
|
||||
import parse
|
||||
import proposals
|
||||
@@ -25,6 +26,8 @@ UNCLEAR_HELP = (
|
||||
"or a note like `Note for Acme Capital: wants the Q3 deck, follow up next week`."
|
||||
)
|
||||
|
||||
EMAIL_POLL_SEC = 20 # how often the bot polls the CRM for new/decided email-activity proposals
|
||||
|
||||
|
||||
async def main():
|
||||
mx = settings.matrix_settings()
|
||||
@@ -37,6 +40,8 @@ async def main():
|
||||
roster = settings.team_roster() # frames the parse: teammates do outreach, aren't prospects
|
||||
if roster:
|
||||
print(f"matrix-intake: team roster loaded ({len(roster)} names)", flush=True)
|
||||
review_room = settings.email_review_room() # CRM-drafted email proposals (empty → feature off)
|
||||
email_threads = {} # Matrix thread-root event_id -> {id, investor_name, note} for an email proposal
|
||||
|
||||
async def handle_intake(room_id, root, text):
|
||||
# A bare yes/no/approve typed in the MAIN timeline (not inside a proposal's thread) is
|
||||
@@ -157,15 +162,103 @@ async def main():
|
||||
store.put(root, proposal)
|
||||
await say(room_id, "I didn't catch that.\n\n" + proposals.render_disambiguation(proposal), root)
|
||||
|
||||
async def handle_email_reply(room_id, root, text):
|
||||
"""An in-thread reply to a CRM-drafted email-proposal card: yes commits, no dismisses, and
|
||||
anything else is a natural-language revision of the note (re-drafted by local Qwen; the
|
||||
human still approves the revised note, so the draft→approve gate holds)."""
|
||||
item = email_threads.get(root)
|
||||
if item is None:
|
||||
return # a threaded reply we don't own (or already resolved)
|
||||
decision = email_proposals.interpret(text)
|
||||
if decision == "approve":
|
||||
# Claim before the await (double-approve guard, like the intake commit path).
|
||||
email_threads.pop(root, None)
|
||||
try:
|
||||
await asyncio.to_thread(crm_client.decide_email_proposal, item["id"], "approve", item.get("note"))
|
||||
except Exception as exc:
|
||||
email_threads[root] = item # restore for retry
|
||||
await say(room_id, f"⚠️ couldn't add it ({str(exc)[:200]}). Reply **yes** to retry, **no** to dismiss.", root)
|
||||
return
|
||||
await say(room_id, f"✅ Added to the grid for **{item.get('investor_name') or 'the investor'}**.", root)
|
||||
elif decision == "reject":
|
||||
email_threads.pop(root, None)
|
||||
try:
|
||||
await asyncio.to_thread(crm_client.decide_email_proposal, item["id"], "dismiss")
|
||||
except Exception as exc:
|
||||
email_threads[root] = item
|
||||
await say(room_id, f"⚠️ couldn't dismiss it ({str(exc)[:200]}). Try again.", root)
|
||||
return
|
||||
await say(room_id, "🗑️ Dismissed — nothing added to the grid.", root)
|
||||
else:
|
||||
try:
|
||||
new_note = await asyncio.to_thread(email_proposals.revise_note, item.get("note") or "", text)
|
||||
except Exception as exc:
|
||||
await say(room_id, f"⚠️ couldn't revise that ({str(exc)[:200]}). Reply **yes** to add as-is, "
|
||||
"**no** to dismiss, or rephrase.", root)
|
||||
return
|
||||
if not new_note:
|
||||
await say(room_id, "I didn't catch a change. Reply **yes** to add the note as-is, **no** to "
|
||||
"dismiss, or tell me how to change it.", root)
|
||||
return
|
||||
item["note"] = new_note
|
||||
email_threads[root] = item
|
||||
await say(room_id, f"✏️ Updated draft note:\n\n{new_note}\n\nReply **yes** to add it, **no** to "
|
||||
"dismiss, or refine again.", root)
|
||||
|
||||
async def poll_email_proposals():
|
||||
"""Poll the CRM for email-activity proposals: post a review card for each new one, rebuild
|
||||
the reply-routing map from already-posted threads (so replies still route after a restart),
|
||||
and announce+close any decided on the web. One failing cycle logs and retries next tick."""
|
||||
while True:
|
||||
try:
|
||||
lists = await asyncio.to_thread(crm_client.list_email_proposals)
|
||||
for it in lists["open"]: # rebuild routing for threads posted before (e.g. a restart)
|
||||
ev = it.get("event_id")
|
||||
if ev and ev not in email_threads:
|
||||
email_threads[ev] = {"id": it["id"], "investor_name": it.get("investor_name"),
|
||||
"note": it.get("proposed_note") or ""}
|
||||
for it in lists["to_post"]:
|
||||
try:
|
||||
resp = await client.room_send(
|
||||
review_room, "m.room.message",
|
||||
matrix_io.thread_content(email_proposals.render_card(it), None))
|
||||
ev = getattr(resp, "event_id", None)
|
||||
if not ev:
|
||||
print(f"matrix-intake: card send returned no event_id for {it['id']}", flush=True)
|
||||
continue
|
||||
await asyncio.to_thread(crm_client.mark_email_proposal_posted, it["id"], ev)
|
||||
email_threads[ev] = {"id": it["id"], "investor_name": it.get("investor_name"),
|
||||
"note": it.get("proposed_note") or ""}
|
||||
except Exception as exc:
|
||||
print(f"matrix-intake: failed to post email proposal {it.get('id')}: {exc}", flush=True)
|
||||
for it in lists["to_close"]: # decided on the web → announce in-thread, then close
|
||||
ev = it.get("event_id")
|
||||
if not ev:
|
||||
continue
|
||||
try:
|
||||
await say(review_room, email_proposals.closure_line(it.get("status")), ev)
|
||||
await asyncio.to_thread(crm_client.mark_email_proposal_closed, it["id"])
|
||||
email_threads.pop(ev, None)
|
||||
except Exception as exc:
|
||||
print(f"matrix-intake: failed to close email proposal {it.get('id')}: {exc}", flush=True)
|
||||
except Exception as exc:
|
||||
print(f"matrix-intake: email-proposal poll error: {exc}", flush=True)
|
||||
await asyncio.sleep(EMAIL_POLL_SEC)
|
||||
|
||||
async def on_message(room: MatrixRoom, event: RoomMessageText):
|
||||
if event.sender == mx["user_id"]:
|
||||
return # never react to our own messages (we post in-thread — this prevents loops)
|
||||
if room.room_id != intake_room:
|
||||
return
|
||||
text = (event.body or "").strip()
|
||||
if not text:
|
||||
return
|
||||
root = matrix_io.thread_root_of(event)
|
||||
# Email-proposal review room: only a threaded reply to a card we posted is actionable.
|
||||
if review_room and room.room_id == review_room:
|
||||
if root and root in email_threads:
|
||||
await handle_email_reply(room.room_id, root, text)
|
||||
return
|
||||
if room.room_id != intake_room:
|
||||
return
|
||||
if root and store.has(root):
|
||||
await handle_reply(room.room_id, root, text)
|
||||
elif root:
|
||||
@@ -180,8 +273,12 @@ async def main():
|
||||
client.add_event_callback(on_message, RoomMessageText)
|
||||
who = await client.whoami()
|
||||
print(f"matrix-intake: listening as {who.user_id} in room {intake_room}", flush=True)
|
||||
tasks = [asyncio.create_task(client.sync_forever(timeout=30000))]
|
||||
if review_room:
|
||||
tasks.append(asyncio.create_task(poll_email_proposals()))
|
||||
print(f"matrix-intake: reviewing email proposals in room {review_room} (every {EMAIL_POLL_SEC}s)", flush=True)
|
||||
try:
|
||||
await client.sync_forever(timeout=30000)
|
||||
await asyncio.gather(*tasks)
|
||||
finally:
|
||||
await client.close()
|
||||
|
||||
|
||||
@@ -98,6 +98,47 @@ def match(proposal):
|
||||
return {"match": match_out, "candidates": candidates}
|
||||
|
||||
|
||||
def list_email_proposals():
|
||||
"""Pull the email-activity review work-lists for the poll loop: {to_post, open, to_close}.
|
||||
to_post = pending, un-posted (post a card); open = posted, awaiting a decision (rebuild the
|
||||
reply-routing map after a restart); to_close = decided on the web (announce in-thread + close)."""
|
||||
status, data = _authed("GET", "/api/intake/email-proposals")
|
||||
if status != 200:
|
||||
raise RuntimeError(f"email-proposals list failed ({status}): {data.get('error') or data}")
|
||||
payload = data.get("data") or {}
|
||||
return {k: (payload.get(k) or []) for k in ("to_post", "open", "to_close")}
|
||||
|
||||
|
||||
def mark_email_proposal_posted(proposal_id, event_id):
|
||||
"""Record the Matrix thread-root event id so the proposal's review state survives a restart."""
|
||||
status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/matrix",
|
||||
{"event_id": event_id})
|
||||
if status != 200:
|
||||
raise RuntimeError(f"mark posted failed ({status}): {data.get('error') or data}")
|
||||
return data.get("data") or {}
|
||||
|
||||
|
||||
def mark_email_proposal_closed(proposal_id):
|
||||
"""Mark the review thread resolved after announcing a web-side decision in it."""
|
||||
status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/matrix",
|
||||
{"closed": True})
|
||||
if status != 200:
|
||||
raise RuntimeError(f"mark closed failed ({status}): {data.get('error') or data}")
|
||||
return data.get("data") or {}
|
||||
|
||||
|
||||
def decide_email_proposal(proposal_id, decision, note=None):
|
||||
"""Relay an in-thread approve/dismiss (with the possibly-revised note) to the CRM. The server
|
||||
appends the note to the grid on approve, tags source='matrix', and closes the thread."""
|
||||
body = {"decision": decision}
|
||||
if note is not None:
|
||||
body["note"] = note
|
||||
status, data = _authed("POST", f"/api/intake/email-proposals/{proposal_id}/decide", body)
|
||||
if status not in (200, 201):
|
||||
raise RuntimeError(f"email-proposal decide failed ({status}): {data.get('error') or data}")
|
||||
return data.get("data") or {}
|
||||
|
||||
|
||||
def build_commit_payload(proposal):
|
||||
"""Pure: map a proposal to the /api/fundraising/log-communication request body.
|
||||
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Email-activity proposal review over Matrix — the CRM→Matrix leg of the email-capture flow.
|
||||
|
||||
The CRM (on the box) drafts a proposed grid note per newly-matched email (local model, no Claude)
|
||||
and queues it for human review. The CRM is stdlib-only and can't post to Matrix itself, so this
|
||||
bot PULLS the pending proposals (crm_client.list_email_proposals), posts a review card to the
|
||||
dedicated review room, and relays the human's in-thread reply back to the CRM. Same draft→approve
|
||||
discipline as the intake bot: nothing is appended to the grid until a human approves — here OR on
|
||||
the web Email Capture panel, the two surfaces kept in sync via the CRM's email_proposal_matrix row.
|
||||
|
||||
This module is the PURE logic (card rendering, reply grammar, note revision) so it's unit-tested
|
||||
offline; the async poll/post/reply wiring lives in bot.py (network + Matrix, live-smoke only).
|
||||
"""
|
||||
import spark
|
||||
|
||||
_YES = {"yes", "y", "approve", "approved", "ok", "confirm", "go", "add", "👍", "✅"}
|
||||
_NO = {"no", "n", "cancel", "discard", "reject", "skip", "stop", "👎", "❌"}
|
||||
|
||||
_SNIPPET_MAX = 400 # email snippet shown on the card; the full body is in the web popup
|
||||
|
||||
|
||||
def _truncate(s, n):
|
||||
s = (s or "").strip()
|
||||
return s if len(s) <= n else s[:n].rstrip() + "…"
|
||||
|
||||
|
||||
def render_card(item):
|
||||
"""The review card posted to the Matrix review room: who/when + a short email snippet + the
|
||||
drafted note. Deliberately compact for mobile — the full scrollable body is in the web Email
|
||||
Capture popup (this is the metadata+snippet+note choice)."""
|
||||
name = item.get("investor_name") or "Unknown investor"
|
||||
direction = "Sent" if item.get("direction") == "sent" else "Received"
|
||||
frm = item.get("from_name") or item.get("from_email") or "?"
|
||||
lines = [f"📧 Proposed **grid note** for **{name}** ({direction})"]
|
||||
if item.get("email_subject"):
|
||||
lines.append(f"· Subject: {item['email_subject']}")
|
||||
if item.get("email_date"):
|
||||
lines.append(f"· Date: {item['email_date']}")
|
||||
lines.append(f"· From: {frm}")
|
||||
snippet = _truncate(item.get("snippet"), _SNIPPET_MAX)
|
||||
if snippet:
|
||||
lines.append(f"· Email: {snippet}")
|
||||
lines.append("")
|
||||
lines.append(f"📝 Draft note: {item.get('proposed_note') or '(empty)'}")
|
||||
lines.append("")
|
||||
lines.append("Reply **yes** to add it to the grid, **no** to dismiss, or just tell me how to "
|
||||
"change the note (e.g. *say we discussed the Q3 raise*).")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def closure_line(status):
|
||||
"""Posted in-thread when a proposal was decided on the WEB while its Matrix thread was open."""
|
||||
verb = "approved ✅ and added to the grid" if status == "approved" else "dismissed 🗑️"
|
||||
return f"This was {verb} on the web — nothing more to do here. Thread closed."
|
||||
|
||||
|
||||
def interpret(text):
|
||||
"""Classify an in-thread reply: 'approve' | 'reject' | 'revise' (anything else → revise the note)."""
|
||||
t = (text or "").strip().lower()
|
||||
if t in _YES:
|
||||
return "approve"
|
||||
if t in _NO:
|
||||
return "reject"
|
||||
return "revise"
|
||||
|
||||
|
||||
REVISE_SYSTEM = (
|
||||
"You revise a single CRM note from a short instruction a venture-fund team member typed. "
|
||||
"You are given the CURRENT note and an INSTRUCTION. Apply the instruction and reply with "
|
||||
"ONLY a JSON object of the form {\"note\": \"<the full revised note>\"}. Keep it to one or two "
|
||||
"factual sentences, no preamble. Output JSON only."
|
||||
)
|
||||
|
||||
|
||||
def revise_note(note, instruction, parse_fn=spark.parse_json):
|
||||
"""Re-draft the note via local Qwen from a free-form instruction (no Claude, no scrub — same
|
||||
local-only basis as the intake parse). Returns the new note text, or None if the model gave
|
||||
nothing usable / unchanged, in which case the caller re-prompts. `parse_fn` is injectable for
|
||||
tests."""
|
||||
prompt = "CURRENT:\n" + (note or "") + "\n\nINSTRUCTION:\n" + (instruction or "").strip()
|
||||
raw = parse_fn(prompt, system=REVISE_SYSTEM, max_tokens=400) or {}
|
||||
new = raw.get("note") if isinstance(raw, dict) else None
|
||||
new = (new or "").strip()
|
||||
if not new or new == (note or "").strip():
|
||||
return None
|
||||
return new
|
||||
@@ -61,3 +61,10 @@ def crm_settings():
|
||||
# unset/empty just means no roster framing, i.e. the prior behavior.
|
||||
def team_roster():
|
||||
return [n.strip() for n in os.environ.get("INTAKE_TEAM_ROSTER", "").split(",") if n.strip()]
|
||||
|
||||
|
||||
# Dedicated room for reviewing CRM-drafted email-activity proposals (the CRM→Matrix push leg).
|
||||
# Separate from the intake room so high-volume email proposals don't drown the conversational
|
||||
# intake flow. Unset/empty disables the whole email-review poll loop (the bot just does intake).
|
||||
def email_review_room():
|
||||
return os.environ.get("MATRIX_EMAIL_REVIEW_ROOM", "").strip()
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
"""Offline tests for the email-proposal review logic (card render, reply grammar, note revision).
|
||||
The network/Matrix wiring lives in bot.py (live-smoke only); this covers the pure functions."""
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
import email_proposals # noqa: E402
|
||||
|
||||
ITEM = {
|
||||
"id": "p1", "investor_name": "Acme Capital", "direction": "received",
|
||||
"from_name": "Jane Doe", "from_email": "jane@acme.com",
|
||||
"email_subject": "Re: Fund III", "email_date": "2026-06-02",
|
||||
"snippet": "thanks for the deck — one question on terms", "proposed_note": "✉ Received: asked about terms",
|
||||
}
|
||||
|
||||
|
||||
def test_interpret_yes_no_else():
|
||||
assert email_proposals.interpret("yes") == "approve"
|
||||
assert email_proposals.interpret(" Y ") == "approve"
|
||||
assert email_proposals.interpret("✅") == "approve"
|
||||
assert email_proposals.interpret("no") == "reject"
|
||||
assert email_proposals.interpret("skip") == "reject"
|
||||
# anything that isn't a clear yes/no is treated as a revision instruction
|
||||
assert email_proposals.interpret("say we discussed the Q3 raise") == "revise"
|
||||
|
||||
|
||||
def test_render_card_has_context_note_and_actions():
|
||||
card = email_proposals.render_card(ITEM)
|
||||
assert "Acme Capital" in card and "Received" in card
|
||||
assert "Jane Doe" in card
|
||||
assert "Re: Fund III" in card and "2026-06-02" in card
|
||||
assert "thanks for the deck" in card
|
||||
assert "✉ Received: asked about terms" in card
|
||||
assert "yes" in card.lower() and "no" in card.lower()
|
||||
|
||||
|
||||
def test_render_card_sent_direction():
|
||||
assert "(Sent)" in email_proposals.render_card(dict(ITEM, direction="sent"))
|
||||
|
||||
|
||||
def test_render_card_truncates_long_snippet():
|
||||
card = email_proposals.render_card(dict(ITEM, snippet="x" * 1000))
|
||||
assert "…" in card and len(card) < 1000
|
||||
|
||||
|
||||
def test_revise_note_applies_model_output():
|
||||
out = email_proposals.revise_note(
|
||||
"old note", "make it about the Q3 raise",
|
||||
parse_fn=lambda prompt, system=None, max_tokens=400: {"note": "Discussed the Q3 raise."})
|
||||
assert out == "Discussed the Q3 raise."
|
||||
|
||||
|
||||
def test_revise_note_noop_or_empty_returns_none():
|
||||
# model echoes the same note unchanged -> None so the caller re-prompts (not "Updated")
|
||||
assert email_proposals.revise_note("same", "x", parse_fn=lambda *a, **k: {"note": "same"}) is None
|
||||
# model returns nothing usable -> None
|
||||
assert email_proposals.revise_note("n", "y", parse_fn=lambda *a, **k: {}) is None
|
||||
assert email_proposals.revise_note("n", "y", parse_fn=lambda *a, **k: None) is None
|
||||
|
||||
|
||||
def test_closure_line_reflects_status():
|
||||
assert "approved" in email_proposals.closure_line("approved").lower()
|
||||
assert "dismiss" in email_proposals.closure_line("dismissed").lower()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
fns = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
|
||||
for fn in fns:
|
||||
fn()
|
||||
print(f"ok {fn.__name__}")
|
||||
print(f"\n{len(fns)} passed")
|
||||
+153
-7
@@ -619,6 +619,14 @@ def parse_iso_utc(ts):
|
||||
def require_admin(user):
|
||||
return bool(user and user.get('role') == 'admin')
|
||||
|
||||
def require_bot_or_admin(user):
|
||||
"""Gate for agent/bot-facing endpoints: a dedicated 'bot' service account OR an admin
|
||||
(admins keep parity for debugging/curl). The 'bot' role is authenticated-but-not-admin —
|
||||
it never passes require_admin, so a bot credential cannot reach user-management, security,
|
||||
or settings. Reach (which endpoints) is controlled here; autonomy (acting without a human)
|
||||
stays governed by the per-feature draft->approve gate, independent of role."""
|
||||
return bool(user and user.get('role') in ('admin', 'bot'))
|
||||
|
||||
def log_audit(conn, user_id, entity_type, entity_id, action, changes=None):
|
||||
conn.execute(
|
||||
"INSERT INTO audit_log (id, user_id, entity_type, entity_id, action, changes) VALUES (?, ?, ?, ?, ?, ?)",
|
||||
@@ -2068,6 +2076,9 @@ class CRMHandler(BaseHTTPRequestHandler):
|
||||
# Matrix intake bot — new-vs-existing lookup for its in-thread proposal
|
||||
if path == '/api/intake/match':
|
||||
return self.handle_intake_match(user, params)
|
||||
# Matrix review bot — email-activity proposal work-lists (to_post/open/to_close)
|
||||
if path == '/api/intake/email-proposals':
|
||||
return self.handle_list_bot_email_proposals(user)
|
||||
|
||||
# Users
|
||||
if path == '/api/users':
|
||||
@@ -2187,6 +2198,10 @@ class CRMHandler(BaseHTTPRequestHandler):
|
||||
return self.handle_decide_activity_proposal(user, path.split('/')[-2], 'approve', body)
|
||||
if re.match(r'^/api/activity/proposals/[^/]+/dismiss$', path):
|
||||
return self.handle_decide_activity_proposal(user, path.split('/')[-2], 'dismiss', body)
|
||||
if re.match(r'^/api/intake/email-proposals/[^/]+/matrix$', path):
|
||||
return self.handle_mark_email_proposal_matrix(user, path.split('/')[-2], body)
|
||||
if re.match(r'^/api/intake/email-proposals/[^/]+/decide$', path):
|
||||
return self.handle_decide_email_proposal_matrix(user, path.split('/')[-2], body)
|
||||
if re.match(r'^/api/thesis/nodes/[^/]+/choose$', path):
|
||||
return self.handle_choose_variant(user, path.split('/')[-2])
|
||||
if re.match(r'^/api/thesis/lines/[^/]+/approve$', path):
|
||||
@@ -3964,6 +3979,59 @@ class CRMHandler(BaseHTTPRequestHandler):
|
||||
return self.send_error_json(res["error"], code)
|
||||
return self.send_json({"data": res})
|
||||
|
||||
# ─── Matrix review-bot bridge for email-activity proposals (bot-or-admin) ───
|
||||
def handle_list_bot_email_proposals(self, user):
|
||||
"""The bot's poll endpoint: {to_post, open, to_close}. Bot-or-admin (the proposals
|
||||
carry LP email content, so this stays off the member tier)."""
|
||||
if not require_bot_or_admin(user):
|
||||
return self.send_error_json("Bot or admin required", 403)
|
||||
conn = get_db()
|
||||
try:
|
||||
return self.send_json({"data": list_bot_email_proposals(conn)})
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def handle_mark_email_proposal_matrix(self, user, proposal_id, body):
|
||||
"""Record Matrix thread state: {event_id} marks the card posted; {closed:true} marks
|
||||
the thread resolved after the bot announced a web-side decision."""
|
||||
if not require_bot_or_admin(user):
|
||||
return self.send_error_json("Bot or admin required", 403)
|
||||
body = body or {}
|
||||
conn = get_db()
|
||||
try:
|
||||
if body.get("closed"):
|
||||
res = mark_proposal_matrix_closed(conn, proposal_id)
|
||||
else:
|
||||
event_id = str(body.get("event_id") or "").strip()
|
||||
if not event_id:
|
||||
return self.send_error_json("event_id or closed is required")
|
||||
res = mark_proposal_matrix_posted(conn, proposal_id, event_id)
|
||||
finally:
|
||||
conn.close()
|
||||
if res.get("error"):
|
||||
return self.send_error_json(res["error"], 404 if res["error"] == "not_found" else 400)
|
||||
return self.send_json({"data": res})
|
||||
|
||||
def handle_decide_email_proposal_matrix(self, user, proposal_id, body):
|
||||
"""In-thread Matrix decision relayed by the bot: approve/dismiss (+ optional edited note),
|
||||
tagged source='matrix' and closing the thread in the same transaction."""
|
||||
if not require_bot_or_admin(user):
|
||||
return self.send_error_json("Bot or admin required", 403)
|
||||
body = body or {}
|
||||
decision = str(body.get("decision") or "").strip()
|
||||
if decision not in ("approve", "dismiss"):
|
||||
return self.send_error_json("decision must be approve or dismiss")
|
||||
conn = get_db()
|
||||
try:
|
||||
res = decide_email_activity_proposal(conn, proposal_id, decision, user['user_id'],
|
||||
body.get('note'), source="matrix", close_matrix=True)
|
||||
finally:
|
||||
conn.close()
|
||||
if res.get("error"):
|
||||
code = {"not_found": 404, "already_decided": 409}.get(res["error"], 400)
|
||||
return self.send_error_json(res["error"], code)
|
||||
return self.send_json({"data": res})
|
||||
|
||||
# ─── UI-triggered index jobs + entity-merge review (Phase 1) ───
|
||||
def handle_index_job(self, user, kind):
|
||||
if not require_admin(user):
|
||||
@@ -4357,8 +4425,10 @@ class CRMHandler(BaseHTTPRequestHandler):
|
||||
return self.send_error_json("password must be at least 8 characters")
|
||||
|
||||
role = body.get('role', 'member')
|
||||
if role not in ('admin', 'member'):
|
||||
return self.send_error_json("role must be admin or member")
|
||||
# 'bot' is a deliberately-provisioned agent service account (kept out of the invite UI's
|
||||
# member/admin dropdown) — authenticated but never an admin. See require_bot_or_admin.
|
||||
if role not in ('admin', 'member', 'bot'):
|
||||
return self.send_error_json("role must be admin, member, or bot")
|
||||
|
||||
conn = get_db()
|
||||
existing = conn.execute(
|
||||
@@ -4417,9 +4487,9 @@ class CRMHandler(BaseHTTPRequestHandler):
|
||||
|
||||
if 'role' in body:
|
||||
role = str(body.get('role'))
|
||||
if role not in ('admin', 'member'):
|
||||
if role not in ('admin', 'member', 'bot'):
|
||||
conn.close()
|
||||
return self.send_error_json("role must be admin or member")
|
||||
return self.send_error_json("role must be admin, member, or bot")
|
||||
sets.append("role = ?")
|
||||
args.append(role)
|
||||
|
||||
@@ -5728,8 +5798,14 @@ def list_email_activity_proposals(conn, status="pending", limit=200):
|
||||
return []
|
||||
|
||||
|
||||
def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_note=None):
|
||||
"""Approve (optionally with an edited note -> append to grid) or dismiss a proposal."""
|
||||
def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_note=None,
|
||||
source="crm_ui", close_matrix=False):
|
||||
"""Approve (optionally with an edited note -> append to grid) or dismiss a proposal.
|
||||
|
||||
`source` records the channel in the audit log ('crm_ui' for the web panel, 'matrix' for an
|
||||
in-thread approval relayed by the review bot). `close_matrix` is set by the bot when the
|
||||
decision was made in-thread: it stamps the Matrix side row closed in the same transaction so
|
||||
the web->Matrix close path (list_bot_email_proposals.to_close) won't re-announce it."""
|
||||
p = conn.execute("SELECT * FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone()
|
||||
if not p:
|
||||
return {"error": "not_found"}
|
||||
@@ -5747,15 +5823,85 @@ def decide_email_activity_proposal(conn, proposal_id, decision, user_id, edited_
|
||||
action, result = "email.activity_dismissed", {"status": "dismissed"}
|
||||
else:
|
||||
return {"error": "bad_decision"}
|
||||
if close_matrix:
|
||||
_mark_proposal_matrix(conn, proposal_id, closed_at=now())
|
||||
conn.execute(
|
||||
"INSERT INTO interaction_log (id, ts, actor_type, actor_id, action, target_type, target_id, payload, source, created_at) "
|
||||
"VALUES (?,?,?,?,?,?,?,?,?,?)",
|
||||
(generate_id(), now(), "human", user_id, action, "fundraising_investor", p["investor_id"],
|
||||
json.dumps({"proposal_id": proposal_id}), "crm_ui", now()))
|
||||
json.dumps({"proposal_id": proposal_id}), source, now()))
|
||||
conn.commit()
|
||||
return result
|
||||
|
||||
|
||||
# ─── Matrix review-bot bridge for email-activity proposals (Features 2/3) ──────
|
||||
# The CRM (box, stdlib, no matrix-nio) can't post to Matrix, so the intake bot (Spark) PULLS
|
||||
# pending proposals via list_bot_email_proposals, posts a review card to the dedicated review
|
||||
# room, and writes the thread-root event_id back here. State lives CRM-side (email_proposal_matrix)
|
||||
# so both surfaces stay in sync and it survives a bot restart. All queries degrade to empty when
|
||||
# the email integration tables are absent (OperationalError), mirroring list_email_activity_proposals.
|
||||
_BOT_PROPOSAL_COLS = (
|
||||
"SELECT p.id, p.investor_name, p.direction, p.summary, p.proposed_note, "
|
||||
"p.email_subject, p.email_date, e.from_name, e.from_email, e.snippet, "
|
||||
"m.event_id AS event_id, p.status AS status "
|
||||
"FROM email_activity_proposals p "
|
||||
"LEFT JOIN email_proposal_matrix m ON m.proposal_id = p.id "
|
||||
"LEFT JOIN emails e ON e.id = p.email_id ")
|
||||
|
||||
|
||||
def list_bot_email_proposals(conn, limit=100):
|
||||
"""The three work-lists the Matrix review bot polls:
|
||||
to_post — pending, not yet posted to Matrix -> bot posts a review card.
|
||||
open — pending, posted, not closed -> live threads; the bot rebuilds its
|
||||
event_id->proposal routing map from these after a restart.
|
||||
to_close — decided on the WEB while a thread is open -> bot announces it in-thread, closes.
|
||||
Each item carries the card content (investor, direction, subject, date, from, snippet, note)."""
|
||||
try:
|
||||
to_post = [dict(r) for r in conn.execute(
|
||||
_BOT_PROPOSAL_COLS + "WHERE p.status='pending' AND (m.proposal_id IS NULL OR m.posted_at IS NULL) "
|
||||
"ORDER BY p.email_date ASC, p.created_at ASC LIMIT ?", (limit,))]
|
||||
open_threads = [dict(r) for r in conn.execute(
|
||||
_BOT_PROPOSAL_COLS + "WHERE p.status='pending' AND m.posted_at IS NOT NULL AND m.closed_at IS NULL "
|
||||
"ORDER BY p.email_date ASC, p.created_at ASC LIMIT ?", (limit,))]
|
||||
to_close = [dict(r) for r in conn.execute(
|
||||
_BOT_PROPOSAL_COLS + "WHERE p.status!='pending' AND m.posted_at IS NOT NULL AND m.closed_at IS NULL "
|
||||
"ORDER BY p.decided_at ASC LIMIT ?", (limit,))]
|
||||
except sqlite3.OperationalError:
|
||||
return {"to_post": [], "open": [], "to_close": []}
|
||||
return {"to_post": to_post, "open": open_threads, "to_close": to_close}
|
||||
|
||||
|
||||
def _mark_proposal_matrix(conn, proposal_id, *, event_id=None, posted_at=None, closed_at=None):
|
||||
"""Idempotent upsert of the 1:1 Matrix side row. Only the passed fields are written."""
|
||||
cols, vals, sets = ["proposal_id"], [proposal_id], []
|
||||
for name, val in (("event_id", event_id), ("posted_at", posted_at), ("closed_at", closed_at)):
|
||||
if val is not None:
|
||||
cols.append(name); vals.append(val); sets.append(f"{name}=excluded.{name}")
|
||||
placeholders = ",".join("?" for _ in cols)
|
||||
sql = f"INSERT INTO email_proposal_matrix ({','.join(cols)}) VALUES ({placeholders})"
|
||||
if sets:
|
||||
sql += " ON CONFLICT(proposal_id) DO UPDATE SET " + ",".join(sets)
|
||||
conn.execute(sql, vals)
|
||||
|
||||
|
||||
def mark_proposal_matrix_posted(conn, proposal_id, event_id):
|
||||
"""Record that the bot posted a review card (thread root = event_id)."""
|
||||
if not conn.execute("SELECT 1 FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone():
|
||||
return {"error": "not_found"}
|
||||
_mark_proposal_matrix(conn, proposal_id, event_id=event_id, posted_at=now())
|
||||
conn.commit()
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
def mark_proposal_matrix_closed(conn, proposal_id):
|
||||
"""Mark the Matrix review thread resolved (the bot announced a web-side decision)."""
|
||||
if not conn.execute("SELECT 1 FROM email_activity_proposals WHERE id=?", (proposal_id,)).fetchone():
|
||||
return {"error": "not_found"}
|
||||
_mark_proposal_matrix(conn, proposal_id, closed_at=now())
|
||||
conn.commit()
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
# ─── Main Entry Point ────────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Test the Matrix review-bot bridge for email-activity proposals (Features 2/3):
|
||||
the bot work-lists (to_post / open / to_close), the Matrix side-row mark helpers, and an
|
||||
in-thread (source='matrix') decision that closes the thread — plus the bot-or-admin role gate.
|
||||
Synthetic data only (guardrail #9). The local model is stubbed.
|
||||
Run: cd backend && python3 test_email_proposal_matrix.py
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
os.environ["CRM_DB_PATH"] = os.path.join(tempfile.mkdtemp(), "crm.db")
|
||||
os.environ.setdefault("CRM_DATA_DIR", os.path.dirname(os.environ["CRM_DB_PATH"]))
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import server # noqa: E402
|
||||
|
||||
server._summarize_email_gist = lambda subject, body: "fundraising update; proposed a call"
|
||||
|
||||
FAILS = []
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
print((" PASS " if cond else " FAIL ") + msg)
|
||||
if not cond:
|
||||
FAILS.append(msg)
|
||||
|
||||
|
||||
def setup():
|
||||
conn = sqlite3.connect(os.environ["CRM_DB_PATH"])
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.executescript("""
|
||||
CREATE TABLE app_settings (key TEXT PRIMARY KEY, value_json TEXT, updated_at TEXT);
|
||||
CREATE TABLE email_accounts (id TEXT, email_address TEXT, sync_enabled INT DEFAULT 1, sync_status TEXT, backfill_complete INT);
|
||||
CREATE TABLE emails (id TEXT PRIMARY KEY, subject TEXT, body_text TEXT, snippet TEXT, from_name TEXT, from_email TEXT, sent_at TEXT, is_matched INT, match_status TEXT);
|
||||
CREATE TABLE email_investor_links (id TEXT, email_id TEXT, fundraising_investor_id TEXT, organization_id TEXT, contact_id TEXT, match_confidence REAL);
|
||||
CREATE TABLE email_activity_proposals (id TEXT PRIMARY KEY, email_id TEXT UNIQUE, investor_id TEXT, investor_name TEXT,
|
||||
direction TEXT, summary TEXT, proposed_note TEXT, email_subject TEXT, email_date TEXT, status TEXT DEFAULT 'pending',
|
||||
decided_by TEXT, decided_at TEXT, final_note TEXT, created_at TEXT);
|
||||
CREATE TABLE email_proposal_matrix (proposal_id TEXT PRIMARY KEY, event_id TEXT, posted_at TEXT, closed_at TEXT, created_at TEXT);
|
||||
CREATE TABLE users (id TEXT PRIMARY KEY, username TEXT);
|
||||
CREATE TABLE fundraising_investors (id TEXT PRIMARY KEY, investor_name TEXT, notes TEXT);
|
||||
CREATE TABLE fundraising_state (id TEXT PRIMARY KEY, grid_json TEXT, views_json TEXT, version INT,
|
||||
updated_by TEXT REFERENCES users(id), updated_at TEXT);
|
||||
CREATE TABLE interaction_log (id TEXT PRIMARY KEY, ts TEXT, actor_type TEXT, actor_id TEXT, action TEXT, target_type TEXT, target_id TEXT, payload TEXT, source TEXT, created_at TEXT);
|
||||
""")
|
||||
conn.execute("INSERT INTO users (id,username) VALUES ('user-1','grant')")
|
||||
conn.execute("INSERT INTO app_settings VALUES ('email_activity_since', ?, ?)", (json.dumps("2026-01-01T00:00:00"), "x"))
|
||||
conn.execute("INSERT INTO email_accounts (id,email_address) VALUES ('a','grant@ten31.xyz')")
|
||||
conn.execute("INSERT INTO fundraising_investors (id,investor_name,notes) VALUES ('inv1','Harbor & Vine','existing note')")
|
||||
grid = {"columns": [], "rows": [{"id": "inv1", "investor_name": "Harbor & Vine", "notes": "existing note"}]}
|
||||
conn.execute("INSERT INTO fundraising_state (id,grid_json,views_json,version) VALUES ('main',?,?,1)", (json.dumps(grid), "[]"))
|
||||
conn.executemany("INSERT INTO emails (id,subject,body_text,snippet,from_name,from_email,sent_at,is_matched,match_status) VALUES (?,?,?,?,?,?,?,1,'matched')", [
|
||||
("e1", "Fund III", "Here is the update", "the quarterly update is attached", "Grant", "grant@ten31.xyz", "2026-06-01T10:00:00"),
|
||||
("e2", "Re: Fund III", "Thanks, a question", "thanks — one question on terms", "LP Contact", "lp@harborvine.example", "2026-06-02T10:00:00"),
|
||||
])
|
||||
conn.executemany("INSERT INTO email_investor_links (id,email_id,fundraising_investor_id,match_confidence) VALUES (?,?, 'inv1', 1.0)",
|
||||
[("l1", "e1"), ("l2", "e2")])
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
|
||||
def main():
|
||||
setup()
|
||||
|
||||
# role gate: bot passes the agent gate but is NOT an admin; member passes neither.
|
||||
check(server.require_bot_or_admin({"role": "bot"}), "bot passes require_bot_or_admin")
|
||||
check(server.require_bot_or_admin({"role": "admin"}), "admin passes require_bot_or_admin")
|
||||
check(not server.require_bot_or_admin({"role": "member"}), "member does NOT pass require_bot_or_admin")
|
||||
check(not server.require_admin({"role": "bot"}), "bot is NOT an admin (no user-mgmt/settings reach)")
|
||||
|
||||
check(server.propose_email_activity_notes().get("proposed") == 2, "drafts 2 proposals")
|
||||
conn = server.get_db()
|
||||
props = server.list_email_activity_proposals(conn, status="pending")
|
||||
by_email = {p["email_id"]: p for p in props}
|
||||
p_a, p_b = by_email["e1"], by_email["e2"]
|
||||
|
||||
# Both are pending + un-posted → both in to_post; card carries from/snippet/note context.
|
||||
lists = server.list_bot_email_proposals(conn)
|
||||
check(len(lists["to_post"]) == 2 and not lists["open"] and not lists["to_close"], "both proposals queued to_post")
|
||||
card = next(it for it in lists["to_post"] if it["id"] == p_a["id"])
|
||||
check(card.get("from_name") == "Grant" and "quarterly update" in (card.get("snippet") or ""), "card carries from_name + snippet")
|
||||
check("✉" in (card.get("proposed_note") or ""), "card carries the drafted note")
|
||||
|
||||
# Post p_a to Matrix → it leaves to_post and becomes an open thread (event id recorded).
|
||||
server.mark_proposal_matrix_posted(conn, p_a["id"], "evtA")
|
||||
lists = server.list_bot_email_proposals(conn)
|
||||
check(len(lists["to_post"]) == 1 and lists["to_post"][0]["id"] == p_b["id"], "posting p_a leaves only p_b to_post")
|
||||
check(len(lists["open"]) == 1 and lists["open"][0]["id"] == p_a["id"] and lists["open"][0]["event_id"] == "evtA",
|
||||
"posted p_a is an open thread carrying its event id")
|
||||
|
||||
# Decide p_a IN-THREAD on Matrix (approve + close in one transaction).
|
||||
r = server.decide_email_activity_proposal(conn, p_a["id"], "approve", "user-1", source="matrix", close_matrix=True)
|
||||
check(r.get("status") == "approved" and r.get("placed_in_grid") is True, "matrix approve appends to the grid")
|
||||
lists = server.list_bot_email_proposals(conn)
|
||||
check(not any(it["id"] == p_a["id"] for it in lists["open"] + lists["to_close"]),
|
||||
"matrix-decided proposal is closed (not re-announced via to_close)")
|
||||
src = conn.execute("SELECT source FROM interaction_log WHERE action='email.activity_approved'").fetchone()["source"]
|
||||
check(src == "matrix", "matrix decision is audited source='matrix'")
|
||||
|
||||
# Web-decide path: post p_b, then dismiss it on the WEB (default source, no close) → the bot
|
||||
# must see it in to_close so it can announce the web decision in-thread, then close.
|
||||
server.mark_proposal_matrix_posted(conn, p_b["id"], "evtB")
|
||||
server.decide_email_activity_proposal(conn, p_b["id"], "dismiss", "user-1") # web path: source crm_ui, no close
|
||||
lists = server.list_bot_email_proposals(conn)
|
||||
check(len(lists["to_close"]) == 1 and lists["to_close"][0]["id"] == p_b["id"] and lists["to_close"][0]["status"] == "dismissed",
|
||||
"web-decided open thread surfaces in to_close")
|
||||
src2 = conn.execute("SELECT source FROM interaction_log WHERE action='email.activity_dismissed'").fetchone()["source"]
|
||||
check(src2 == "crm_ui", "web decision is audited source='crm_ui'")
|
||||
|
||||
server.mark_proposal_matrix_closed(conn, p_b["id"])
|
||||
lists = server.list_bot_email_proposals(conn)
|
||||
check(not lists["to_close"] and not lists["open"], "closing the thread clears the work-lists")
|
||||
|
||||
# Marking a non-existent proposal is a clean not_found, not a crash.
|
||||
check(server.mark_proposal_matrix_posted(conn, "nope", "evtX").get("error") == "not_found", "mark posted on unknown id -> not_found")
|
||||
conn.close()
|
||||
|
||||
if FAILS:
|
||||
print(f"\nFAILED ({len(FAILS)})")
|
||||
for f in FAILS:
|
||||
print(" - " + f)
|
||||
sys.exit(1)
|
||||
print("\nALL PASS (email-proposal Matrix bridge)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user