ten31-database/docs/guides/matrix-intake.md

paths

paths
backend/matrix_intake/**

Matrix intake bot

Read this before editing backend/matrix_intake/. The bot turns a typed message in a dedicated Matrix room into a proposed fundraising-grid add/edit, gated on in-thread human approval before any write. Phase status: M1 + M2 deployed & live (text intake + approval + write; bot on the Spark, CRM endpoints on the box at v0.1.0:85; live-smoked 2026-06-17). M3 (business-card photo) deferred — Spark Control has no vision model yet.

Post-deploy UX pass — BUILT, not yet deployed (2026-06-17): fuzzy investor matching (server-side, v0.1.0:86 — needs s9pk build+install) + in-thread disambiguation and conversational natural-language edits (bot-side — needs a Spark git pull + restart). See Fuzzy matching below. Tests green (27/27 backend + the offline bot suite); not yet live-smoked — the disambiguation grammar and the Qwen revise leg need a Matrix smoke.

What it is (and isn't)

• A separate process, not part of the CRM. Its only third-party dep, matrix-nio, lives in backend/matrix_intake/requirements.txt and must never be added to the stdlib CRM (backend/server.py). Runs on the Spark (placement per standards/guides/placement.md).
• It drafts; a human approves. Nothing is written autonomously — every CRM write follows a yes reply in the proposal thread. This is exempt from "agents draft, humans send" the same way the digest is: it's internal data entry to our own CRM, not outward LP contact.
• It is not a parallel write path. It reuses the CRM's own canonical endpoint POST /api/fundraising/log-communication (create-if-missing + contact upsert + note + relational sync + audit) for both new-investor and existing-note cases. Don't reimplement grid mutation in the bot.

Flow

1. Top-level message in the intake room → parse.parse_message → local Qwen via Spark Control (spark.py reuses backend/ingest/llm.py; temp 0, JSON only) extracts {intent, investor_name, contact_name, contact_email, contact_title, note}. The original message text is stashed on the proposal as _source_text (needed later for revise's email-integrity check).
2. crm_client.match (GET /api/intake/match) resolves new-vs-existing. It returns both an exact match (returns the grid row id so an approved note lands on exactly that investor, no duplicate) and, when there's no exact match, a ranked list of fuzzy candidates (see Fuzzy matching below).
3. Three outcomes drive what gets posted, all in a thread rooted at the user's message, plus a brief main-timeline nudge (a plain reply — matrix_io.make_reply) so it isn't missed:
   • Exact match → auto-attach: proposal flips to meeting_note with _match_id set, rendered as the normal approval card.
   • Fuzzy candidates, no exact → a disambiguation card (proposals.render_disambiguation): the proposal is held at _stage="disambiguate" with _candidates, and the human must pick a number / new / no before it becomes an approval-stage proposal.
   • Neither → the new-investor approval card. The nudge is a pointer only, not a reply target — you need the thread to act. The pending proposal is held in memory keyed by the thread root (proposals.ProposalStore).
4. User replies in the thread. handle_reply branches on _stage:
   • disambiguate (handle_disambiguation): a number attaches to that candidate (→ meeting_note
     • _match_id, re-rendered for approval); new proceeds as a new investor; no discards.
   • approval: yes commits; no discards; edit field=value is the deterministic fast-path edit; anything else is treated as a natural-language revision — parse.revise sends {current proposal + instruction} back through local Qwen and re-renders the revised card (a no-op revision is detected via proposals.same_fields and re-prompts instead of saying "Updated"). On yes, crm_client.commit POSTs to log-communication tagged source="matrix_intake" (provenance in the audit log). A bare yes/no typed top-level (not in the thread) while a proposal is pending gets a "reply in the thread" redirect (store.any_pending() guard in handle_intake), not a misparsed new intake.

Fuzzy matching (server-side, ships in the s9pk)

GET /api/intake/match returns {match, candidates}. find_intake_match is unchanged — exact-after-normalization, and an exact match still auto-attaches without disambiguation. find_intake_candidates (new) is the fuzzy layer, deterministic, no LLM: it scans the same canonical grid blob and scores each row by max(name similarity, email near-match), keeping rows ≥ min_score (0.62), ranked, capped at 5:

• Name (_name_similarity): max of stdlib difflib sequence ratio (near-spellings — "Charlie"/"Charles") and token-set Jaccard (word-order). Legal-entity suffixes (LLC/LP/Inc/… via _strip_legal_suffix) are stripped first, so "Acme Capital" ~ "Acme Capital LLC" scores 1.0 (a near-certain duplicate find_intake_match misses because it compares the full string) — and is surfaced as a candidate, never auto-attached (the human still confirms).
• Email (_email_edit_distance): Levenshtein ≤ 2 against each contact email (dist 1→0.9, 2→0.8). Distance 0 is an exact email — that's find_intake_match's job, skipped here.
• Recall-favoring by design: a shared common name-word ("… Capital") can lift an unrelated firm into the 0.6–0.8 band. Acceptable — it's a ranked, human-confirmed shortlist, and the cost of an occasional stray suggestion is far lower than missing a real near-duplicate. Semantic pruning of the shortlist (the "Charlie really is Charles" judgment) is a deferred LLM-judge re-rank — fed only the shortlist, never the whole LP list — intentionally NOT built in this pass, because the deterministic filter already surfaces every duplicate the human then resolves.

Rules / gotchas

• Module-name collision: the intake config module is settings.py, not config.py, because backend/ingest/config.py is imported (as bare config) through spark → llm. A second config module would shadow it in sys.modules and break llm (CHAT_MODEL). Keep intake module names from colliding with ingest's (config, http_util, llm).
• Email integrity: parse.normalize only keeps an address that literally appears in the source message — the model must never mint one (a wrong email is worse than none). It takes the first address in the text, so a two-person message ("Alice a@x.com and Bob b@y.com") could attach the wrong one; the human sees it in the proposal and can edit email=… before approving. Cross-referencing multiple addresses to the named contact is a deliberate non-goal for v1.
• Conversational revise keeps the email rule: parse.revise re-runs a free-form correction through Qwen but never trusts the model's email field. A changed address is accepted only if it literally appears in the instruction text (searched first), else the existing integrity-checked address is kept (_apply_revision). The model can edit name/contact/title/note freely but cannot mint an email. A revision that nulls both investor and contact is rejected (the proposal can't be emptied to something unactionable). Revise edits fields on the current proposal; it does not re-run the matcher if you rename the firm mid-thread (a known v1 limit — the human still approves).
• Deploy is split across two surfaces (mind which one carries a change): the fuzzy candidates come from server.py → ship in the s9pk (build + install, version-bumped). The bot's disambiguation flow + revise live in backend/matrix_intake/ → ship on the Spark via git pull + restart. A bot restart alone won't deliver candidates (the box would return an empty list and the bot just proposes new — safe, but no fuzzy surfacing until the s9pk is installed). Same lesson as the v83→v84 /api/intake/match 404.
• Double-approve guard: handle_reply pops the pending proposal from the store before awaiting the commit, so a second yes arriving mid-write is a no-op (asyncio is cooperative; the pop is atomic w.r.t. other events). On commit failure the proposal is restored for retry. Known minor: in the disambiguate stage the pick re-stores an approval-stage proposal before its await say, so a rapidly-repeated 1 can have the second one fall through to the NL-revise path (a wasted Spark round-trip that re-prompts) — harmless, nothing commits, not guarded (low likelihood on a ~5-person team).
• Local-only parse: intake text is real LP substance but goes ONLY to local Qwen via Spark Control, never Claude — so no scrub boundary applies (same basis as the digest). Never call a Spark directly; always go through SPARK_CONTROL_URL.
• Auth: the CRM has no service-key path; the bot logs in as a dedicated CRM user (CRM_BOT_USERNAME/CRM_BOT_PASSWORD) → Bearer JWT, re-login once on 401.
• Tests are offline: test_parse.py / test_proposals.py / test_crm_client.py stub the network; backend/test_intake_endpoints.py boots the real server against a temp DB and covers /api/intake/match + the create→match (no-duplicate) contract + provenance. A live Matrix smoke needs creds + matrix-nio installed on the Spark — it can't run in CI.
• Grid note line: the bot sends a blank subject when there's a note so the CRM's one-line note summary shows the note text (the CRM renders subject-or-body); a provenance label is sent only when there's no note. v0.1.0:85 also dropped the redundant [note] type tag from that server-side line (informative types like [call] keep theirs).

Deployment & ops

• Runs on the Spark (SSH alias modelo32, host spark-32d0): repo at /home/modelo/ten31-database, deps in a venv (.venv; only matrix-nio). Launched detached: nohup ./.venv/bin/python backend/matrix_intake/bot.py >/tmp/intake-bot.log 2>&1 &, pid in /tmp/intake-bot.pid; startup logs listening as … in room ….
• Restart after a git pull of bot code: kill $(cat /tmp/intake-bot.pid), relaunch as above, re-write the pid. A restart drops in-memory pending proposals (re-send to recover).
• NOT a managed service yet — won't survive a Spark reboot; restart-on-boot (systemd) is an open TODO.
• Server-side endpoints ship in the s9pk, not the bot. GET /api/intake/match and the source provenance on log-communication live in backend/server.py, so they reach the box only via an s9pk build + install — a bot restart won't deliver them. (Missed in v83: the box 404'd /api/intake/match until v0.1.0:84.)
• CRM_API_BASE is the box over the LAN, not localhost (bot on the Spark, CRM on the box). https://immense-voyage.local (443) is the StartOS dashboard, not the CRM — the CRM has its own interface address (the URL you open in a browser); container port 8080 isn't LAN-reachable.

Config

All in .env (names in .env.example): MATRIX_HOMESERVER, MATRIX_USER, MATRIX_ACCESS_TOKEN, MATRIX_DEVICE_ID, MATRIX_INTAKE_ROOM; CRM_API_BASE, CRM_BOT_USERNAME, CRM_BOT_PASSWORD, CRM_API_VERIFY_TLS. Spark settings are inherited from the ingest client (SPARK_CONTROL_URL, CRM_CHAT_MODEL).