Keysat
0b893295e1
Close the two locked post-deploy enhancements for the Matrix intake bot.
Fuzzy matching (server-side, ships in the s9pk): new find_intake_candidates in
server.py returns ranked deterministic near-matches (difflib name similarity +
token-set Jaccard, legal-suffix-aware, + email Levenshtein <= 2); GET
/api/intake/match now returns {match, candidates}. The bot surfaces a numbered
shortlist so a near-duplicate (Charlie/Charles, Acme Capital vs Acme Capital LLC,
a one-char email typo) is confirmed by a human instead of silently creating a
second investor. Exact match still auto-attaches; fuzzy candidates are never
auto-attached. The optional LLM-judge re-rank is deferred.
Conversational edits (bot-side, ships on the Spark): any in-thread reply that
isn't yes/no/edit field=value is treated as a natural-language revision and
re-run through local Qwen (parse.revise). Email integrity is preserved -- a
changed address must literally appear in the instruction; the model's email
field is structurally unreachable. No-op revisions re-prompt.
Docs/current-state brought current; 27/27 backend tests green.
160 lines
11 KiB
Markdown
160 lines
11 KiB
Markdown
---
|
||
paths:
|
||
- backend/matrix_intake/**
|
||
---
|
||
|
||
# Matrix intake bot
|
||
|
||
Read this before editing `backend/matrix_intake/`. The bot turns a typed message in a
|
||
dedicated Matrix room into a proposed fundraising-grid add/edit, gated on **in-thread human
|
||
approval** before any write. Phase status: **M1 + M2 deployed & live** (text intake + approval + write; bot on the Spark,
|
||
CRM endpoints on the box at **v0.1.0:85**; live-smoked 2026-06-17). **M3 (business-card photo)
|
||
deferred** — Spark Control has no vision model yet.
|
||
|
||
**Post-deploy UX pass — BUILT, not yet deployed (2026-06-17):** fuzzy investor matching
|
||
(server-side, **v0.1.0:86** — needs s9pk build+install) + in-thread disambiguation and
|
||
conversational natural-language edits (bot-side — needs a Spark `git pull` + restart). See
|
||
*Fuzzy matching* below. Tests green (27/27 backend + the offline bot suite); **not yet
|
||
live-smoked** — the disambiguation grammar and the Qwen `revise` leg need a Matrix smoke.
|
||
|
||
## What it is (and isn't)
|
||
|
||
- A **separate process**, not part of the CRM. Its only third-party dep, `matrix-nio`, lives
|
||
in `backend/matrix_intake/requirements.txt` and **must never** be added to the stdlib CRM
|
||
(`backend/server.py`). Runs on the Spark (placement per `standards/guides/placement.md`).
|
||
- It **drafts; a human approves.** Nothing is written autonomously — every CRM write follows a
|
||
`yes` reply in the proposal thread. This is exempt from "agents draft, humans send" the same
|
||
way the digest is: it's internal data entry to our own CRM, not outward LP contact.
|
||
- It is **not** a parallel write path. It reuses the CRM's own canonical endpoint
|
||
`POST /api/fundraising/log-communication` (create-if-missing + contact upsert + note +
|
||
relational sync + audit) for both new-investor and existing-note cases. Don't reimplement
|
||
grid mutation in the bot.
|
||
|
||
## Flow
|
||
|
||
1. Top-level message in the intake room → `parse.parse_message` → local **Qwen via Spark
|
||
Control** (`spark.py` reuses `backend/ingest/llm.py`; temp 0, JSON only) extracts
|
||
`{intent, investor_name, contact_name, contact_email, contact_title, note}`. The original
|
||
message text is stashed on the proposal as `_source_text` (needed later for `revise`'s
|
||
email-integrity check).
|
||
2. `crm_client.match` (`GET /api/intake/match`) resolves new-vs-existing. It returns **both** an
|
||
exact `match` (returns the **grid row id** so an approved note lands on exactly that investor,
|
||
no duplicate) **and**, when there's no exact match, a ranked list of fuzzy `candidates` (see
|
||
*Fuzzy matching* below).
|
||
3. Three outcomes drive what gets posted, all **in a thread** rooted at the user's message, plus a
|
||
brief **main-timeline nudge** (a plain reply — `matrix_io.make_reply`) so it isn't missed:
|
||
- **Exact match** → auto-attach: proposal flips to `meeting_note` with `_match_id` set, rendered
|
||
as the normal approval card.
|
||
- **Fuzzy candidates, no exact** → a **disambiguation** card (`proposals.render_disambiguation`):
|
||
the proposal is held at `_stage="disambiguate"` with `_candidates`, and the human must pick a
|
||
**number** / `new` / `no` before it becomes an approval-stage proposal.
|
||
- **Neither** → the new-investor approval card.
|
||
The nudge is a **pointer only, not a reply target** — you need the thread to act. The pending
|
||
proposal is held in memory keyed by the thread root (`proposals.ProposalStore`).
|
||
4. User replies **in the thread**. `handle_reply` branches on `_stage`:
|
||
- **disambiguate** (`handle_disambiguation`): a number attaches to that candidate (→ `meeting_note`
|
||
+ `_match_id`, re-rendered for approval); `new` proceeds as a new investor; `no` discards.
|
||
- **approval**: `yes` commits; `no` discards; `edit field=value` is the deterministic fast-path
|
||
edit; **anything else is treated as a natural-language revision** — `parse.revise` sends
|
||
`{current proposal + instruction}` back through local Qwen and re-renders the revised card (a
|
||
no-op revision is detected via `proposals.same_fields` and re-prompts instead of saying
|
||
"Updated"). On `yes`, `crm_client.commit` POSTs to `log-communication` tagged
|
||
`source="matrix_intake"` (provenance in the audit log).
|
||
A bare `yes`/`no` typed **top-level** (not in the thread) while a proposal is pending gets a
|
||
"reply in the thread" redirect (`store.any_pending()` guard in `handle_intake`), not a
|
||
misparsed new intake.
|
||
|
||
## Fuzzy matching (server-side, ships in the s9pk)
|
||
|
||
`GET /api/intake/match` returns `{match, candidates}`. `find_intake_match` is unchanged —
|
||
**exact-after-normalization**, and an exact match still auto-attaches without disambiguation.
|
||
`find_intake_candidates` (new) is the fuzzy layer, **deterministic, no LLM**: it scans the same
|
||
canonical grid blob and scores each row by `max(`name similarity`, `email near-match`)`, keeping
|
||
rows ≥ `min_score` (0.62), ranked, capped at 5:
|
||
- **Name** (`_name_similarity`): max of stdlib `difflib` sequence ratio (near-spellings —
|
||
"Charlie"/"Charles") and token-set Jaccard (word-order). **Legal-entity suffixes**
|
||
(LLC/LP/Inc/… via `_strip_legal_suffix`) are stripped first, so "Acme Capital" ~ "Acme Capital
|
||
LLC" scores 1.0 (a near-certain duplicate `find_intake_match` misses because it compares the
|
||
full string) — and is surfaced as a candidate, **never auto-attached** (the human still confirms).
|
||
- **Email** (`_email_edit_distance`): Levenshtein ≤ 2 against each contact email (dist 1→0.9,
|
||
2→0.8). Distance 0 is an exact email — that's `find_intake_match`'s job, skipped here.
|
||
- **Recall-favoring by design:** a shared common name-word ("… Capital") can lift an unrelated firm
|
||
into the 0.6–0.8 band. Acceptable — it's a *ranked, human-confirmed* shortlist, and the cost of an
|
||
occasional stray suggestion is far lower than missing a real near-duplicate. **Semantic pruning of
|
||
the shortlist (the "Charlie really is Charles" judgment) is a deferred LLM-judge re-rank** — fed
|
||
only the shortlist, never the whole LP list — intentionally NOT built in this pass, because the
|
||
deterministic filter already surfaces every duplicate the human then resolves.
|
||
|
||
## Rules / gotchas
|
||
|
||
- **Module-name collision:** the intake config module is `settings.py`, **not** `config.py`,
|
||
because `backend/ingest/config.py` is imported (as bare `config`) through `spark → llm`. A
|
||
second `config` module would shadow it in `sys.modules` and break `llm` (`CHAT_MODEL`).
|
||
Keep intake module names from colliding with ingest's (`config`, `http_util`, `llm`).
|
||
- **Email integrity:** `parse.normalize` only keeps an address that literally appears in the
|
||
source message — the model must never mint one (a wrong email is worse than none). It takes
|
||
the **first** address in the text, so a two-person message ("Alice a@x.com and Bob b@y.com")
|
||
could attach the wrong one; the human sees it in the proposal and can `edit email=…` before
|
||
approving. Cross-referencing multiple addresses to the named contact is a deliberate non-goal
|
||
for v1.
|
||
- **Conversational revise keeps the email rule:** `parse.revise` re-runs a free-form correction
|
||
through Qwen but **never trusts the model's email field**. A changed address is accepted only if
|
||
it literally appears in the *instruction text* (searched first), else the existing
|
||
integrity-checked address is kept (`_apply_revision`). The model can edit name/contact/title/note
|
||
freely but cannot mint an email. A revision that nulls both investor and contact is rejected (the
|
||
proposal can't be emptied to something unactionable). Revise edits fields on the current proposal;
|
||
it does **not** re-run the matcher if you rename the firm mid-thread (a known v1 limit — the human
|
||
still approves).
|
||
- **Deploy is split across two surfaces** (mind which one carries a change): the fuzzy
|
||
**`candidates`** come from `server.py` → ship in the **s9pk** (build + install, version-bumped).
|
||
The bot's **disambiguation flow + `revise`** live in `backend/matrix_intake/` → ship on the
|
||
**Spark** via `git pull` + restart. A bot restart alone won't deliver `candidates` (the box would
|
||
return an empty list and the bot just proposes new — safe, but no fuzzy surfacing until the s9pk
|
||
is installed). Same lesson as the v83→v84 `/api/intake/match` 404.
|
||
- **Double-approve guard:** `handle_reply` pops the pending proposal from the store *before*
|
||
awaiting the commit, so a second `yes` arriving mid-write is a no-op (asyncio is cooperative;
|
||
the pop is atomic w.r.t. other events). On commit failure the proposal is restored for retry.
|
||
*Known minor:* in the **disambiguate** stage the pick re-stores an approval-stage proposal
|
||
before its `await say`, so a rapidly-repeated `1` can have the second one fall through to the
|
||
NL-revise path (a wasted Spark round-trip that re-prompts) — harmless, nothing commits, not
|
||
guarded (low likelihood on a ~5-person team).
|
||
- **Local-only parse:** intake text is real LP substance but goes ONLY to local Qwen via Spark
|
||
Control, never Claude — so no scrub boundary applies (same basis as the digest). Never call a
|
||
Spark directly; always go through `SPARK_CONTROL_URL`.
|
||
- **Auth:** the CRM has no service-key path; the bot logs in as a dedicated CRM user
|
||
(`CRM_BOT_USERNAME`/`CRM_BOT_PASSWORD`) → Bearer JWT, re-login once on 401.
|
||
- **Tests** are offline: `test_parse.py` / `test_proposals.py` / `test_crm_client.py` stub the
|
||
network; `backend/test_intake_endpoints.py` boots the real server against a temp DB and
|
||
covers `/api/intake/match` + the create→match (no-duplicate) contract + provenance. A **live
|
||
Matrix smoke** needs creds + `matrix-nio` installed on the Spark — it can't run in CI.
|
||
- **Grid note line:** the bot sends a **blank `subject`** when there's a note so the CRM's
|
||
one-line note summary shows the note text (the CRM renders subject-or-body); a provenance
|
||
label is sent only when there's no note. v0.1.0:85 also dropped the redundant `[note]` type
|
||
tag from that server-side line (informative types like `[call]` keep theirs).
|
||
|
||
## Deployment & ops
|
||
|
||
- **Runs on the Spark** (SSH alias `modelo32`, host `spark-32d0`): repo at
|
||
`/home/modelo/ten31-database`, deps in a venv (`.venv`; only `matrix-nio`). Launched detached:
|
||
`nohup ./.venv/bin/python backend/matrix_intake/bot.py >/tmp/intake-bot.log 2>&1 &`, pid in
|
||
`/tmp/intake-bot.pid`; startup logs `listening as … in room …`.
|
||
- **Restart after a `git pull` of bot code:** `kill $(cat /tmp/intake-bot.pid)`, relaunch as
|
||
above, re-write the pid. A restart **drops in-memory pending proposals** (re-send to recover).
|
||
- **NOT a managed service yet** — won't survive a Spark reboot; restart-on-boot (systemd) is an
|
||
open TODO.
|
||
- **Server-side endpoints ship in the s9pk, not the bot.** `GET /api/intake/match` and the
|
||
`source` provenance on `log-communication` live in `backend/server.py`, so they reach the box
|
||
only via an **s9pk build + install** — a bot restart won't deliver them. (Missed in v83: the
|
||
box 404'd `/api/intake/match` until **v0.1.0:84**.)
|
||
- **`CRM_API_BASE` is the box over the LAN, not localhost** (bot on the Spark, CRM on the box).
|
||
`https://immense-voyage.local` (443) is the **StartOS dashboard**, not the CRM — the CRM has
|
||
its own interface address (the URL you open in a browser); container port 8080 isn't
|
||
LAN-reachable.
|
||
|
||
## Config
|
||
|
||
All in `.env` (names in `.env.example`): `MATRIX_HOMESERVER`, `MATRIX_USER`,
|
||
`MATRIX_ACCESS_TOKEN`, `MATRIX_DEVICE_ID`, `MATRIX_INTAKE_ROOM`; `CRM_API_BASE`,
|
||
`CRM_BOT_USERNAME`, `CRM_BOT_PASSWORD`, `CRM_API_VERIFY_TLS`. Spark settings are inherited from
|
||
the ingest client (`SPARK_CONTROL_URL`, `CRM_CHAT_MODEL`).
|