Device-test round 2: 4 in-app fixes + Matrix intake cleanup (v0.1.0:99)

Grant's real-phone testing surfaced seven items; this lands six (the seventh,
in-app camera card intake, is planned in docs/handoffs/in-app-card-intake-plan.md).

CRM half — ships in the s9pk (v0.1.0:99):
- Intake fuzzy match no longer over-indexes on generic firm words. _name_similarity
  now compares DISTINCTIVE tokens only (generic descriptors — "Investment Group",
  "Capital", "Family Office" — stripped via _GENERIC_ORG_WORDS) for both the difflib
  ratio and the Jaccard, so "Fortitude Investment Group" stops surfacing Aether/Russell
  while "Aether Capital" still surfaces "Aether Investment Group". +2 regression cases.
- Mobile grid "Last contact"/staleness sort is reversible. SortSheet gains opt-in
  dir/onToggleDir; other surfaces (Contacts/Pipeline) are untouched.
- Mobile "Edit investor" prefills a contact's saved email. GET /api/fundraising/state
  heals a blank grid pill email from the linked classic contact
  (fundraising_contacts.contact_id -> contacts.email), fill-only, by pill order then
  name; the next one-row save persists it. +test_grid_email_heal.py.
- Mobile quick-log pencil icon renders. iOS collapses a sole, centered, attribute-only
  -sized flex-child <svg>; .quicklog-btn svg now gets explicit CSS width/height + flex:none
  (the pattern the working bottom-tab/sort-pill icons use). The v97 fix only changed color.

Matrix intake bot — ships on the Spark (bot-only, NOT the s9pk):
- Approve/reject now redacts the whole intake thread (card + ack + main-timeline nudge +
  the user's own photo/note), mirroring the email-review room; redact_thread takes the
  room as an arg and matches replies by m.thread OR m.in_reply_to (so the nudge clears).
  No more in-Matrix confirmation after a commit (the thread vanishing is the ack).
  Needs the bot to hold a redact/moderator power level in the intake room.
- New one-time backend/matrix_intake/redact_intake.py clears the room's pre-existing
  backlog (dry-run default; --apply).

Tests 42/42 green; frontend render-smoke green. Frontend fixes are inspection + render
-smoke verified (on-device confirm pending); the bot redaction is live-smoke only.
This commit is contained in:
Keysat
2026-06-20 12:32:56 -05:00
parent 7fe5f57c6e
commit a917280bbb
13 changed files with 606 additions and 58 deletions
+9 -8
View File
@@ -15,7 +15,7 @@
- Optional runtime deps, used only if present: `bcrypt`, `PyJWT` (`jwt`), `cryptography` (Gmail module).
- **MCP + ingest** (in the Docker image, not the bare CRM): `mcp==1.2.0` (FastMCP, `backend/mcp/server.py`), `fastembed==0.4.2`, `anthropic`, `cryptography==42.0.5`.
- **Packaging:** StartOS 0.4, TypeScript SDK (`@start9labs/start-sdk`) under `start9/0.4/startos/`. Live target is `start9/0.4/`.
- **Local models** (bge-m3 embeddings, bge-reranker-v2-m3, `/api/search`, Qdrant): always via Spark Control. Contract: `docs/EMBEDDINGS.md`.
- **Local models** (bge-m3 embeddings, bge-reranker-v2-m3, `/api/search`, Qdrant): always via Spark Control. Contract: `docs/EMBEDDINGS.md`. The chat model (`CRM_CHAT_MODEL`, the daily-driver Qwen) is **vision-capable** — Spark Control's `/v1/chat/completions` is a dumb passthrough, so OpenAI **multimodal** `image_url` data-URIs work unchanged (used by the intake bot's business-card OCR; reuse `llm.chat_vision`).
## Commands
@@ -108,11 +108,12 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude
## Current state
_**Box live at v0.1.0:97 (deployed 2026-06-20)** — the full mobile-first redesign (Phases 07 + P3b + drag-reorder + **8a8i**) **+ installable PWA (Option A)** went live at v95; **v96** brought the login/first-admin screen into mobile/PWA conformance; **v97** is the first round of Grant's real-phone feedback — viewport zoom-lock (`maximum-scale=1` + `user-scalable=no`: no pinch, no iOS auto-zoom-on-focus for our <16px inputs; app-wide) + mobile top-bar fixes (account initial flex-centered & dc-aligned; quick-log pencil bumped to `--text-secondary` for affordance). All CSS-only, desktop untouched. **The fundraising grid + email capture is the canonical system of record.** Active thread: **mobile-first redesign DEPLOYED; Grant is now real-phone device-testing and reporting polish items** (v97 is the first batch). Build reference: `design/phase8-conformance.md` (the 8a8i spec; fully landed — archive-eligible). History: git log + `start9/0.4/startos/versions/`._
_**Box live at v0.1.0:98; v0.1.0:99 built + installing this session (2026-06-20)**. This session = **Grant's real-phone device-test round 2**: four in-app fixes (CRM half → v99 s9pk) + a Matrix intake-bot cleanup (Spark) + a written plan for in-app camera card intake. **The fundraising grid + email capture is the canonical system of record.** History: git log + `start9/0.4/startos/versions/`._
- **Mobile redesign — 4 core surfaces built (Grid · Contacts · Pipeline · Reminders), each a rules-of-hooks-safe `useIsMobile()``Mobile*`/`Desktop*` pair (desktop untouched).** Foundation: bottom-tab bar + `:root` mobile vars; 4-stage enum; derived grid signals injected-on-GET/stripped-on-write at both points; mobile writes use **one-row endpoints only** (log-communication, pipeline link/stage, reminders, `update-row`) — never whole-grid PUT.
- **Phase 8 complete (8a8i)** — all mobile cards/details/sheets/shell match the dc anatomy; the durable per-primitive record is the **Design convention's primitives list**, per-phase detail in git log. Installable PWA is a durable Conventions bullet (+ `ROADMAP.md` "Mobile PWA").
- **Live (deployed):** **mobile zoom-lock + top-bar polish (v97, 2026-06-20)**; **login page mobile/PWA conformance (v96)**; **mobile-first redesign (Phases 08i) + light theme + installable PWA + 4-stage pipeline funnel (migration 0007) — all v95 (2026-06-20)**; W2 NL query (v94); W1 reminders (v93); grid Pipeline (v88); Matrix intake + Gmail capture (DWD) + daily digest; Thesis/Architect (dual-approval); outreach — all draft-only.
- **Tests:** **40/40 backend green** (`python3 backend/run_tests.py`), `py_compile` clean. Mobile surfaces interaction-verified via throwaway 375px jsdom harnesses (deleted after); harness recipe + the `Simulate.change` input gotcha live in the `mobile-surface-verification` memory.
- **Next — Grant's real-phone device-test (v97 is live):** the whole mobile set + PWA + login + top-bar fixes shipped 2026-06-20. **Confirm on v97:** tapping a field no longer zooms the page; pinch is disabled; top-bar **account initial is centered** and the **quick-log pencil is visible**. Plus the standing gate: light/dark across the 4 surfaces **+ login** (375px gutters, card not under the status bar in standalone), **install-to-home-screen → standalone launch** (Safari Share → Add to Home Screen; expect full-screen, dark status bar, T31 icon), safe-area/tab-bar clearance, swipe/sheet interactions (only smoke/jsdom-verified so far). Known PWA item to eyeball: the iOS status bar is fixed `black` (light-theme cosmetic seam — see Open/risks).
- **Open / risks:** all mobile work + light theme **deployed (v95/v96/v97) but real-phone device-test only just starting** (Grant); **viewport now `user-scalable=no`** — intentional native feel for this internal tool; OS accessibility zoom still works, but it's a known a11y trade-off if scope ever widens beyond the team; **quick-log pencil deviates from the dc `--t3` spec → `--text-secondary`** on Grant's "can't see it" report (the dc grey thin-outline reads as empty next to the color sun emoji; confirm on-device it now reads clearly); `.login-title` CSS (`index.html:1869`) is **dead** (defined, never rendered — login uses logo + subtitle; trivial removal candidate); `MobileDetailRow` unused-but-retained (legacy-usage sweep); Pipeline detail "Committed" tile shows grid-committed not deal-expected (forecast in a footnote); `handle_get_opportunity` (single-opp GET) deliberately does NOT inject `existing_investor`/`last_contact_date` — no surface needs it (the card uses the list injection; the detail derives `existing` from the contact fetch); W2 happy-path only; **Claude/Architect path unverified live on the box**; v2.0 reserve-asset spine **not canonical**; doc drift — `crm-overview.md`/`EVALUATION.md` still call `lp_profiles` live; **PWA iOS status bar is fixed `black` at launch** (`apple-mobile-web-app-status-bar-style`) — v96 fixed the *login-card* collision (it now respects `env(safe-area-inset-top)`), but the global header still gets a black strip above it in *light* theme (proper fix = `black-translucent` + header top-safe-area padding + theme-synced `theme-color`; deferred, validate on-device; dark is default so low-priority); PWA manifest/icons sent with no `Cache-Control` (consistent with all static routes — a manifest change post-install may be served stale by iOS until it re-fetches).
- **Device-test round 2 — 4 fixes shipped to v0.1.0:99 (CRM half).** (1) **Intake fuzzy match** no longer over-indexes on generic firm words — `_name_similarity` scores **distinctive** tokens only (generic descriptors like "Investment Group"/"Capital"/"Family Office" stripped via `_GENERIC_ORG_WORDS`); "Fortitude Investment Group" no longer surfaces Aether/Russell. (2) **Mobile grid "Last contact"/staleness sort is reversible** (`SortSheet` opt-in `dir`/`onToggleDir`; other surfaces untouched). (3) **Mobile "Edit investor" prefills a contact's email**`GET /api/fundraising/state` heals a blank grid pill email from `fundraising_contacts.contact_id → contacts.email` (fill-only, by pill order then name; next one-row save persists it; `fundraising_contact_emails_by_row`). (4) **Quick-log pencil icon renders**`.quicklog-btn svg { width;height;flex:none }` (iOS collapses a sole, centered, attribute-only-sized flex-child svg; the v97 fix only changed its color).
- **Matrix intake-bot — thread auto-delete on decision + retroactive purge (Spark, bot-only, NOT in the s9pk).** Approve/reject now `redact_thread(intake_room, root)` (clears card + ack + main-timeline nudge + the user's photo/note), mirroring the email-review room; the scan now also catches the un-threaded nudge (`m.in_reply_to`). New one-time `backend/matrix_intake/redact_intake.py` (dry-run default; `--apply`) clears the room backlog. **Needs the bot to hold a redact/moderator power level in the intake room** to clear users' messages (manual Element step). No more in-Matrix "✅ logged" confirmation after a commit (by design, like email). Detail: `docs/guides/matrix-intake.md`.
- **In-app camera card intake (#7) — PLAN written, not built.** `docs/handoffs/in-app-card-intake-plan.md`: reuses the nio-free transcribe/parse core (`server.py` already imports `llm`; `matrix_intake/parse.py`+`spark.py` are nio-free) → **one endpoint** `POST /api/intake/card` + **one mobile component** (camera button left of the pencil). No bot refactor, no new dep, no migration. **Awaiting Grant's call** on 4 decisions (provenance tag, form-edits-only v1, member access, ships-in-s9pk).
- **Mobile-first redesign — deployed (v95v97), on-device test in progress.** 4 surfaces (Grid·Contacts·Pipeline·Reminders) + light theme + installable PWA + 4-stage funnel; desktop untouched. Standing gate: light/dark across surfaces + login (375px gutters, safe-area), install→standalone launch, swipe/sheet interactions (only jsdom-smoked). Other live features: W2 NL query (v94), W1 reminders (v93), grid Pipeline (v88), Gmail capture + daily digest, Thesis/Architect (dual-approval), outreach — all draft-only. **Business-card intake (M3, Matrix bot) LIVE since v98** (vision OCR via the daily-driver model; `source="matrix_card"`; captures name/email/title/city/LinkedIn/phone/mobile, integrity-checked).
- **Tests: 42/42 backend green** (`python3 backend/run_tests.py`), `py_compile` clean, frontend render-smoke green (`make render-smoke`). New: `test_grid_email_heal.py` + intake generic-word cases. Vision/OCR (Matrix + the planned in-app path) is **live-smoke only**.
- **Next:** (1) Grant device-tests the 4 v99 fixes on his phone (esp. the **pencil icon** — root-caused but device-confirm) + re-tests cards (OCR/phone mapping); (2) Grant gives the intake bot **mod power** in the room, then `redact_intake.py --apply` on the Spark; (3) Grant's call on the #7 plan; (4) finish the standing mobile on-device gate.
- **Open / risks:** **vision OCR can misread a character** on a card small-in-frame (resolution-bound — Spark Control downscales to ~2 MP; `mara.com→marac.com` reproduced at temp 0; mitigations: fill the frame, or a future client-side crop); **iPhone HEIC** may not decode in vLLM (most clients send JPEG); phone/mobile/LinkedIn land on the **contact record**, not the grid pill (by design — only city syncs to the pill); intake redaction needs the bot's room **mod power** or users' messages linger; **Claude/Architect path unverified live on the box**; v2.0 reserve-asset spine **not canonical**; **PWA iOS status bar fixed `black`** in light theme (header seam; deferred, dark is default); doc drift — `crm-overview.md`/`EVALUATION.md` still call `lp_profiles` live; assorted minor UI cleanups (`.login-title` dead CSS, `MobileDetailRow` unused, Pipeline "Committed" tile shows grid-committed) tracked in git history.
+7 -4
View File
@@ -103,19 +103,19 @@
**W3 — Bot grid-mutations behind a Matrix approval gate.** Generalize the email-proposal scaffold (`email_proposal_matrix` + propose→post→decide→apply) into one `agent_proposals` table (kind discriminator + JSON payload + target). Bot proposes set-commitment / assign-fund / change-stage / set-reminder; a human approves/edits/rejects in Matrix (**any member**); then apply. **Surgical, version-checked mutations — never blob RMW:** stage rides the existing `opportunities` link + validated stage endpoint; reminders write the W1 table; set-commitment/assign-fund need a version-checked single-cell upsert into the grid blob. Triggers the deferred **scoped service-token** item below (per-mutation-kind allowlist on the bot credential; money/merge/delete always require human approval regardless of scope — the autonomy axis). Parse on local Qwen, not Claude.
### Matrix-bridge intake for the fundraising grid — M1+M2 BUILT (deploy + live smoke pending)
*Requested 2026-06-16. **M1 (scaffold + parse + in-thread propose) and M2 (match + write-on-approve) built, tested (26/26), not yet deployed** — code in `backend/matrix_intake/`, guide at `docs/guides/matrix-intake.md`. Remaining: install `matrix-nio` + creds on the Spark, create the CRM bot user, and a **live Matrix smoke** (can't run in CI). M3 (business-card photo) deferred until Spark Control has a vision model. Next major build after this is **Pipeline adoption** (see below).*
### Matrix-bridge intake for the fundraising grid — M1+M2+M3 BUILT & LIVE
*Requested 2026-06-16. **M1+M2 live since v0.1.0:86 (2026-06-17); M3 (business-card photo) shipped & live 2026-06-20** — code in `backend/matrix_intake/`, guide at `docs/guides/matrix-intake.md`. M3 unblocked once the Spark Control daily-driver model became vision-capable: the bot OCRs a card via Spark Control's `/v1/chat/completions` multimodal passthrough (same `CRM_CHAT_MODEL`), then runs the existing intake flow; captures contact name/email/title/city/LinkedIn/phone/mobile (server half of phone/mobile = s9pk v0.1.0:98). Remaining: ongoing on-device card testing (OCR accuracy on small-in-frame cards). Next major build is **Pipeline adoption** (see below).*
Use the **matrix-bridge** repo's pattern to listen on a dedicated ten31-database Matrix room. Send a message (with an optional business-card photo) and a local LLM **via Spark Control** parses it into the fundraising-grid schema and **auto-creates the investor entity + contact row**. For an existing investor, send a meeting note and it **appends an interaction-log entry**. Approval gate: the bot replies in Matrix with the proposed add/edit; the user approves / rejects / edits in-thread before the write commits (keeps the draft→human-approve guardrail).
- Fits the "grid is canonical" direction (writes land in `fundraising_*`) and the never-send-autonomously rule (in-thread human approval before any write).
**Locked design (2026-06-16, approved) — build now, M1 then M2:**
- **Separate component, shared scaffold:** new `backend/matrix_intake/` (its own process; lifts matrix-bridge's connect/prime-then-listen/threaded-reply plumbing). `matrix-nio` is isolated to this component's `requirements.txt` — it never enters the stdlib CRM runtime. Keeps the CRM write credential + LP data out of the general-purpose matrix-bridge bot (blast-radius + data-sovereignty), and lets the two iterate independently. Runs on the Spark (placement settled against `standards/guides/placement.md` at deploy).
- **v1 = text-only.** Business-card photo deferred to M3 — Spark Control fronts chat/embeddings/rerank but **no vision model** today, so photo→fields isn't buildable end-to-end yet.
- **~~v1 = text-only~~ — M3 business-card photo SHIPPED (2026-06-20).** The Spark Control daily-driver model is now vision-capable (multimodal `image_url` passthrough), so card→text→fields works end-to-end. Transcribe-then-reuse (vision OCRs to text; the existing text extractor pulls fields) preserves the email/phone integrity rules. See the matrix-intake guide.
- **Parse:** local Qwen via Spark Control `/v1/chat/completions` (temp 0, JSON-only), reusing the existing Spark client pattern (`backend/redaction`/`backend/ingest`).
- **Approval handshake (the one stateful piece):** in-memory pending-proposal store keyed by Matrix thread root; user replies **yes / edit field=value / no** in-thread. Satisfies never-write-autonomously; exempt from "agents draft, humans send" (internal data entry, like the digest).
- **CRM-side:** `POST /api/intake/investor` (service-auth) creates a new investor+contact **through the existing grid-save path** (so relational sync + audit + backup-on-write happen as with a UI edit; bot never does whole-blob RMW) or appends a meeting note to the interaction log for an existing investor; `GET /api/intake/match?q=` fuzzy-matches via the existing entity-resolution/email-matcher. New investor needs no fund at intake.
- **Phases:** M1 = scaffold + parse + in-thread propose, **no writes** (proves Matrix↔Spark). M2 = intake endpoint + match + write-on-approve + tests. M3 (deferred) = business-card photo.
- **Phases:** M1 = scaffold + parse + in-thread propose, **no writes** (proves Matrix↔Spark). M2 = intake endpoint + match + write-on-approve + tests. **M3 = business-card photo (SHIPPED 2026-06-20).**
**Post-deploy enhancement — fuzzy match + in-thread confirm (Grant, 2026-06-17). DEPLOYED & LIVE 2026-06-17 (v0.1.0:86; box migration chain …85→86 clean, `candidates` endpoint verified); Matrix live-smoke pending.** Today `find_intake_match` is **exact-after-normalization** (`_normalize_text` = lowercase+strip), so near-misses — "Charlie" vs "Charles" (same last name), "Acme Capital" vs "Acme Capital LLC", a one-character email typo — return no match and the bot proposes a **new** investor, risking a duplicate the human approves without realizing a near-match exists. The existing in-thread approval gate is useless against this because the human is never *shown* the near-match. Fix: matcher returns **ranked fuzzy candidates** (deterministic pre-filter: normalized name similarity / token overlap + email edit-distance ≤ ~2), surfaced in-thread for the human to confirm or pick, with the **local Spark LLM optionally re-ranking/judging the shortlist** (good at Charlie/Charles + legal-suffix equivalence; fed only the shortlist, never the whole LP list). Keeps the approval gate but makes it effective against duplicates. Land **after** the live smoke — net-new logic + reply grammar + tests; the current exact match is safe and its failure mode (a duplicate) is recoverable via the existing entity-merge subsystem (`backend/entity_*.py`).
- **As built:** `find_intake_candidates` in `server.py` (deterministic — stdlib `difflib` name similarity + token-set Jaccard, legal-suffix-aware via `_strip_legal_suffix`, + email Levenshtein ≤ 2; ranked, ≥0.62, top 5). `GET /api/intake/match` now returns `{match, candidates}`. Bot: a new `_stage="disambiguate"` shortlist (`proposals.render_disambiguation` / `interpret_disambiguation` / `attach_to_candidate` / `promote_to_new`) — human picks a number / `new` / `no`. **The optional LLM-judge re-rank was deliberately deferred** (the deterministic filter already surfaces the named cases; an LLM judge is the right *pruner* for shortlist noise — build if the deterministic ranking proves too noisy in practice). Tests: `test_intake_endpoints.py` (server fuzzy cases), `matrix_intake/test_proposals.py` (disambiguation grammar), `matrix_intake/test_crm_client.py` (candidate shape).
@@ -129,6 +129,9 @@ Use the **matrix-bridge** repo's pattern to listen on a dedicated ten31-database
**Long-term — extract the intake bot to its own repo (recommended, not yet done).** Containerizing from this monorepo is the pragmatic now-state, but the bot is a genuinely separate deployable (own process, own `matrix-nio` dep, own lifecycle); its only CRM coupling is the HTTP API (a clean network contract) plus ~40 lines of stdlib Spark client (cheap to vendor). The tell: the spark-control Update button would run `git reset --hard origin/main` on the **whole CRM clone** — wrong blast radius. `matrix-bridge` is already a dedicated repo; mirror it. The extraction is a migration (new Gitea repo, move code + tests + guide, vendor the client, re-point the Spark deploy), so it's deferred until worth the lift — do it *before* wiring the spark-control card if both land in the same push.
### In-app camera business-card intake — PLAN written, awaiting go-ahead (Grant, 2026-06-20)
*A camera button in the mobile top bar (left of the quick-log pencil) → take/choose a photo → the same vision-transcribe → parse → fuzzy-match → edit/approve/reject flow the Matrix card intake (M3) runs, surfaced as an inline mobile sheet. **Detailed plan: `docs/handoffs/in-app-card-intake-plan.md`.*** Key finding: the reusable core is nio-free and already reachable from the CRM (`server.py` imports `llm`; `matrix_intake/parse.py`+`spark.py` import no `matrix-nio`), so it's **one endpoint** (`POST /api/intake/card`) + **one mobile component**, no bot refactor / new dep / migration. Four decisions pending Grant before building: provenance tag (`app_card`?), form-edits-only for v1 (no conversational NL-edit), any-member access, ships-in-s9pk. Reuses the New-investor sheet pre-filled + the proven `.quicklog-btn svg` icon-sizing fix for the camera button.
### Scoped service-credential auth path for automated CRM writers
*Surfaced 2026-06-17 while deploying the Matrix intake bot. **Decision: defer — the bot uses a dedicated member username/password for now.** The CRM has no API-key/service-token path; its only auth is username+password → JWT. A dedicated **member** login is appropriately scoped against what matters operationally (no admin: can't manage users, reset data, or change settings) and unblocks the live smoke today.*
+32 -21
View File
@@ -171,9 +171,13 @@ async def main():
store.put(root, proposal) # commit failed — restore so the user can retry
await say(room_id, f"⚠️ write failed, nothing committed: {exc}", root)
return
await say(room_id, f"{summary}", root)
# Committed → clear the whole thread (card + ack + nudge + the user's note/photo),
# like the email-review room. The thread vanishing is the acknowledgment; a confirmation
# reply would just keep it alive (and need redacting too). Needs the bot's redact/mod
# power in the intake room to clear the user's own messages — else those linger.
await redact_thread(room_id, root)
elif action == "reject":
await say(room_id, "🗑️ Discarded — nothing written.", root)
await redact_thread(room_id, root)
elif action == "edit":
field, value = payload
proposal = proposals.apply_edit(proposal, field, value)
@@ -212,42 +216,49 @@ async def main():
await say(room_id, " OK — adding as a new investor:\n\n"
+ proposals.render(updated), root)
elif action == "reject":
await say(room_id, "🗑️ Discarded — nothing written.", root)
await redact_thread(room_id, root) # discard → clear the thread, like an approve
else: # unrecognized — re-show the shortlist
store.put(root, proposal)
await say(room_id, "I didn't catch that.\n\n" + proposals.render_disambiguation(proposal), root)
async def redact_card(event_id):
"""Redact one event (best-effort). Redacting our OWN message needs no special power;
redacting someone else's reply needs the bot to hold a redact/mod power level."""
async def redact_card(room_id, event_id):
"""Redact one event in `room_id` (best-effort). Redacting our OWN message needs no special
power; redacting someone else's message (a human reply, or the user's original card photo /
intake note) needs the bot to hold a redact/mod power level in that room."""
try:
await client.room_redact(review_room, event_id, reason="proposal resolved")
await client.room_redact(room_id, event_id, reason="proposal resolved")
except Exception as exc:
print(f"matrix-intake: could not redact {event_id}: {exc}", flush=True)
async def redact_thread(root):
"""Clear a resolved thread: redact the card AND every reply under it, so the thread drops
out of the threads view (not just the main timeline). The card is ours (always redactable);
the human's yes/no reply needs the bot's redact/mod power — if it lacks power that redact
just no-ops and the reply lingers. Finds replies by scanning recent room history for
m.thread events pointing at this root (the triggering reply is already synced, so a
backward scan from the current token includes it)."""
await redact_card(root)
async def redact_thread(room_id, root):
"""Clear a resolved thread in `room_id`: redact the root AND every message that hangs off it
the m.thread children (cards/acks/human replies) AND the main-timeline **nudge** (a plain
m.in_reply_to reply, not a thread child), so the thread drops out of both the threads view
and the timeline. For email-review the root is the bot's card; for intake it's the USER'S
own note/photo, so clearing it (and the human reply) needs the bot's redact/mod power in that
room — without it those just no-op and linger. Replies are found by scanning recent history
from the current sync token (the triggering reply is already synced, so a backward scan
includes it)."""
await redact_card(room_id, root)
token = getattr(client, "next_batch", None)
if not token:
return
try:
scanned = 0
for _ in range(MAX_THREAD_SCAN_PAGES):
resp = await client.room_messages(review_room, start=token,
resp = await client.room_messages(room_id, start=token,
direction=MessageDirection.back, limit=100)
chunk = getattr(resp, "chunk", None)
if not chunk:
break
for ev in chunk:
rel = ((getattr(ev, "source", None) or {}).get("content", {}) or {}).get("m.relates_to") or {}
if rel.get("rel_type") == "m.thread" and rel.get("event_id") == root:
await redact_card(ev.event_id)
in_reply = (rel.get("m.in_reply_to") or {}).get("event_id")
# A thread child carries event_id==root; the un-threaded nudge carries only
# m.in_reply_to.event_id==root. Catch both so the thread AND its main-timeline
# pointer clear together.
if rel.get("event_id") == root or in_reply == root:
await redact_card(room_id, ev.event_id)
token = getattr(resp, "end", None)
scanned += len(chunk)
if not token or scanned > 1000:
@@ -275,7 +286,7 @@ async def main():
return
# Success → clear the whole thread (card + replies). No confirmation: the thread
# vanishing is the acknowledgment, and a confirmation reply would keep it alive.
await redact_thread(root)
await redact_thread(review_room, root)
elif decision == "reject":
email_threads.pop(root, None)
try:
@@ -284,7 +295,7 @@ async def main():
email_threads[root] = item
await say(room_id, email_proposals.frame(f"⚠️ couldn't dismiss it ({str(exc)[:200]}). Try again."), root)
return
await redact_thread(root)
await redact_thread(review_room, root)
else:
try:
new_note = await asyncio.to_thread(email_proposals.revise_note, item.get("note") or "", text)
@@ -332,7 +343,7 @@ async def main():
if not ev:
continue
try:
await redact_thread(ev)
await redact_thread(review_room, ev)
await asyncio.to_thread(crm_client.mark_email_proposal_closed, it["id"])
email_threads.pop(ev, None)
except Exception as exc:
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env python3
"""One-time maintenance: clear the intake room's backlog of resolved/stale messages.
Going forward the bot redacts each intake thread when it's approved/rejected (bot card + ack +
nudge + the user's own note/photo). This clears the messages that piled up BEFORE that shipped.
The intake room is single-purpose and the bot keeps **no durable pending state** (its proposal
store is in-memory and is lost on every restart), so nothing in the room is "still live" after a
restart — every message in it is safe to redact. This walks the room history and redacts every
m.room.message event (text + business-card images), bot's and humans' alike.
Redacting another user's message (the humans' notes/photos) needs the bot to hold a **redact /
moderator power level** in the intake room — without it those just no-op and linger (the bot's own
messages still clear). Make the bot a moderator of the intake room in Element first.
Safe by default: prints what it WOULD redact and does nothing. Pass --apply to actually redact.
Run on the Spark via the bot's own creds/image:
docker compose run --rm matrix-intake python -u backend/matrix_intake/redact_intake.py
docker compose run --rm matrix-intake python -u backend/matrix_intake/redact_intake.py --apply
"""
import asyncio
import sys
from nio import AsyncClient, MessageDirection
import settings
MAX_PAGES = 50 # 50 * 100 events is far more history than this room holds
async def main(apply):
mx = settings.matrix_settings()
intake_room = mx.get("intake_room")
if not intake_room:
print("MATRIX_INTAKE_ROOM is not set — nothing to do.")
return
client = AsyncClient(mx["homeserver"], mx["user_id"])
client.restore_login(user_id=mx["user_id"], device_id=mx["device_id"], access_token=mx["token"])
try:
sync = await client.sync(timeout=10000, full_state=False)
token = sync.next_batch
targets = [] # (event_id, label)
seen = set()
for _ in range(MAX_PAGES):
resp = await client.room_messages(intake_room, start=token,
direction=MessageDirection.back, limit=100)
chunk = getattr(resp, "chunk", None)
if not chunk:
break
for ev in chunk:
src = getattr(ev, "source", None) or {}
if src.get("type") != "m.room.message":
continue # only chat messages + images; leave membership/state events alone
eid = getattr(ev, "event_id", None)
if not eid or eid in seen:
continue
seen.add(eid)
content = src.get("content") or {}
if not content:
continue # already redacted (content stripped) — skip
msgtype = content.get("msgtype") or "?"
body = (content.get("body", "") or "").replace("\n", " ")
who = "bot " if getattr(ev, "sender", None) == mx["user_id"] else "user"
targets.append((eid, f"{who} [{msgtype}] {body[:60]}"))
token = getattr(resp, "end", None)
if not token:
break
print(f"messages to clear in the intake room: {len(targets)}")
fails = 0
for eid, label in targets:
print(("APPLY redact " if apply else "WOULD redact ") + eid + " :: " + label)
if apply:
r = await client.room_redact(intake_room, eid, reason="retroactive intake-room cleanup")
if not hasattr(r, "event_id"):
fails += 1
print(f" ! redact failed (need mod power for others' messages?): {r}")
print(("done — redacted " if apply else "dry run — would redact ")
+ f"{len(targets) - (fails if apply else 0)}/{len(targets)} event(s)"
+ (f"; {fails} failed" if apply and fails else "") + ".")
finally:
await client.close()
if __name__ == "__main__":
asyncio.run(main(apply="--apply" in sys.argv[1:]))
+86 -6
View File
@@ -1305,14 +1305,40 @@ def _strip_legal_suffix(normalized_name):
return " ".join(toks)
# Generic firm-descriptor words that carry almost no identifying signal: nearly every firm name
# contains one ("… Investment Group", "… Capital", "… Family Office"). Two names that overlap ONLY
# on these are NOT duplicates — 'Fortitude Investment Group' is not 'Aether Investment Group'. We
# compare on the DISTINCTIVE remainder so a shared descriptor can't inflate the score (the earlier
# "Capital/Ventures/Partners are distinctive enough to keep" assumption produced false shortlists —
# Grant, 2026-06-20). If a name is ALL descriptor ('Family Office'), we fall back to its full tokens
# so there's still something to compare.
_GENERIC_ORG_WORDS = frozenset({
"investment", "investments", "investing", "investor", "investors",
"capital", "ventures", "venture", "partners", "partner", "group",
"fund", "funds", "management", "advisors", "advisers", "advisory",
"asset", "assets", "holdings", "holding", "family", "office",
"trust", "associates", "equity", "financial", "finance", "global",
"international", "company", "enterprises", "wealth", "the", "and", "of",
})
def _distinctive_tokens(normalized_name):
"""Tokens of a (legal-suffix-stripped) name with generic firm descriptors removed. Falls back to
the full token list when the name is nothing but descriptors, so an all-generic name still compares."""
toks = re.findall(r"[a-z0-9]+", normalized_name)
keep = [t for t in toks if t not in _GENERIC_ORG_WORDS]
return keep or toks
def _name_similarity(a, b):
"""0..1 fuzzy similarity between two investor names: the max of difflib's sequence ratio
(catches near-spellings 'Charlie'/'Charles') and token-set Jaccard overlap (catches
word-order differences). Legal-entity suffixes are stripped first, so two names differing
only by 'LLC'/'LP'/'Inc' score 1.0 (a near-certain duplicate to surface find_intake_match
won't have caught it, since it compares the full string). Favors recall: a shared common
name-word ('… Capital') can lift unrelated firms into the 0.60.8 band acceptable noise in
a ranked, human-confirmed shortlist; semantic pruning is the deferred LLM-judge's job."""
won't have caught it, since it compares the full string). Both the ratio and the Jaccard run on
the DISTINCTIVE tokens (generic descriptors like 'Investment Group'/'Capital' removed), so firms
that share only a descriptor don't surface as look-alikes; 'Aether Capital' ~ 'Aether Capital
Partners' still scores 1.0 on the distinctive 'aether'. Still recall-favoring on real overlap."""
a = _normalize_text(a)
b = _normalize_text(b)
if not a or not b:
@@ -1323,9 +1349,10 @@ def _name_similarity(a, b):
sb = _strip_legal_suffix(b) or b
if sa == sb:
return 1.0
ratio = difflib.SequenceMatcher(None, sa, sb).ratio()
ta = set(re.findall(r"[a-z0-9]+", sa))
tb = set(re.findall(r"[a-z0-9]+", sb))
da = _distinctive_tokens(sa) # order-preserving for the sequence ratio
db = _distinctive_tokens(sb)
ratio = difflib.SequenceMatcher(None, " ".join(da), " ".join(db)).ratio()
ta, tb = set(da), set(db)
jaccard = len(ta & tb) / len(ta | tb) if (ta or tb) else 0.0
return max(ratio, jaccard)
@@ -1881,6 +1908,45 @@ def existing_investor_by_source_row(conn):
return out
def fundraising_contact_emails_by_row(conn):
"""{ source_row_id: {'order': {sort_order: email}, 'name': {normalized_name: email}} } of the
authoritative email per grid contact, for HEALING blank pill emails on read.
The grid blob is canonical for the edit sheet, but an email can reach the linked classic
contact (via email capture / a contact edit) without ever being written back into the blob
pill so the mobile "Edit investor" sheet shows an empty email for a contact the directory
clearly has (Grant, 2026-06-20). We recover it from the relational mirror: prefer the synced
fundraising_contacts.email, else the linked classic contacts.email (the source that actually
holds the captured address). Keyed by sort_order (pills and fundraising_contacts share the
blob order the robust key) with a normalized-name fallback. Only non-blank emails are
returned; filling is fill-only-when-blank in the handler, so it heals and converges (the next
one-row save persists the recovered email into the blob)."""
out = {}
rows = conn.execute(
"""
SELECT fi.source_row_id AS srid, fc.sort_order AS so, fc.full_name AS name,
COALESCE(NULLIF(TRIM(fc.email), ''), c.email) AS email
FROM fundraising_investors fi
JOIN fundraising_contacts fc ON fc.investor_id = fi.id
LEFT JOIN contacts c ON c.id = fc.contact_id AND c.deleted_at IS NULL
"""
).fetchall()
for r in rows:
email = str(r['email'] or '').strip()
if not email:
continue
srid = str(r['srid'] or '')
if not srid:
continue
bucket = out.setdefault(srid, {'order': {}, 'name': {}})
if r['so'] is not None:
bucket['order'][int(r['so'])] = email
nm = _normalize_text(r['name'])
if nm:
bucket['name'][nm] = email
return out
def contact_grid_signals(conn, contact_id=None):
"""Return {contacts.id: {'committed': float, 'pipeline_stage': str|None, 'priority': bool}} for
every classic contact linked to a fundraising-grid investor (via fundraising_contacts.contact_id,
@@ -5830,6 +5896,7 @@ class CRMHandler(BaseHTTPRequestHandler):
reminder_by_row = reminder_status_by_source_row(conn)
existing_by_row = existing_investor_by_source_row(conn)
recency_by_row = staleness_by_source_row(conn)
emails_by_row = fundraising_contact_emails_by_row(conn)
conn.close()
try:
@@ -5873,6 +5940,19 @@ class CRMHandler(BaseHTTPRequestHandler):
last_activity, staleness = recency_by_row.get(srid, (None, ''))
r['last_activity_at'] = last_activity
r['staleness'] = staleness
# Heal blank pill emails from the relational mirror (fill-only — never overwrite a value
# already in the blob). Unlike the read-only columns above, email is a REAL blob field,
# so this is a backfill, not a derived signal: it needs NO strip point, and the next
# one-row save legitimately persists it. Match by pill order, then by name.
heal = emails_by_row.get(srid)
pills = r.get('contacts')
if heal and isinstance(pills, list):
for i, c in enumerate(pills):
if not isinstance(c, dict) or str(c.get('email') or '').strip():
continue
found = heal['order'].get(i) or heal['name'].get(_normalize_text(c.get('name')))
if found:
c['email'] = found
return self.send_json({
"data": {
+135
View File
@@ -0,0 +1,135 @@
#!/usr/bin/env python3
"""Regression: GET /api/fundraising/state heals blank grid-pill emails from the relational mirror.
The grid blob is canonical for the mobile "Edit investor" sheet, but an email can reach a linked
classic contact (email capture / a contact edit) without ever being written back into the blob pill
— so the edit form showed an empty email for a contact the directory clearly had (Grant, 2026-06-20).
The state handler now fills a blank pill email from fundraising_contacts.email, else the linked
contacts.email, matched by pill order then name. This asserts:
- a blank pill whose linked contact has an email is HEALED on read;
- a blank pill whose linked contact is also blank stays blank;
- a pill that already carries an email in the blob is NEVER overwritten (fill-only).
Synthetic data only.
Run: cd backend && python3 test_grid_email_heal.py
"""
import http.client
import json
import os
import sqlite3
import sys
import tempfile
import threading
from http.server import ThreadingHTTPServer
_DATA = tempfile.mkdtemp()
os.environ["CRM_DATA_DIR"] = _DATA
os.environ["CRM_DB_PATH"] = os.path.join(_DATA, "crm.db")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import server # noqa: E402
FAILS = []
def check(cond, msg):
print((" PASS " if cond else " FAIL ") + msg)
if not cond:
FAILS.append(msg)
class _Quiet(server.CRMHandler):
def log_message(self, *a):
pass
def _get_state(port, token):
conn = http.client.HTTPConnection("127.0.0.1", port, timeout=10)
conn.request("GET", "/api/fundraising/state", headers={"Authorization": "Bearer " + token})
resp = conn.getresponse()
raw = resp.read().decode("utf-8", "replace")
conn.close()
return resp.status, (json.loads(raw) if raw else None)
GRID = {
"columns": [{"id": "investor_name", "label": "Investor", "type": "text"},
{"id": "contacts", "label": "Contacts", "type": "contacts"}],
"rows": [
{"id": "rowW", "investor_name": "Wyoming", "notes": "",
"contacts": [{"name": "Philip Treick", "email": "", "title": ""},
{"name": "Jose Briones", "email": "", "title": ""}]},
{"id": "rowA", "investor_name": "Acme Capital", "notes": "",
"contacts": [{"name": "Jane Doe", "email": "keep@acme.com", "title": ""}]},
],
}
def seed():
c = sqlite3.connect(os.environ["CRM_DB_PATH"])
c.execute("INSERT INTO users (id,username,email,password_hash,full_name,role,is_active) "
"VALUES ('u1','grant','grant@ten31.example','x','Grant','admin',1)")
c.execute("INSERT INTO fundraising_state (id, grid_json, views_json, version) "
"VALUES ('main', ?, '[]', 1) "
"ON CONFLICT(id) DO UPDATE SET grid_json = excluded.grid_json", (json.dumps(GRID),))
# Classic contacts directory: Jose has the captured email the blob never got; Philip is blank.
c.execute("INSERT INTO contacts (id,first_name,last_name,email) VALUES "
"('c-phil','Philip','Treick',''),"
"('c-jose','Jose','Briones','jbriones@uwyo.edu'),"
"('c-jane','Jane','Doe','other@acme.com')") # differs from the blob's keep@acme.com
# Relational mirror (what sync_fundraising_relational would build): blank fc.email, linked contact_id.
c.execute("INSERT INTO fundraising_investors (id,investor_name,source_row_id,total_invested) VALUES "
"('inv-w','Wyoming','rowW',0),('inv-a','Acme Capital','rowA',0)")
c.execute("INSERT INTO fundraising_contacts (id,investor_id,full_name,email,sort_order,contact_id) VALUES "
"('fc-phil','inv-w','Philip Treick','',0,'c-phil'),"
"('fc-jose','inv-w','Jose Briones','',1,'c-jose'),"
"('fc-jane','inv-a','Jane Doe','',0,'c-jane')")
c.commit()
c.close()
def main():
server.init_db()
seed()
token = server.create_token("u1", "grant", "admin")
httpd = ThreadingHTTPServer(("127.0.0.1", 0), _Quiet)
port = httpd.server_address[1]
threading.Thread(target=httpd.serve_forever, daemon=True).start()
try:
st, d = _get_state(port, token)
rows = ((d or {}).get("data", {}).get("grid", {}) or {}).get("rows", [])
by_id = {r.get("id"): r for r in rows}
w = by_id.get("rowW", {})
a = by_id.get("rowA", {})
wc = w.get("contacts", [])
ac = a.get("contacts", [])
print("\n[heal: blank pill email filled from the linked contact (Jose)]")
jose = next((c for c in wc if c.get("name") == "Jose Briones"), {})
check(st == 200 and jose.get("email") == "jbriones@uwyo.edu",
f"Jose pill healed to jbriones@uwyo.edu (got {jose.get('email')!r})")
print("\n[heal: blank pill whose contact is also blank stays blank (Philip)]")
phil = next((c for c in wc if c.get("name") == "Philip Treick"), {})
check(phil.get("email", "") == "",
f"Philip pill stays blank (got {phil.get('email')!r})")
print("\n[heal: a pill that already has an email is never overwritten (Jane)]")
jane = next((c for c in ac if c.get("name") == "Jane Doe"), {})
check(jane.get("email") == "keep@acme.com",
f"Jane pill keeps its blob email, not the contact's (got {jane.get('email')!r})")
finally:
httpd.shutdown()
print()
if FAILS:
print(f"FAILED ({len(FAILS)}):")
for f in FAILS:
print(f" - {f}")
sys.exit(1)
print("ALL PASS (grid email heal)")
if __name__ == "__main__":
main()
+20
View File
@@ -75,6 +75,12 @@ GRID = {
"contacts": [{"name": "Charlie Brown", "email": "cb@brown.fund", "title": ""}]},
{"id": "rowBeta", "investor_name": "Beta Capital LLC", "notes": "",
"contacts": [{"name": "Pat Roe", "email": "pat@beta.com", "title": ""}]},
# Generic-descriptor decoys: share only "investment group" / "investments" with the
# Fortitude card below — must NOT surface as look-alikes (the 2026-06-20 false-positive fix).
{"id": "rowAether", "investor_name": "Aether Investment Group", "notes": "",
"contacts": [{"name": "Ada Ng", "email": "ada@aether.com", "title": ""}]},
{"id": "rowRussell", "investor_name": "Russell Investments", "notes": "",
"contacts": [{"name": "Russ Lee", "email": "russ@russell.com", "title": ""}]},
],
}
@@ -178,6 +184,20 @@ def main():
check(st == 200 and data.get("match") is None and data.get("candidates") == [],
f"unrelated query -> no match, no candidates (got {data})")
print("\n[fuzzy: shared generic words alone do NOT surface look-alikes (Fortitude vs Aether/Russell)]")
st, d = _req(port, "GET", "/api/intake/match?q=Fortitude%20Investment%20Group", token)
data = (d or {}).get("data", {})
cids = [c["id"] for c in data.get("candidates", [])]
check(data.get("match") is None and "rowAether" not in cids and "rowRussell" not in cids,
f"generic-only overlap -> no decoy candidates (got {data})")
print("\n[fuzzy: a shared DISTINCTIVE word still surfaces (Aether Capital ~ Aether Investment Group)]")
st, d = _req(port, "GET", "/api/intake/match?q=Aether%20Capital", token)
data = (d or {}).get("data", {})
cids = [c["id"] for c in data.get("candidates", [])]
check(data.get("match") is None and "rowAether" in cids,
f"distinctive overlap -> rowAether candidate (got {data})")
print("\n[match: missing q and email -> 400]")
st, _ = _req(port, "GET", "/api/intake/match", token)
check(st == 400, f"no params -> 400 (got {st})")
+20
View File
@@ -69,6 +69,26 @@ Spark). See *Fuzzy matching* below. Tests green (27/27 backend + the offline bot
A bare `yes`/`no` typed **top-level** (not in the thread) while a proposal is pending gets a
"reply in the thread" redirect (`store.any_pending()` guard in `handle_intake`), not a
misparsed new intake.
5. **On a conclusive decision (approve or reject) the whole thread is redacted** (Grant, 2026-06-20)
— exactly like the email-review room. `handle_reply`/`handle_disambiguation` call
`redact_thread(intake_room, root)` instead of posting a `✅`/`🗑️` confirmation: it clears the
bot's card + `📇 Reading…` ack + the main-timeline **nudge** + **the user's own note/photo** (the
thread root). The thread vanishing is the acknowledgment; a confirmation reply would just keep it
alive. Only conclusive decisions clear — an `edit`/NL-revise keeps the thread (still pending), and
a commit **failure** posts a `⚠️` and restores the proposal (no redact, so the user can retry).
`redact_thread` now takes the **room** as its first arg and matches replies by `m.thread`
event_id **or** `m.in_reply_to` event_id (so the un-threaded nudge clears too). **Prerequisite:
the bot needs a `redact`/moderator power level in the intake room** to clear the *humans'*
messages (its own need none) — without it the user's note/photo lingers (best-effort, no error).
Same Element caveat as email: turn OFF "show removed messages" so the placeholders disappear.
**Trade-off:** there's no longer an in-Matrix confirmation of *what* was logged after a successful
commit (the record is in the CRM) — by design, matching the email room; revisit if wanted.
**Retroactive backfill:** `backend/matrix_intake/redact_intake.py` (dry-run default; `--apply`)
clears the room's pre-existing backlog. The intake room keeps **no durable pending state** (the
proposal store is in-memory, lost on restart), so every message in it is stale after a restart and
safe to redact — it clears **every** `m.room.message` (text + card images), bot's and humans'
alike. Run on the Spark: `docker compose run --rm matrix-intake python -u
backend/matrix_intake/redact_intake.py [--apply]`.
## Business-card capture (M3 — image intake)
+128
View File
@@ -0,0 +1,128 @@
# Plan: in-app business-card intake (mobile camera → transcribe → approve)
**Status:** PLAN — awaiting Grant's go-ahead before building (requested 2026-06-20).
**Goal:** a camera button in the mobile top bar (left of the quick-log pencil) → take a photo or
pick one from the library → the **same** vision-transcribe → parse → fuzzy-match → edit/approve/reject
flow the Matrix card intake runs (M3), surfaced as an inline mobile sheet instead of a Matrix thread.
## Why this is cheaper than it looks
The reusable core is already nio-free and already reachable from the CRM monolith:
- `server.py` **already** imports the Spark client: `sys.path.insert(.../ingest); import llm`
(`server.py:6485`). `llm.chat_vision` (multimodal → Spark Control passthrough) is the exact call
the bot's card OCR uses.
- `backend/matrix_intake/parse.py` imports only `json`, `re`, `spark`; `spark.py` imports only
`os`, `sys`, `llm`. **Neither imports `matrix-nio`.** So `server.py` can import `parse` +
`spark` the same way it imports `llm`, with **no bot refactor and no matrix-nio in the CRM**.
- Match (`find_intake_match` / `find_intake_candidates`) and the write
(`/api/fundraising/log-communication`, create-if-missing + contact upsert + audit) are **already**
in `server.py`. The fuzzy matcher just got the generic-word fix (v-next).
- The whole `backend/` tree (incl. `ingest/` + `matrix_intake/`) is `COPY`'d into the s9pk image
(`start9/0.4/Dockerfile:62`), so the imports resolve on the box at runtime.
So this is **one new endpoint + one new mobile component**, reusing everything downstream. No
migration, no new dependency, no change to the live Matrix bot.
## Architecture decision
**Recommended (Option A): `server.py` imports `matrix_intake.parse` + `matrix_intake.spark`
directly.** Add `backend/matrix_intake` to `sys.path` (guarded like the existing `ingest` insert)
and `import parse, spark`. Pros: zero bot churn, reuses the tested transcribe + email-integrity
parse verbatim, no divergence. Con: the CRM now imports two modules out of the bot package — a mild
coupling, but they're pure (no nio, no CRM HTTP; we do NOT import the bot's `crm_client`).
**Alternative (Option B): extract a shared `backend/intake_core/`** (`vision.py` = transcribe,
`extract.py` = parse/normalize/revise) that both the bot and `server.py` import. Cleaner ownership,
but it refactors the live, working bot for no functional gain right now. **Defer** unless the
coupling in A bites later (e.g. if the bot's `parse` grows nio-coupled helpers).
> Module-name caution (from the matrix-intake guide): the bot imports by **bare name** (`import
> spark`, `import llm`, `import config`). `server.py` already has `ingest/` on its path (so `llm`,
> `config`, `http_util` resolve). Adding `matrix_intake/` lets `parse`/`spark` resolve with **no
> name overlap** against `ingest`'s modules — verified. Import only `parse` + `spark` (not
> `crm_client`/`settings`/`bot`).
## Server: one new endpoint
`POST /api/intake/card` — authenticated **member+** (it's a UI feature for the team; NOT bot-gated,
NOT admin-only). Body is JSON, image as base64 (no multipart parsing in the stdlib handler):
```
{ "image_b64": "<base64>", "mime": "image/jpeg" }
→ 200 { "transcription": "...",
"proposal": { investor_name, contact_name, contact_email, contact_title,
city, linkedin_url, phone, mobile, note },
"match": { id, name } | null,
"candidates": [ { id, name, score, matched_on }, ... ] }
```
Handler flow (mirrors `bot.handle_card``handle_intake`, minus Matrix):
1. Auth (reuse the standard authenticated gate).
2. Decode + size-guard the base64; default mime `image/jpeg`.
3. `transcription = spark.transcribe_card(image_b64, mime)` (Spark Control vision). On error → 502
`{ ok:false, reason:"vision_unavailable" }`; on `<5` chars → 200 `{ ok:false, reason:"unreadable" }`.
4. `proposal = parse.parse_message("New investor — from a business card:\n"+transcription, roster=None)`
— the **email-integrity rule rides along** (an address/phone/LinkedIn is kept only if it literally
appears in the transcription, never minted), exactly as in Matrix. `roster=None` for v1 (the
team-roster framing is a Spark-side env; the box can pass None — prior parse behavior).
5. `match = find_intake_match(...)`, `candidates = find_intake_candidates(...)` from the proposal's
`investor_name`/`contact_email`.
6. Return `{ transcription, proposal, match, candidates }`.
**Approve** reuses the existing write — the mobile sheet POSTs the (possibly-edited) proposal to
`POST /api/fundraising/log-communication` with `create_investor_if_missing` (new) or the picked
`_match_id`/candidate row (existing), tagged **`source="app_card"`** (a new provenance value
distinct from `matrix_card`/`matrix_intake`). **Reject** is client-only (discard the sheet) — no
server call, nothing written.
**No NL-edit for v1.** In a form UI, editing the fields directly is easier than chatting "change the
email to…". The fields are inline-editable in the sheet. (A `/api/intake/card/revise` wrapping
`parse.revise` is a possible later enhancement if Grant wants conversational corrections.)
## Mobile UI: one new component
`MobileCardCapture` in the shell top bar, **left of `MobileQuickLog`** (the cluster at
`index.html:14609`). Reuses `<BottomSheet>`, `.sheet-input`, `StageChip`, and the New-investor
sheet patterns (8g).
- **Trigger:** a camera-icon button + a hidden `<input type="file" accept="image/*">`
**omit `capture`** so iOS shows the native action sheet (Take Photo / Photo Library / Browse),
satisfying "take a photo OR use an existing photo." (`capture="environment"` would force the camera
and skip the library — not what we want.)
- **On select:** read the file as a data URL; **downscale via `<canvas>` to ~2000px max dimension**
before base64 (native, no library) — keeps the payload under the StartOS reverse-proxy body cap
(the matrix-intake guide's known `413` risk) and the model downscales to ~2 MP anyway. Don't
over-shrink (hurts OCR). Show a "📇 Reading the card…" loading state.
- **POST** to `/api/intake/card`; on `unreadable`/`vision_unavailable` show a "try a clearer,
well-lit, fill-the-frame photo" message with a Retake button (same UX as the bot's `📇` replies).
- **Approval sheet** (the New-investor sheet, pre-filled from `proposal`): editable investor name +
contact name/email/title/city/phone/mobile/LinkedIn; a small "Existing investor?" block when
`match`/`candidates` came back (pick a candidate row, or "Add as new"); **Approve** → the
log-communication write; **Reject/Retake** → discard.
- **Icon:** reuse the proven `.quicklog-btn svg` sizing fix (explicit CSS `width/height` + `flex:none`)
so the new camera SVG doesn't hit the same iOS sole-flex-child collapse.
## Scope / effort / risk
- **Server:** ~1 endpoint + the `parse`/`spark` import guard + an `app_card` source tag. Small.
- **Frontend:** ~1 component (camera button + capture input + canvas downscale + approval sheet) +
CSS for the button. Medium (the approval sheet is the New-investor sheet pre-filled).
- **No** migration, **no** new dependency, **no** Matrix-bot change.
- **Tests:** an endpoint test that stubs `spark.transcribe_card` (like `test_spark.py` does) and
asserts the transcribe→parse→match shape + the email-integrity passthrough; the real vision/OCR
path stays **live-smoke only** (same as Matrix M3). Frontend is inspection + on-device.
- **Risks:** (1) large-image upload vs the reverse-proxy cap — mitigated by client-side canvas
downscale; (2) iOS file-input behavior across Safari/standalone-PWA — verify on-device; (3) OCR
accuracy is the **shared** Matrix limitation (resolution-bound; "fill the frame"), not new here;
(4) the `parse`/`spark` import must be lazy/guarded so a dev `./start.sh` without Spark reachable
still boots (the endpoint just 502s) — mirror the existing lazy `import llm` at `server.py:6485`.
## Open questions for Grant (resolve before building)
1. **Provenance tag**`source="app_card"` OK (distinct from `matrix_card`)?
2. **v1 = form-field edits only** (no conversational NL-edit on the card) — agreed? NL-edit can
follow if wanted.
3. **Who can use it** — any authenticated member (recommended), or restrict?
4. **Ships in the s9pk** (server endpoint + frontend) — so it needs a version bump + build + install,
unlike the bot-only M3. Confirm that's the intended delivery path.
+50 -9
View File
@@ -2281,6 +2281,14 @@
.sort-row-label { font-size: 15px; font-weight: 500; color: var(--text-primary); }
.sort-row-hint { font-family: 'IBM Plex Mono', monospace; font-size: 11px; color: var(--text-subtle); }
.sort-row-check { flex: none; color: var(--accent); font-size: 15px; }
/* Reversible-direction control under a selected sort row (e.g. Staleness oldest/newest). */
.sort-dir { display: flex; gap: 8px; padding: 0 4px 2px; }
.sort-dir-opt {
flex: 1; cursor: pointer; font-family: inherit; font-size: 13px; font-weight: 500;
min-height: 40px; padding: 0 12px; border-radius: 9px;
border: 1px solid var(--border); background: var(--bg-input); color: var(--text-secondary);
}
.sort-dir-opt.active { border-color: var(--accent); color: var(--accent); background: var(--bg-panel-elevated); }
/* AZ directory: sticky letter headers over a card list. */
.az-header {
@@ -2560,6 +2568,11 @@
display: inline-flex; align-items: center; justify-content: center;
}
.quicklog-btn:active { background: var(--bg-hover); }
/* iOS Safari collapses an inline <svg> that is the sole, centered flex child of a
fixed-size flex button when the svg carries only width/height *attributes* (it renders
as a ~1px dot). Explicit CSS dimensions + flex:none fix it — the same reason the working
.bottom-tab-icon (sized box) and .sort-pill (flex:none + text) icons render. */
.quicklog-btn svg { width: 18px; height: 18px; flex: none; }
.quicklog-hint { font-size: 13px; color: var(--text-subtle); line-height: 1.5; margin: 0 0 12px; }
.quicklog-pool { display: flex; flex-direction: column; gap: 8px; margin-top: 12px; }
.quicklog-empty { font-size: 13px; color: var(--text-subtle); padding: 16px 4px; }
@@ -4445,19 +4458,39 @@
{label}
</button>
);
const SortSheet = ({ open, onClose, title, options, value, onPick }) => (
// Optional `dir`/`onToggleDir` add a reversible-direction control under the selected option
// when that option is `reversible` (e.g. the grid's Staleness — oldest vs most-recent first).
// Callers that omit onToggleDir behave exactly as before (every row taps-and-closes).
const SortSheet = ({ open, onClose, title, options, value, onPick, dir, onToggleDir }) => (
<BottomSheet open={open} onClose={onClose} title={title}>
<div className="sort-list">
{options.map((o) => (
<button key={o.id} type="button" className={`sort-row ${value === o.id ? 'active' : ''}`}
onClick={() => { onPick(o.id); onClose(); }}>
{options.map((o) => {
const reversible = o.reversible && !!onToggleDir;
return (
<React.Fragment key={o.id}>
<button type="button" className={`sort-row ${value === o.id ? 'active' : ''}`}
onClick={() => { onPick(o.id); if (!reversible) onClose(); }}>
<span className="sort-row-main">
<span className="sort-row-label">{o.label}</span>
{o.hint && <span className="sort-row-hint">{o.hint}</span>}
</span>
{value === o.id && <span className="sort-row-check"></span>}
</button>
))}
{value === o.id && reversible && (
<div className="sort-dir" role="group" aria-label="Sort direction">
<button type="button" className={`sort-dir-opt ${dir !== 'asc' ? 'active' : ''}`}
onClick={() => { onToggleDir('desc'); onClose(); }}>
{(o.dirLabels && o.dirLabels.desc) || 'Descending'}
</button>
<button type="button" className={`sort-dir-opt ${dir === 'asc' ? 'active' : ''}`}
onClick={() => { onToggleDir('asc'); onClose(); }}>
{(o.dirLabels && o.dirLabels.asc) || 'Ascending'}
</button>
</div>
)}
</React.Fragment>
);
})}
</div>
</BottomSheet>
);
@@ -4468,7 +4501,8 @@
{ id: 'name', pill: 'Name', label: 'Name', hint: 'A → Z' },
{ id: 'stage', pill: 'Stage', label: 'Pipeline stage', hint: 'Lead → Commitment' },
{ id: 'committed', pill: 'Committed', label: 'Committed', hint: 'Most first' },
{ id: 'staleness', pill: 'Staleness', label: 'Last contact', hint: 'Most stale first' },
{ id: 'staleness', pill: 'Staleness', label: 'Last contact', hint: 'Most stale first', reversible: true,
dirLabels: { desc: 'Most stale first', asc: 'Most recent first' } },
{ id: 'priority', pill: 'Priority', label: 'Priority', hint: 'Flagged first' },
];
// Pipeline sorts within a stage (dc PipelineApp:580). "Staleness" uses the opp's updated_at as
@@ -10166,6 +10200,7 @@
const [selectedId, setSelectedId] = useState(null);
const [sheet, setSheet] = useState(null); // 'view' | 'create' | 'note' | 'stage' | 'reminder' | 'sort'
const [sortKey, setSortKey] = useState('name'); // GRID_SORTS
const [sortDir, setSortDir] = useState('desc'); // only the reversible sorts (Staleness) read this
const [busy, setBusy] = useState(false);
const [createForm, setCreateForm] = useState({ name: '', contactName: '', contactEmail: '', note: '', priority: false, stage: '', reminderTitle: '', reminderDue: '' });
const [reminderForm, setReminderForm] = useState({ title: '', due_date: '', details: '' });
@@ -10263,6 +10298,8 @@
const byName = (a, b) => String(a.investor_name || '')
.localeCompare(String(b.investor_name || ''), undefined, { sensitivity: 'base' });
const staleDays = (r) => { const d = daysSince(r.last_activity_at); return d == null ? Number.MAX_SAFE_INTEGER : d; };
// Direction multiplier for the reversible sorts; 'desc' keeps each key's natural default.
const dirMul = sortDir === 'asc' ? -1 : 1;
const cmp = {
name: byName,
stage: (a, b) => {
@@ -10270,11 +10307,13 @@
return (oi(a) - oi(b)) || byName(a, b);
},
committed: (a, b) => (gridRollup(b, fundColumnIds) - gridRollup(a, fundColumnIds)) || byName(a, b),
staleness: (a, b) => (staleDays(b) - staleDays(a)) || byName(a, b), // larger days first = most stale first
// default (desc): larger days first = most stale first; asc flips to most-recent first.
// Name stays the ascending tiebreak regardless of direction.
staleness: (a, b) => (dirMul * (staleDays(b) - staleDays(a))) || byName(a, b),
priority: (a, b) => ((b.priority ? 1 : 0) - (a.priority ? 1 : 0)) || byName(a, b), // row.priority: boolean
};
return [...searched].sort(cmp[sortKey] || byName);
}, [rows, activeViewObj, columns, fundColumnIds, search, sortKey]);
}, [rows, activeViewObj, columns, fundColumnIds, search, sortKey, sortDir]);
const selectedRow = useMemo(() => rows.find((r) => r.id === selectedId) || null, [rows, selectedId]);
const closeSheet = () => setSheet(null);
@@ -10533,7 +10572,9 @@
)}
<SortSheet open={sheet === 'sort'} onClose={closeSheet} title="Sort investors"
options={GRID_SORTS} value={sortKey} onPick={setSortKey} />
options={GRID_SORTS} value={sortKey}
onPick={(id) => { setSortKey(id); setSortDir('desc'); }}
dir={sortDir} onToggleDir={setSortDir} />
<BottomSheet open={sheet === 'view'} onClose={closeSheet} title="Views">
{views.map((v) => (
+3 -2
View File
@@ -63,8 +63,9 @@ export const PACKAGE_TITLE = 'Ten31 Database'
// * 0.1.0:95 (mobile-first redesign goes live + installable PWA. The Grid, Pipeline, Reminders & Contacts screens are touch-native on phones below 768px [safe-area bottom-tab nav, card lists, drag-dismiss bottom sheets, swipe actions, full-screen Grid detail, SVG tab icons + ·Ten31· wordmark], with an app-wide light theme + toggle. Installable home-screen PWA: manifest.webmanifest [standalone display, #0b1118 theme] + square/apple-touch icons + a pre-auth /manifest.webmanifest route; iOS-first, no service worker. Pipeline funnel v2: 4-stage lead→engaged→diligence→commitment [in-app migration 0007] with derived grid signals [pipeline_stage/existing_investor/recency] injected-on-GET, stripped-on-write. Desktop UI unchanged; no LLM path. Bundles the previously deploy-pending mobile Phases 08 + drag-reorder views + the PWA)
// * 0.1.0:96 (login page mobile/PWA conformance — the one surface the v95 mobile redesign skipped. CSS-only: 100vh→100dvh [dynamic viewport, fixes the centered card tucking under the iOS standalone status bar], a <768px media query adding 16px screen gutters + env[safe-area-inset] top/bottom clearance + touch-sized fields [inputs 46px/15px, button 46px], full-bleed card on small phones, and the §4 card depth shadow on the login card to match .section. No markup/JS/schema change; desktop login unchanged)
// * 0.1.0:97 (mobile top-bar polish + native zoom behaviour. Viewport meta gains maximum-scale=1 + user-scalable=no: kills pinch-zoom AND the iOS auto-zoom-on-focus that jerked the page in on every <16px input tap [app-wide, not just login]; OS accessibility zoom still works. Top-bar account initial now flex-centered + dc-aligned [IBM Plex Mono, accent-light, 13px — was defaulting to inline/baseline, off-center]. Quick-log pencil bumped --text-muted→--text-secondary for real affordance [the dc t3 grey thin-outline read as empty next to the color sun emoji on-device]. CSS-only; no JS/schema change)
// * Current: 0.1.0:98 (business-card intake [Matrix bot] captures a contact's phone, mobile/cell, city + LinkedIn from a scanned card onto the contact record — cell to Mobile, office to Phone, fax skipped. Server half: _upsert_contact_from_fundraising now accepts phone+mobile on the contact dict [city+linkedin already worked]. The bot's transcription/extraction/card changes ship on the Spark [git pull + rebuild]. No schema change [contacts columns already exist]; no user-facing CRM change)
export const PACKAGE_VERSION = '0.1.0:98'
// * 0.1.0:98 (business-card intake [Matrix bot] captures a contact's phone, mobile/cell, city + LinkedIn from a scanned card onto the contact record — cell to Mobile, office to Phone, fax skipped. Server half: _upsert_contact_from_fundraising now accepts phone+mobile on the contact dict [city+linkedin already worked]. The bot's transcription/extraction/card changes ship on the Spark [git pull + rebuild]. No schema change [contacts columns already exist]; no user-facing CRM change)
// * Current: 0.1.0:99 (Grant device-test round 2, CRM half: intake fuzzy match scores DISTINCTIVE tokens only [no more "Investment Group"/"Capital"/"Family Office" false look-alikes]; mobile grid "Last contact"/staleness sort is reversible; mobile Edit-investor prefills a contact's email [GET /api/fundraising/state heals a blank grid pill from the linked classic contact, fill-only]; mobile quick-log pencil icon renders [CSS sizing on the sole flex-child svg]. The Matrix intake thread-redaction change ships on the Spark, not here. No schema change; no migration)
export const PACKAGE_VERSION = '0.1.0:99'
export const DATA_MOUNT_PATH = '/data'
export const WEB_PORT = 8080
+2 -2
View File
@@ -61,6 +61,6 @@ import { v_0_1_0_97 } from './v0.1.0.97'
import { v_0_1_0_98 } from './v0.1.0.98'
export const versionGraph = VersionGraph.of({
current: v_0_1_0_98,
other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73, v_0_1_0_74, v_0_1_0_75, v_0_1_0_76, v_0_1_0_77, v_0_1_0_78, v_0_1_0_79, v_0_1_0_80, v_0_1_0_81, v_0_1_0_82, v_0_1_0_83, v_0_1_0_84, v_0_1_0_85, v_0_1_0_86, v_0_1_0_87, v_0_1_0_88, v_0_1_0_89, v_0_1_0_90, v_0_1_0_91, v_0_1_0_92, v_0_1_0_93, v_0_1_0_94, v_0_1_0_95, v_0_1_0_96, v_0_1_0_97],
current: v_0_1_0_99,
other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73, v_0_1_0_74, v_0_1_0_75, v_0_1_0_76, v_0_1_0_77, v_0_1_0_78, v_0_1_0_79, v_0_1_0_80, v_0_1_0_81, v_0_1_0_82, v_0_1_0_83, v_0_1_0_84, v_0_1_0_85, v_0_1_0_86, v_0_1_0_87, v_0_1_0_88, v_0_1_0_89, v_0_1_0_90, v_0_1_0_91, v_0_1_0_92, v_0_1_0_93, v_0_1_0_94, v_0_1_0_95, v_0_1_0_96, v_0_1_0_97, v_0_1_0_98],
})
+22
View File
@@ -0,0 +1,22 @@
import { VersionInfo } from '@start9labs/start-sdk'
// Grant's real-phone device-test round 2 — four in-app fixes (CRM half; the Matrix intake
// thread-redaction change ships on the Spark, not here):
// - Intake fuzzy match no longer over-indexes on generic firm words ("Investment Group",
// "Capital", "Family Office"): _name_similarity scores DISTINCTIVE tokens only.
// - Mobile grid "Last contact" (staleness) sort is reversible (most-stale / most-recent first).
// - Mobile "Edit investor" prefills a contact's email: GET /api/fundraising/state heals a blank
// grid pill email from the linked classic contact (fill-only; next save persists it).
// - Mobile quick-log pencil icon renders (CSS sizing on the sole flex-child svg — iOS fix).
// No schema change — no migration.
export const v_0_1_0_99 = VersionInfo.of({
version: '0.1.0:99',
releaseNotes: {
en_US: [
'Mobile polish + intake fixes: reversible "Last contact" sort, Edit-investor now shows a',
"contact's saved email, the quick-log pencil icon renders, and business-card/intake",
'duplicate-matching ignores generic firm words like "Investment Group" / "Capital".',
].join(' '),
},
migrations: { up: async () => {}, down: async () => {} },
})