diff --git a/AGENTS.md b/AGENTS.md index 64732ac..9f60ca0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,7 +15,7 @@ - Optional runtime deps, used only if present: `bcrypt`, `PyJWT` (`jwt`), `cryptography` (Gmail module). - **MCP + ingest** (in the Docker image, not the bare CRM): `mcp==1.2.0` (FastMCP, `backend/mcp/server.py`), `fastembed==0.4.2`, `anthropic`, `cryptography==42.0.5`. - **Packaging:** StartOS 0.4, TypeScript SDK (`@start9labs/start-sdk`) under `start9/0.4/startos/`. Live target is `start9/0.4/`. -- **Local models** (bge-m3 embeddings, bge-reranker-v2-m3, `/api/search`, Qdrant): always via Spark Control. Contract: `docs/EMBEDDINGS.md`. +- **Local models** (bge-m3 embeddings, bge-reranker-v2-m3, `/api/search`, Qdrant): always via Spark Control. Contract: `docs/EMBEDDINGS.md`. The chat model (`CRM_CHAT_MODEL`, the daily-driver Qwen) is **vision-capable** — Spark Control's `/v1/chat/completions` is a dumb passthrough, so OpenAI **multimodal** `image_url` data-URIs work unchanged (used by the intake bot's business-card OCR; reuse `llm.chat_vision`). ## Commands @@ -108,11 +108,12 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude ## Current state -_**Box live at v0.1.0:97 (deployed 2026-06-20)** — the full mobile-first redesign (Phases 0–7 + P3b + drag-reorder + **8a–8i**) **+ installable PWA (Option A)** went live at v95; **v96** brought the login/first-admin screen into mobile/PWA conformance; **v97** is the first round of Grant's real-phone feedback — viewport zoom-lock (`maximum-scale=1` + `user-scalable=no`: no pinch, no iOS auto-zoom-on-focus for our <16px inputs; app-wide) + mobile top-bar fixes (account initial flex-centered & dc-aligned; quick-log pencil bumped to `--text-secondary` for affordance). All CSS-only, desktop untouched. **The fundraising grid + email capture is the canonical system of record.** Active thread: **mobile-first redesign DEPLOYED; Grant is now real-phone device-testing and reporting polish items** (v97 is the first batch). Build reference: `design/phase8-conformance.md` (the 8a–8i spec; fully landed — archive-eligible). History: git log + `start9/0.4/startos/versions/`._ +_**Box live at v0.1.0:98; v0.1.0:99 built + installing this session (2026-06-20)**. This session = **Grant's real-phone device-test round 2**: four in-app fixes (CRM half → v99 s9pk) + a Matrix intake-bot cleanup (Spark) + a written plan for in-app camera card intake. **The fundraising grid + email capture is the canonical system of record.** History: git log + `start9/0.4/startos/versions/`._ -- **Mobile redesign — 4 core surfaces built (Grid · Contacts · Pipeline · Reminders), each a rules-of-hooks-safe `useIsMobile()` → `Mobile*`/`Desktop*` pair (desktop untouched).** Foundation: bottom-tab bar + `:root` mobile vars; 4-stage enum; derived grid signals injected-on-GET/stripped-on-write at both points; mobile writes use **one-row endpoints only** (log-communication, pipeline link/stage, reminders, `update-row`) — never whole-grid PUT. -- **Phase 8 complete (8a–8i)** — all mobile cards/details/sheets/shell match the dc anatomy; the durable per-primitive record is the **Design convention's primitives list**, per-phase detail in git log. Installable PWA is a durable Conventions bullet (+ `ROADMAP.md` "Mobile PWA"). -- **Live (deployed):** **mobile zoom-lock + top-bar polish (v97, 2026-06-20)**; **login page mobile/PWA conformance (v96)**; **mobile-first redesign (Phases 0–8i) + light theme + installable PWA + 4-stage pipeline funnel (migration 0007) — all v95 (2026-06-20)**; W2 NL query (v94); W1 reminders (v93); grid Pipeline (v88); Matrix intake + Gmail capture (DWD) + daily digest; Thesis/Architect (dual-approval); outreach — all draft-only. -- **Tests:** **40/40 backend green** (`python3 backend/run_tests.py`), `py_compile` clean. Mobile surfaces interaction-verified via throwaway 375px jsdom harnesses (deleted after); harness recipe + the `Simulate.change` input gotcha live in the `mobile-surface-verification` memory. -- **Next — Grant's real-phone device-test (v97 is live):** the whole mobile set + PWA + login + top-bar fixes shipped 2026-06-20. **Confirm on v97:** tapping a field no longer zooms the page; pinch is disabled; top-bar **account initial is centered** and the **quick-log pencil is visible**. Plus the standing gate: light/dark across the 4 surfaces **+ login** (375px gutters, card not under the status bar in standalone), **install-to-home-screen → standalone launch** (Safari Share → Add to Home Screen; expect full-screen, dark status bar, T31 icon), safe-area/tab-bar clearance, swipe/sheet interactions (only smoke/jsdom-verified so far). Known PWA item to eyeball: the iOS status bar is fixed `black` (light-theme cosmetic seam — see Open/risks). -- **Open / risks:** all mobile work + light theme **deployed (v95/v96/v97) but real-phone device-test only just starting** (Grant); **viewport now `user-scalable=no`** — intentional native feel for this internal tool; OS accessibility zoom still works, but it's a known a11y trade-off if scope ever widens beyond the team; **quick-log pencil deviates from the dc `--t3` spec → `--text-secondary`** on Grant's "can't see it" report (the dc grey thin-outline reads as empty next to the color sun emoji; confirm on-device it now reads clearly); `.login-title` CSS (`index.html:1869`) is **dead** (defined, never rendered — login uses logo + subtitle; trivial removal candidate); `MobileDetailRow` unused-but-retained (legacy-usage sweep); Pipeline detail "Committed" tile shows grid-committed not deal-expected (forecast in a footnote); `handle_get_opportunity` (single-opp GET) deliberately does NOT inject `existing_investor`/`last_contact_date` — no surface needs it (the card uses the list injection; the detail derives `existing` from the contact fetch); W2 happy-path only; **Claude/Architect path unverified live on the box**; v2.0 reserve-asset spine **not canonical**; doc drift — `crm-overview.md`/`EVALUATION.md` still call `lp_profiles` live; **PWA iOS status bar is fixed `black` at launch** (`apple-mobile-web-app-status-bar-style`) — v96 fixed the *login-card* collision (it now respects `env(safe-area-inset-top)`), but the global header still gets a black strip above it in *light* theme (proper fix = `black-translucent` + header top-safe-area padding + theme-synced `theme-color`; deferred, validate on-device; dark is default so low-priority); PWA manifest/icons sent with no `Cache-Control` (consistent with all static routes — a manifest change post-install may be served stale by iOS until it re-fetches). +- **Device-test round 2 — 4 fixes shipped to v0.1.0:99 (CRM half).** (1) **Intake fuzzy match** no longer over-indexes on generic firm words — `_name_similarity` scores **distinctive** tokens only (generic descriptors like "Investment Group"/"Capital"/"Family Office" stripped via `_GENERIC_ORG_WORDS`); "Fortitude Investment Group" no longer surfaces Aether/Russell. (2) **Mobile grid "Last contact"/staleness sort is reversible** (`SortSheet` opt-in `dir`/`onToggleDir`; other surfaces untouched). (3) **Mobile "Edit investor" prefills a contact's email** — `GET /api/fundraising/state` heals a blank grid pill email from `fundraising_contacts.contact_id → contacts.email` (fill-only, by pill order then name; next one-row save persists it; `fundraising_contact_emails_by_row`). (4) **Quick-log pencil icon renders** — `.quicklog-btn svg { width;height;flex:none }` (iOS collapses a sole, centered, attribute-only-sized flex-child svg; the v97 fix only changed its color). +- **Matrix intake-bot — thread auto-delete on decision + retroactive purge (Spark, bot-only, NOT in the s9pk).** Approve/reject now `redact_thread(intake_room, root)` (clears card + ack + main-timeline nudge + the user's photo/note), mirroring the email-review room; the scan now also catches the un-threaded nudge (`m.in_reply_to`). New one-time `backend/matrix_intake/redact_intake.py` (dry-run default; `--apply`) clears the room backlog. **Needs the bot to hold a redact/moderator power level in the intake room** to clear users' messages (manual Element step). No more in-Matrix "✅ logged" confirmation after a commit (by design, like email). Detail: `docs/guides/matrix-intake.md`. +- **In-app camera card intake (#7) — PLAN written, not built.** `docs/handoffs/in-app-card-intake-plan.md`: reuses the nio-free transcribe/parse core (`server.py` already imports `llm`; `matrix_intake/parse.py`+`spark.py` are nio-free) → **one endpoint** `POST /api/intake/card` + **one mobile component** (camera button left of the pencil). No bot refactor, no new dep, no migration. **Awaiting Grant's call** on 4 decisions (provenance tag, form-edits-only v1, member access, ships-in-s9pk). +- **Mobile-first redesign — deployed (v95–v97), on-device test in progress.** 4 surfaces (Grid·Contacts·Pipeline·Reminders) + light theme + installable PWA + 4-stage funnel; desktop untouched. Standing gate: light/dark across surfaces + login (375px gutters, safe-area), install→standalone launch, swipe/sheet interactions (only jsdom-smoked). Other live features: W2 NL query (v94), W1 reminders (v93), grid Pipeline (v88), Gmail capture + daily digest, Thesis/Architect (dual-approval), outreach — all draft-only. **Business-card intake (M3, Matrix bot) LIVE since v98** (vision OCR via the daily-driver model; `source="matrix_card"`; captures name/email/title/city/LinkedIn/phone/mobile, integrity-checked). +- **Tests: 42/42 backend green** (`python3 backend/run_tests.py`), `py_compile` clean, frontend render-smoke green (`make render-smoke`). New: `test_grid_email_heal.py` + intake generic-word cases. Vision/OCR (Matrix + the planned in-app path) is **live-smoke only**. +- **Next:** (1) Grant device-tests the 4 v99 fixes on his phone (esp. the **pencil icon** — root-caused but device-confirm) + re-tests cards (OCR/phone mapping); (2) Grant gives the intake bot **mod power** in the room, then `redact_intake.py --apply` on the Spark; (3) Grant's call on the #7 plan; (4) finish the standing mobile on-device gate. +- **Open / risks:** **vision OCR can misread a character** on a card small-in-frame (resolution-bound — Spark Control downscales to ~2 MP; `mara.com→marac.com` reproduced at temp 0; mitigations: fill the frame, or a future client-side crop); **iPhone HEIC** may not decode in vLLM (most clients send JPEG); phone/mobile/LinkedIn land on the **contact record**, not the grid pill (by design — only city syncs to the pill); intake redaction needs the bot's room **mod power** or users' messages linger; **Claude/Architect path unverified live on the box**; v2.0 reserve-asset spine **not canonical**; **PWA iOS status bar fixed `black`** in light theme (header seam; deferred, dark is default); doc drift — `crm-overview.md`/`EVALUATION.md` still call `lp_profiles` live; assorted minor UI cleanups (`.login-title` dead CSS, `MobileDetailRow` unused, Pipeline "Committed" tile shows grid-committed) tracked in git history. diff --git a/ROADMAP.md b/ROADMAP.md index 3ab31ce..7d28135 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -103,19 +103,19 @@ **W3 — Bot grid-mutations behind a Matrix approval gate.** Generalize the email-proposal scaffold (`email_proposal_matrix` + propose→post→decide→apply) into one `agent_proposals` table (kind discriminator + JSON payload + target). Bot proposes set-commitment / assign-fund / change-stage / set-reminder; a human approves/edits/rejects in Matrix (**any member**); then apply. **Surgical, version-checked mutations — never blob RMW:** stage rides the existing `opportunities` link + validated stage endpoint; reminders write the W1 table; set-commitment/assign-fund need a version-checked single-cell upsert into the grid blob. Triggers the deferred **scoped service-token** item below (per-mutation-kind allowlist on the bot credential; money/merge/delete always require human approval regardless of scope — the autonomy axis). Parse on local Qwen, not Claude. -### Matrix-bridge intake for the fundraising grid — M1+M2 BUILT (deploy + live smoke pending) -*Requested 2026-06-16. **M1 (scaffold + parse + in-thread propose) and M2 (match + write-on-approve) built, tested (26/26), not yet deployed** — code in `backend/matrix_intake/`, guide at `docs/guides/matrix-intake.md`. Remaining: install `matrix-nio` + creds on the Spark, create the CRM bot user, and a **live Matrix smoke** (can't run in CI). M3 (business-card photo) deferred until Spark Control has a vision model. Next major build after this is **Pipeline adoption** (see below).* +### Matrix-bridge intake for the fundraising grid — M1+M2+M3 BUILT & LIVE +*Requested 2026-06-16. **M1+M2 live since v0.1.0:86 (2026-06-17); M3 (business-card photo) shipped & live 2026-06-20** — code in `backend/matrix_intake/`, guide at `docs/guides/matrix-intake.md`. M3 unblocked once the Spark Control daily-driver model became vision-capable: the bot OCRs a card via Spark Control's `/v1/chat/completions` multimodal passthrough (same `CRM_CHAT_MODEL`), then runs the existing intake flow; captures contact name/email/title/city/LinkedIn/phone/mobile (server half of phone/mobile = s9pk v0.1.0:98). Remaining: ongoing on-device card testing (OCR accuracy on small-in-frame cards). Next major build is **Pipeline adoption** (see below).* Use the **matrix-bridge** repo's pattern to listen on a dedicated ten31-database Matrix room. Send a message (with an optional business-card photo) and a local LLM **via Spark Control** parses it into the fundraising-grid schema and **auto-creates the investor entity + contact row**. For an existing investor, send a meeting note and it **appends an interaction-log entry**. Approval gate: the bot replies in Matrix with the proposed add/edit; the user approves / rejects / edits in-thread before the write commits (keeps the draft→human-approve guardrail). - Fits the "grid is canonical" direction (writes land in `fundraising_*`) and the never-send-autonomously rule (in-thread human approval before any write). **Locked design (2026-06-16, approved) — build now, M1 then M2:** - **Separate component, shared scaffold:** new `backend/matrix_intake/` (its own process; lifts matrix-bridge's connect/prime-then-listen/threaded-reply plumbing). `matrix-nio` is isolated to this component's `requirements.txt` — it never enters the stdlib CRM runtime. Keeps the CRM write credential + LP data out of the general-purpose matrix-bridge bot (blast-radius + data-sovereignty), and lets the two iterate independently. Runs on the Spark (placement settled against `standards/guides/placement.md` at deploy). -- **v1 = text-only.** Business-card photo deferred to M3 — Spark Control fronts chat/embeddings/rerank but **no vision model** today, so photo→fields isn't buildable end-to-end yet. +- **~~v1 = text-only~~ — M3 business-card photo SHIPPED (2026-06-20).** The Spark Control daily-driver model is now vision-capable (multimodal `image_url` passthrough), so card→text→fields works end-to-end. Transcribe-then-reuse (vision OCRs to text; the existing text extractor pulls fields) preserves the email/phone integrity rules. See the matrix-intake guide. - **Parse:** local Qwen via Spark Control `/v1/chat/completions` (temp 0, JSON-only), reusing the existing Spark client pattern (`backend/redaction`/`backend/ingest`). - **Approval handshake (the one stateful piece):** in-memory pending-proposal store keyed by Matrix thread root; user replies **yes / edit field=value / no** in-thread. Satisfies never-write-autonomously; exempt from "agents draft, humans send" (internal data entry, like the digest). - **CRM-side:** `POST /api/intake/investor` (service-auth) creates a new investor+contact **through the existing grid-save path** (so relational sync + audit + backup-on-write happen as with a UI edit; bot never does whole-blob RMW) or appends a meeting note to the interaction log for an existing investor; `GET /api/intake/match?q=` fuzzy-matches via the existing entity-resolution/email-matcher. New investor needs no fund at intake. -- **Phases:** M1 = scaffold + parse + in-thread propose, **no writes** (proves Matrix↔Spark). M2 = intake endpoint + match + write-on-approve + tests. M3 (deferred) = business-card photo. +- **Phases:** M1 = scaffold + parse + in-thread propose, **no writes** (proves Matrix↔Spark). M2 = intake endpoint + match + write-on-approve + tests. **M3 = business-card photo (SHIPPED 2026-06-20).** **Post-deploy enhancement — fuzzy match + in-thread confirm (Grant, 2026-06-17). DEPLOYED & LIVE 2026-06-17 (v0.1.0:86; box migration chain …85→86 clean, `candidates` endpoint verified); Matrix live-smoke pending.** Today `find_intake_match` is **exact-after-normalization** (`_normalize_text` = lowercase+strip), so near-misses — "Charlie" vs "Charles" (same last name), "Acme Capital" vs "Acme Capital LLC", a one-character email typo — return no match and the bot proposes a **new** investor, risking a duplicate the human approves without realizing a near-match exists. The existing in-thread approval gate is useless against this because the human is never *shown* the near-match. Fix: matcher returns **ranked fuzzy candidates** (deterministic pre-filter: normalized name similarity / token overlap + email edit-distance ≤ ~2), surfaced in-thread for the human to confirm or pick, with the **local Spark LLM optionally re-ranking/judging the shortlist** (good at Charlie/Charles + legal-suffix equivalence; fed only the shortlist, never the whole LP list). Keeps the approval gate but makes it effective against duplicates. Land **after** the live smoke — net-new logic + reply grammar + tests; the current exact match is safe and its failure mode (a duplicate) is recoverable via the existing entity-merge subsystem (`backend/entity_*.py`). - **As built:** `find_intake_candidates` in `server.py` (deterministic — stdlib `difflib` name similarity + token-set Jaccard, legal-suffix-aware via `_strip_legal_suffix`, + email Levenshtein ≤ 2; ranked, ≥0.62, top 5). `GET /api/intake/match` now returns `{match, candidates}`. Bot: a new `_stage="disambiguate"` shortlist (`proposals.render_disambiguation` / `interpret_disambiguation` / `attach_to_candidate` / `promote_to_new`) — human picks a number / `new` / `no`. **The optional LLM-judge re-rank was deliberately deferred** (the deterministic filter already surfaces the named cases; an LLM judge is the right *pruner* for shortlist noise — build if the deterministic ranking proves too noisy in practice). Tests: `test_intake_endpoints.py` (server fuzzy cases), `matrix_intake/test_proposals.py` (disambiguation grammar), `matrix_intake/test_crm_client.py` (candidate shape). @@ -129,6 +129,9 @@ Use the **matrix-bridge** repo's pattern to listen on a dedicated ten31-database **Long-term — extract the intake bot to its own repo (recommended, not yet done).** Containerizing from this monorepo is the pragmatic now-state, but the bot is a genuinely separate deployable (own process, own `matrix-nio` dep, own lifecycle); its only CRM coupling is the HTTP API (a clean network contract) plus ~40 lines of stdlib Spark client (cheap to vendor). The tell: the spark-control Update button would run `git reset --hard origin/main` on the **whole CRM clone** — wrong blast radius. `matrix-bridge` is already a dedicated repo; mirror it. The extraction is a migration (new Gitea repo, move code + tests + guide, vendor the client, re-point the Spark deploy), so it's deferred until worth the lift — do it *before* wiring the spark-control card if both land in the same push. +### In-app camera business-card intake — PLAN written, awaiting go-ahead (Grant, 2026-06-20) +*A camera button in the mobile top bar (left of the quick-log pencil) → take/choose a photo → the same vision-transcribe → parse → fuzzy-match → edit/approve/reject flow the Matrix card intake (M3) runs, surfaced as an inline mobile sheet. **Detailed plan: `docs/handoffs/in-app-card-intake-plan.md`.*** Key finding: the reusable core is nio-free and already reachable from the CRM (`server.py` imports `llm`; `matrix_intake/parse.py`+`spark.py` import no `matrix-nio`), so it's **one endpoint** (`POST /api/intake/card`) + **one mobile component**, no bot refactor / new dep / migration. Four decisions pending Grant before building: provenance tag (`app_card`?), form-edits-only for v1 (no conversational NL-edit), any-member access, ships-in-s9pk. Reuses the New-investor sheet pre-filled + the proven `.quicklog-btn svg` icon-sizing fix for the camera button. + ### Scoped service-credential auth path for automated CRM writers *Surfaced 2026-06-17 while deploying the Matrix intake bot. **Decision: defer — the bot uses a dedicated member username/password for now.** The CRM has no API-key/service-token path; its only auth is username+password → JWT. A dedicated **member** login is appropriately scoped against what matters operationally (no admin: can't manage users, reset data, or change settings) and unblocks the live smoke today.* diff --git a/backend/matrix_intake/bot.py b/backend/matrix_intake/bot.py index d60d80a..9699de1 100644 --- a/backend/matrix_intake/bot.py +++ b/backend/matrix_intake/bot.py @@ -171,9 +171,13 @@ async def main(): store.put(root, proposal) # commit failed — restore so the user can retry await say(room_id, f"⚠️ write failed, nothing committed: {exc}", root) return - await say(room_id, f"✅ {summary}", root) + # Committed → clear the whole thread (card + ack + nudge + the user's note/photo), + # like the email-review room. The thread vanishing is the acknowledgment; a confirmation + # reply would just keep it alive (and need redacting too). Needs the bot's redact/mod + # power in the intake room to clear the user's own messages — else those linger. + await redact_thread(room_id, root) elif action == "reject": - await say(room_id, "🗑️ Discarded — nothing written.", root) + await redact_thread(room_id, root) elif action == "edit": field, value = payload proposal = proposals.apply_edit(proposal, field, value) @@ -212,42 +216,49 @@ async def main(): await say(room_id, "➕ OK — adding as a new investor:\n\n" + proposals.render(updated), root) elif action == "reject": - await say(room_id, "🗑️ Discarded — nothing written.", root) + await redact_thread(room_id, root) # discard → clear the thread, like an approve else: # unrecognized — re-show the shortlist store.put(root, proposal) await say(room_id, "I didn't catch that.\n\n" + proposals.render_disambiguation(proposal), root) - async def redact_card(event_id): - """Redact one event (best-effort). Redacting our OWN message needs no special power; - redacting someone else's reply needs the bot to hold a redact/mod power level.""" + async def redact_card(room_id, event_id): + """Redact one event in `room_id` (best-effort). Redacting our OWN message needs no special + power; redacting someone else's message (a human reply, or the user's original card photo / + intake note) needs the bot to hold a redact/mod power level in that room.""" try: - await client.room_redact(review_room, event_id, reason="proposal resolved") + await client.room_redact(room_id, event_id, reason="proposal resolved") except Exception as exc: print(f"matrix-intake: could not redact {event_id}: {exc}", flush=True) - async def redact_thread(root): - """Clear a resolved thread: redact the card AND every reply under it, so the thread drops - out of the threads view (not just the main timeline). The card is ours (always redactable); - the human's yes/no reply needs the bot's redact/mod power — if it lacks power that redact - just no-ops and the reply lingers. Finds replies by scanning recent room history for - m.thread events pointing at this root (the triggering reply is already synced, so a - backward scan from the current token includes it).""" - await redact_card(root) + async def redact_thread(room_id, root): + """Clear a resolved thread in `room_id`: redact the root AND every message that hangs off it + — the m.thread children (cards/acks/human replies) AND the main-timeline **nudge** (a plain + m.in_reply_to reply, not a thread child), so the thread drops out of both the threads view + and the timeline. For email-review the root is the bot's card; for intake it's the USER'S + own note/photo, so clearing it (and the human reply) needs the bot's redact/mod power in that + room — without it those just no-op and linger. Replies are found by scanning recent history + from the current sync token (the triggering reply is already synced, so a backward scan + includes it).""" + await redact_card(room_id, root) token = getattr(client, "next_batch", None) if not token: return try: scanned = 0 for _ in range(MAX_THREAD_SCAN_PAGES): - resp = await client.room_messages(review_room, start=token, + resp = await client.room_messages(room_id, start=token, direction=MessageDirection.back, limit=100) chunk = getattr(resp, "chunk", None) if not chunk: break for ev in chunk: rel = ((getattr(ev, "source", None) or {}).get("content", {}) or {}).get("m.relates_to") or {} - if rel.get("rel_type") == "m.thread" and rel.get("event_id") == root: - await redact_card(ev.event_id) + in_reply = (rel.get("m.in_reply_to") or {}).get("event_id") + # A thread child carries event_id==root; the un-threaded nudge carries only + # m.in_reply_to.event_id==root. Catch both so the thread AND its main-timeline + # pointer clear together. + if rel.get("event_id") == root or in_reply == root: + await redact_card(room_id, ev.event_id) token = getattr(resp, "end", None) scanned += len(chunk) if not token or scanned > 1000: @@ -275,7 +286,7 @@ async def main(): return # Success → clear the whole thread (card + replies). No confirmation: the thread # vanishing is the acknowledgment, and a confirmation reply would keep it alive. - await redact_thread(root) + await redact_thread(review_room, root) elif decision == "reject": email_threads.pop(root, None) try: @@ -284,7 +295,7 @@ async def main(): email_threads[root] = item await say(room_id, email_proposals.frame(f"⚠️ couldn't dismiss it ({str(exc)[:200]}). Try again."), root) return - await redact_thread(root) + await redact_thread(review_room, root) else: try: new_note = await asyncio.to_thread(email_proposals.revise_note, item.get("note") or "", text) @@ -332,7 +343,7 @@ async def main(): if not ev: continue try: - await redact_thread(ev) + await redact_thread(review_room, ev) await asyncio.to_thread(crm_client.mark_email_proposal_closed, it["id"]) email_threads.pop(ev, None) except Exception as exc: diff --git a/backend/matrix_intake/redact_intake.py b/backend/matrix_intake/redact_intake.py new file mode 100644 index 0000000..f8fadd4 --- /dev/null +++ b/backend/matrix_intake/redact_intake.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""One-time maintenance: clear the intake room's backlog of resolved/stale messages. + +Going forward the bot redacts each intake thread when it's approved/rejected (bot card + ack + +nudge + the user's own note/photo). This clears the messages that piled up BEFORE that shipped. + +The intake room is single-purpose and the bot keeps **no durable pending state** (its proposal +store is in-memory and is lost on every restart), so nothing in the room is "still live" after a +restart — every message in it is safe to redact. This walks the room history and redacts every +m.room.message event (text + business-card images), bot's and humans' alike. + +Redacting another user's message (the humans' notes/photos) needs the bot to hold a **redact / +moderator power level** in the intake room — without it those just no-op and linger (the bot's own +messages still clear). Make the bot a moderator of the intake room in Element first. + +Safe by default: prints what it WOULD redact and does nothing. Pass --apply to actually redact. +Run on the Spark via the bot's own creds/image: + docker compose run --rm matrix-intake python -u backend/matrix_intake/redact_intake.py + docker compose run --rm matrix-intake python -u backend/matrix_intake/redact_intake.py --apply +""" +import asyncio +import sys + +from nio import AsyncClient, MessageDirection + +import settings + +MAX_PAGES = 50 # 50 * 100 events is far more history than this room holds + + +async def main(apply): + mx = settings.matrix_settings() + intake_room = mx.get("intake_room") + if not intake_room: + print("MATRIX_INTAKE_ROOM is not set — nothing to do.") + return + client = AsyncClient(mx["homeserver"], mx["user_id"]) + client.restore_login(user_id=mx["user_id"], device_id=mx["device_id"], access_token=mx["token"]) + try: + sync = await client.sync(timeout=10000, full_state=False) + token = sync.next_batch + targets = [] # (event_id, label) + seen = set() + for _ in range(MAX_PAGES): + resp = await client.room_messages(intake_room, start=token, + direction=MessageDirection.back, limit=100) + chunk = getattr(resp, "chunk", None) + if not chunk: + break + for ev in chunk: + src = getattr(ev, "source", None) or {} + if src.get("type") != "m.room.message": + continue # only chat messages + images; leave membership/state events alone + eid = getattr(ev, "event_id", None) + if not eid or eid in seen: + continue + seen.add(eid) + content = src.get("content") or {} + if not content: + continue # already redacted (content stripped) — skip + msgtype = content.get("msgtype") or "?" + body = (content.get("body", "") or "").replace("\n", " ") + who = "bot " if getattr(ev, "sender", None) == mx["user_id"] else "user" + targets.append((eid, f"{who} [{msgtype}] {body[:60]}")) + token = getattr(resp, "end", None) + if not token: + break + + print(f"messages to clear in the intake room: {len(targets)}") + fails = 0 + for eid, label in targets: + print(("APPLY redact " if apply else "WOULD redact ") + eid + " :: " + label) + if apply: + r = await client.room_redact(intake_room, eid, reason="retroactive intake-room cleanup") + if not hasattr(r, "event_id"): + fails += 1 + print(f" ! redact failed (need mod power for others' messages?): {r}") + print(("done — redacted " if apply else "dry run — would redact ") + + f"{len(targets) - (fails if apply else 0)}/{len(targets)} event(s)" + + (f"; {fails} failed" if apply and fails else "") + ".") + finally: + await client.close() + + +if __name__ == "__main__": + asyncio.run(main(apply="--apply" in sys.argv[1:])) diff --git a/backend/server.py b/backend/server.py index 2ce23f4..6ff733b 100644 --- a/backend/server.py +++ b/backend/server.py @@ -1305,14 +1305,40 @@ def _strip_legal_suffix(normalized_name): return " ".join(toks) +# Generic firm-descriptor words that carry almost no identifying signal: nearly every firm name +# contains one ("… Investment Group", "… Capital", "… Family Office"). Two names that overlap ONLY +# on these are NOT duplicates — 'Fortitude Investment Group' is not 'Aether Investment Group'. We +# compare on the DISTINCTIVE remainder so a shared descriptor can't inflate the score (the earlier +# "Capital/Ventures/Partners are distinctive enough to keep" assumption produced false shortlists — +# Grant, 2026-06-20). If a name is ALL descriptor ('Family Office'), we fall back to its full tokens +# so there's still something to compare. +_GENERIC_ORG_WORDS = frozenset({ + "investment", "investments", "investing", "investor", "investors", + "capital", "ventures", "venture", "partners", "partner", "group", + "fund", "funds", "management", "advisors", "advisers", "advisory", + "asset", "assets", "holdings", "holding", "family", "office", + "trust", "associates", "equity", "financial", "finance", "global", + "international", "company", "enterprises", "wealth", "the", "and", "of", +}) + + +def _distinctive_tokens(normalized_name): + """Tokens of a (legal-suffix-stripped) name with generic firm descriptors removed. Falls back to + the full token list when the name is nothing but descriptors, so an all-generic name still compares.""" + toks = re.findall(r"[a-z0-9]+", normalized_name) + keep = [t for t in toks if t not in _GENERIC_ORG_WORDS] + return keep or toks + + def _name_similarity(a, b): """0..1 fuzzy similarity between two investor names: the max of difflib's sequence ratio (catches near-spellings — 'Charlie'/'Charles') and token-set Jaccard overlap (catches word-order differences). Legal-entity suffixes are stripped first, so two names differing only by 'LLC'/'LP'/'Inc' score 1.0 (a near-certain duplicate to surface — find_intake_match - won't have caught it, since it compares the full string). Favors recall: a shared common - name-word ('… Capital') can lift unrelated firms into the 0.6–0.8 band — acceptable noise in - a ranked, human-confirmed shortlist; semantic pruning is the deferred LLM-judge's job.""" + won't have caught it, since it compares the full string). Both the ratio and the Jaccard run on + the DISTINCTIVE tokens (generic descriptors like 'Investment Group'/'Capital' removed), so firms + that share only a descriptor don't surface as look-alikes; 'Aether Capital' ~ 'Aether Capital + Partners' still scores 1.0 on the distinctive 'aether'. Still recall-favoring on real overlap.""" a = _normalize_text(a) b = _normalize_text(b) if not a or not b: @@ -1323,9 +1349,10 @@ def _name_similarity(a, b): sb = _strip_legal_suffix(b) or b if sa == sb: return 1.0 - ratio = difflib.SequenceMatcher(None, sa, sb).ratio() - ta = set(re.findall(r"[a-z0-9]+", sa)) - tb = set(re.findall(r"[a-z0-9]+", sb)) + da = _distinctive_tokens(sa) # order-preserving for the sequence ratio + db = _distinctive_tokens(sb) + ratio = difflib.SequenceMatcher(None, " ".join(da), " ".join(db)).ratio() + ta, tb = set(da), set(db) jaccard = len(ta & tb) / len(ta | tb) if (ta or tb) else 0.0 return max(ratio, jaccard) @@ -1881,6 +1908,45 @@ def existing_investor_by_source_row(conn): return out +def fundraising_contact_emails_by_row(conn): + """{ source_row_id: {'order': {sort_order: email}, 'name': {normalized_name: email}} } of the + authoritative email per grid contact, for HEALING blank pill emails on read. + + The grid blob is canonical for the edit sheet, but an email can reach the linked classic + contact (via email capture / a contact edit) without ever being written back into the blob + pill — so the mobile "Edit investor" sheet shows an empty email for a contact the directory + clearly has (Grant, 2026-06-20). We recover it from the relational mirror: prefer the synced + fundraising_contacts.email, else the linked classic contacts.email (the source that actually + holds the captured address). Keyed by sort_order (pills and fundraising_contacts share the + blob order — the robust key) with a normalized-name fallback. Only non-blank emails are + returned; filling is fill-only-when-blank in the handler, so it heals and converges (the next + one-row save persists the recovered email into the blob).""" + out = {} + rows = conn.execute( + """ + SELECT fi.source_row_id AS srid, fc.sort_order AS so, fc.full_name AS name, + COALESCE(NULLIF(TRIM(fc.email), ''), c.email) AS email + FROM fundraising_investors fi + JOIN fundraising_contacts fc ON fc.investor_id = fi.id + LEFT JOIN contacts c ON c.id = fc.contact_id AND c.deleted_at IS NULL + """ + ).fetchall() + for r in rows: + email = str(r['email'] or '').strip() + if not email: + continue + srid = str(r['srid'] or '') + if not srid: + continue + bucket = out.setdefault(srid, {'order': {}, 'name': {}}) + if r['so'] is not None: + bucket['order'][int(r['so'])] = email + nm = _normalize_text(r['name']) + if nm: + bucket['name'][nm] = email + return out + + def contact_grid_signals(conn, contact_id=None): """Return {contacts.id: {'committed': float, 'pipeline_stage': str|None, 'priority': bool}} for every classic contact linked to a fundraising-grid investor (via fundraising_contacts.contact_id, @@ -5830,6 +5896,7 @@ class CRMHandler(BaseHTTPRequestHandler): reminder_by_row = reminder_status_by_source_row(conn) existing_by_row = existing_investor_by_source_row(conn) recency_by_row = staleness_by_source_row(conn) + emails_by_row = fundraising_contact_emails_by_row(conn) conn.close() try: @@ -5873,6 +5940,19 @@ class CRMHandler(BaseHTTPRequestHandler): last_activity, staleness = recency_by_row.get(srid, (None, '')) r['last_activity_at'] = last_activity r['staleness'] = staleness + # Heal blank pill emails from the relational mirror (fill-only — never overwrite a value + # already in the blob). Unlike the read-only columns above, email is a REAL blob field, + # so this is a backfill, not a derived signal: it needs NO strip point, and the next + # one-row save legitimately persists it. Match by pill order, then by name. + heal = emails_by_row.get(srid) + pills = r.get('contacts') + if heal and isinstance(pills, list): + for i, c in enumerate(pills): + if not isinstance(c, dict) or str(c.get('email') or '').strip(): + continue + found = heal['order'].get(i) or heal['name'].get(_normalize_text(c.get('name'))) + if found: + c['email'] = found return self.send_json({ "data": { diff --git a/backend/test_grid_email_heal.py b/backend/test_grid_email_heal.py new file mode 100644 index 0000000..a59923a --- /dev/null +++ b/backend/test_grid_email_heal.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 +"""Regression: GET /api/fundraising/state heals blank grid-pill emails from the relational mirror. + +The grid blob is canonical for the mobile "Edit investor" sheet, but an email can reach a linked +classic contact (email capture / a contact edit) without ever being written back into the blob pill +— so the edit form showed an empty email for a contact the directory clearly had (Grant, 2026-06-20). +The state handler now fills a blank pill email from fundraising_contacts.email, else the linked +contacts.email, matched by pill order then name. This asserts: + - a blank pill whose linked contact has an email is HEALED on read; + - a blank pill whose linked contact is also blank stays blank; + - a pill that already carries an email in the blob is NEVER overwritten (fill-only). +Synthetic data only. + +Run: cd backend && python3 test_grid_email_heal.py +""" +import http.client +import json +import os +import sqlite3 +import sys +import tempfile +import threading +from http.server import ThreadingHTTPServer + +_DATA = tempfile.mkdtemp() +os.environ["CRM_DATA_DIR"] = _DATA +os.environ["CRM_DB_PATH"] = os.path.join(_DATA, "crm.db") + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import server # noqa: E402 + +FAILS = [] + + +def check(cond, msg): + print((" PASS " if cond else " FAIL ") + msg) + if not cond: + FAILS.append(msg) + + +class _Quiet(server.CRMHandler): + def log_message(self, *a): + pass + + +def _get_state(port, token): + conn = http.client.HTTPConnection("127.0.0.1", port, timeout=10) + conn.request("GET", "/api/fundraising/state", headers={"Authorization": "Bearer " + token}) + resp = conn.getresponse() + raw = resp.read().decode("utf-8", "replace") + conn.close() + return resp.status, (json.loads(raw) if raw else None) + + +GRID = { + "columns": [{"id": "investor_name", "label": "Investor", "type": "text"}, + {"id": "contacts", "label": "Contacts", "type": "contacts"}], + "rows": [ + {"id": "rowW", "investor_name": "Wyoming", "notes": "", + "contacts": [{"name": "Philip Treick", "email": "", "title": ""}, + {"name": "Jose Briones", "email": "", "title": ""}]}, + {"id": "rowA", "investor_name": "Acme Capital", "notes": "", + "contacts": [{"name": "Jane Doe", "email": "keep@acme.com", "title": ""}]}, + ], +} + + +def seed(): + c = sqlite3.connect(os.environ["CRM_DB_PATH"]) + c.execute("INSERT INTO users (id,username,email,password_hash,full_name,role,is_active) " + "VALUES ('u1','grant','grant@ten31.example','x','Grant','admin',1)") + c.execute("INSERT INTO fundraising_state (id, grid_json, views_json, version) " + "VALUES ('main', ?, '[]', 1) " + "ON CONFLICT(id) DO UPDATE SET grid_json = excluded.grid_json", (json.dumps(GRID),)) + # Classic contacts directory: Jose has the captured email the blob never got; Philip is blank. + c.execute("INSERT INTO contacts (id,first_name,last_name,email) VALUES " + "('c-phil','Philip','Treick','')," + "('c-jose','Jose','Briones','jbriones@uwyo.edu')," + "('c-jane','Jane','Doe','other@acme.com')") # differs from the blob's keep@acme.com + # Relational mirror (what sync_fundraising_relational would build): blank fc.email, linked contact_id. + c.execute("INSERT INTO fundraising_investors (id,investor_name,source_row_id,total_invested) VALUES " + "('inv-w','Wyoming','rowW',0),('inv-a','Acme Capital','rowA',0)") + c.execute("INSERT INTO fundraising_contacts (id,investor_id,full_name,email,sort_order,contact_id) VALUES " + "('fc-phil','inv-w','Philip Treick','',0,'c-phil')," + "('fc-jose','inv-w','Jose Briones','',1,'c-jose')," + "('fc-jane','inv-a','Jane Doe','',0,'c-jane')") + c.commit() + c.close() + + +def main(): + server.init_db() + seed() + token = server.create_token("u1", "grant", "admin") + + httpd = ThreadingHTTPServer(("127.0.0.1", 0), _Quiet) + port = httpd.server_address[1] + threading.Thread(target=httpd.serve_forever, daemon=True).start() + try: + st, d = _get_state(port, token) + rows = ((d or {}).get("data", {}).get("grid", {}) or {}).get("rows", []) + by_id = {r.get("id"): r for r in rows} + w = by_id.get("rowW", {}) + a = by_id.get("rowA", {}) + wc = w.get("contacts", []) + ac = a.get("contacts", []) + + print("\n[heal: blank pill email filled from the linked contact (Jose)]") + jose = next((c for c in wc if c.get("name") == "Jose Briones"), {}) + check(st == 200 and jose.get("email") == "jbriones@uwyo.edu", + f"Jose pill healed to jbriones@uwyo.edu (got {jose.get('email')!r})") + + print("\n[heal: blank pill whose contact is also blank stays blank (Philip)]") + phil = next((c for c in wc if c.get("name") == "Philip Treick"), {}) + check(phil.get("email", "") == "", + f"Philip pill stays blank (got {phil.get('email')!r})") + + print("\n[heal: a pill that already has an email is never overwritten (Jane)]") + jane = next((c for c in ac if c.get("name") == "Jane Doe"), {}) + check(jane.get("email") == "keep@acme.com", + f"Jane pill keeps its blob email, not the contact's (got {jane.get('email')!r})") + finally: + httpd.shutdown() + + print() + if FAILS: + print(f"FAILED ({len(FAILS)}):") + for f in FAILS: + print(f" - {f}") + sys.exit(1) + print("ALL PASS (grid email heal)") + + +if __name__ == "__main__": + main() diff --git a/backend/test_intake_endpoints.py b/backend/test_intake_endpoints.py index 407402c..8d0bcf6 100644 --- a/backend/test_intake_endpoints.py +++ b/backend/test_intake_endpoints.py @@ -75,6 +75,12 @@ GRID = { "contacts": [{"name": "Charlie Brown", "email": "cb@brown.fund", "title": ""}]}, {"id": "rowBeta", "investor_name": "Beta Capital LLC", "notes": "", "contacts": [{"name": "Pat Roe", "email": "pat@beta.com", "title": ""}]}, + # Generic-descriptor decoys: share only "investment group" / "investments" with the + # Fortitude card below — must NOT surface as look-alikes (the 2026-06-20 false-positive fix). + {"id": "rowAether", "investor_name": "Aether Investment Group", "notes": "", + "contacts": [{"name": "Ada Ng", "email": "ada@aether.com", "title": ""}]}, + {"id": "rowRussell", "investor_name": "Russell Investments", "notes": "", + "contacts": [{"name": "Russ Lee", "email": "russ@russell.com", "title": ""}]}, ], } @@ -178,6 +184,20 @@ def main(): check(st == 200 and data.get("match") is None and data.get("candidates") == [], f"unrelated query -> no match, no candidates (got {data})") + print("\n[fuzzy: shared generic words alone do NOT surface look-alikes (Fortitude vs Aether/Russell)]") + st, d = _req(port, "GET", "/api/intake/match?q=Fortitude%20Investment%20Group", token) + data = (d or {}).get("data", {}) + cids = [c["id"] for c in data.get("candidates", [])] + check(data.get("match") is None and "rowAether" not in cids and "rowRussell" not in cids, + f"generic-only overlap -> no decoy candidates (got {data})") + + print("\n[fuzzy: a shared DISTINCTIVE word still surfaces (Aether Capital ~ Aether Investment Group)]") + st, d = _req(port, "GET", "/api/intake/match?q=Aether%20Capital", token) + data = (d or {}).get("data", {}) + cids = [c["id"] for c in data.get("candidates", [])] + check(data.get("match") is None and "rowAether" in cids, + f"distinctive overlap -> rowAether candidate (got {data})") + print("\n[match: missing q and email -> 400]") st, _ = _req(port, "GET", "/api/intake/match", token) check(st == 400, f"no params -> 400 (got {st})") diff --git a/docs/guides/matrix-intake.md b/docs/guides/matrix-intake.md index 5d01408..895f273 100644 --- a/docs/guides/matrix-intake.md +++ b/docs/guides/matrix-intake.md @@ -69,6 +69,26 @@ Spark). See *Fuzzy matching* below. Tests green (27/27 backend + the offline bot A bare `yes`/`no` typed **top-level** (not in the thread) while a proposal is pending gets a "reply in the thread" redirect (`store.any_pending()` guard in `handle_intake`), not a misparsed new intake. +5. **On a conclusive decision (approve or reject) the whole thread is redacted** (Grant, 2026-06-20) + — exactly like the email-review room. `handle_reply`/`handle_disambiguation` call + `redact_thread(intake_room, root)` instead of posting a `✅`/`🗑️` confirmation: it clears the + bot's card + `📇 Reading…` ack + the main-timeline **nudge** + **the user's own note/photo** (the + thread root). The thread vanishing is the acknowledgment; a confirmation reply would just keep it + alive. Only conclusive decisions clear — an `edit`/NL-revise keeps the thread (still pending), and + a commit **failure** posts a `⚠️` and restores the proposal (no redact, so the user can retry). + `redact_thread` now takes the **room** as its first arg and matches replies by `m.thread` + event_id **or** `m.in_reply_to` event_id (so the un-threaded nudge clears too). **Prerequisite: + the bot needs a `redact`/moderator power level in the intake room** to clear the *humans'* + messages (its own need none) — without it the user's note/photo lingers (best-effort, no error). + Same Element caveat as email: turn OFF "show removed messages" so the placeholders disappear. + **Trade-off:** there's no longer an in-Matrix confirmation of *what* was logged after a successful + commit (the record is in the CRM) — by design, matching the email room; revisit if wanted. + **Retroactive backfill:** `backend/matrix_intake/redact_intake.py` (dry-run default; `--apply`) + clears the room's pre-existing backlog. The intake room keeps **no durable pending state** (the + proposal store is in-memory, lost on restart), so every message in it is stale after a restart and + safe to redact — it clears **every** `m.room.message` (text + card images), bot's and humans' + alike. Run on the Spark: `docker compose run --rm matrix-intake python -u + backend/matrix_intake/redact_intake.py [--apply]`. ## Business-card capture (M3 — image intake) diff --git a/docs/handoffs/in-app-card-intake-plan.md b/docs/handoffs/in-app-card-intake-plan.md new file mode 100644 index 0000000..6395f01 --- /dev/null +++ b/docs/handoffs/in-app-card-intake-plan.md @@ -0,0 +1,128 @@ +# Plan: in-app business-card intake (mobile camera → transcribe → approve) + +**Status:** PLAN — awaiting Grant's go-ahead before building (requested 2026-06-20). +**Goal:** a camera button in the mobile top bar (left of the quick-log pencil) → take a photo or +pick one from the library → the **same** vision-transcribe → parse → fuzzy-match → edit/approve/reject +flow the Matrix card intake runs (M3), surfaced as an inline mobile sheet instead of a Matrix thread. + +## Why this is cheaper than it looks + +The reusable core is already nio-free and already reachable from the CRM monolith: + +- `server.py` **already** imports the Spark client: `sys.path.insert(.../ingest); import llm` + (`server.py:6485`). `llm.chat_vision` (multimodal → Spark Control passthrough) is the exact call + the bot's card OCR uses. +- `backend/matrix_intake/parse.py` imports only `json`, `re`, `spark`; `spark.py` imports only + `os`, `sys`, `llm`. **Neither imports `matrix-nio`.** So `server.py` can import `parse` + + `spark` the same way it imports `llm`, with **no bot refactor and no matrix-nio in the CRM**. +- Match (`find_intake_match` / `find_intake_candidates`) and the write + (`/api/fundraising/log-communication`, create-if-missing + contact upsert + audit) are **already** + in `server.py`. The fuzzy matcher just got the generic-word fix (v-next). +- The whole `backend/` tree (incl. `ingest/` + `matrix_intake/`) is `COPY`'d into the s9pk image + (`start9/0.4/Dockerfile:62`), so the imports resolve on the box at runtime. + +So this is **one new endpoint + one new mobile component**, reusing everything downstream. No +migration, no new dependency, no change to the live Matrix bot. + +## Architecture decision + +**Recommended (Option A): `server.py` imports `matrix_intake.parse` + `matrix_intake.spark` +directly.** Add `backend/matrix_intake` to `sys.path` (guarded like the existing `ingest` insert) +and `import parse, spark`. Pros: zero bot churn, reuses the tested transcribe + email-integrity +parse verbatim, no divergence. Con: the CRM now imports two modules out of the bot package — a mild +coupling, but they're pure (no nio, no CRM HTTP; we do NOT import the bot's `crm_client`). + +**Alternative (Option B): extract a shared `backend/intake_core/`** (`vision.py` = transcribe, +`extract.py` = parse/normalize/revise) that both the bot and `server.py` import. Cleaner ownership, +but it refactors the live, working bot for no functional gain right now. **Defer** unless the +coupling in A bites later (e.g. if the bot's `parse` grows nio-coupled helpers). + +> Module-name caution (from the matrix-intake guide): the bot imports by **bare name** (`import +> spark`, `import llm`, `import config`). `server.py` already has `ingest/` on its path (so `llm`, +> `config`, `http_util` resolve). Adding `matrix_intake/` lets `parse`/`spark` resolve with **no +> name overlap** against `ingest`'s modules — verified. Import only `parse` + `spark` (not +> `crm_client`/`settings`/`bot`). + +## Server: one new endpoint + +`POST /api/intake/card` — authenticated **member+** (it's a UI feature for the team; NOT bot-gated, +NOT admin-only). Body is JSON, image as base64 (no multipart parsing in the stdlib handler): + +``` +{ "image_b64": "", "mime": "image/jpeg" } +→ 200 { "transcription": "...", + "proposal": { investor_name, contact_name, contact_email, contact_title, + city, linkedin_url, phone, mobile, note }, + "match": { id, name } | null, + "candidates": [ { id, name, score, matched_on }, ... ] } +``` + +Handler flow (mirrors `bot.handle_card` → `handle_intake`, minus Matrix): +1. Auth (reuse the standard authenticated gate). +2. Decode + size-guard the base64; default mime `image/jpeg`. +3. `transcription = spark.transcribe_card(image_b64, mime)` (Spark Control vision). On error → 502 + `{ ok:false, reason:"vision_unavailable" }`; on `<5` chars → 200 `{ ok:false, reason:"unreadable" }`. +4. `proposal = parse.parse_message("New investor — from a business card:\n"+transcription, roster=None)` + — the **email-integrity rule rides along** (an address/phone/LinkedIn is kept only if it literally + appears in the transcription, never minted), exactly as in Matrix. `roster=None` for v1 (the + team-roster framing is a Spark-side env; the box can pass None — prior parse behavior). +5. `match = find_intake_match(...)`, `candidates = find_intake_candidates(...)` from the proposal's + `investor_name`/`contact_email`. +6. Return `{ transcription, proposal, match, candidates }`. + +**Approve** reuses the existing write — the mobile sheet POSTs the (possibly-edited) proposal to +`POST /api/fundraising/log-communication` with `create_investor_if_missing` (new) or the picked +`_match_id`/candidate row (existing), tagged **`source="app_card"`** (a new provenance value +distinct from `matrix_card`/`matrix_intake`). **Reject** is client-only (discard the sheet) — no +server call, nothing written. + +**No NL-edit for v1.** In a form UI, editing the fields directly is easier than chatting "change the +email to…". The fields are inline-editable in the sheet. (A `/api/intake/card/revise` wrapping +`parse.revise` is a possible later enhancement if Grant wants conversational corrections.) + +## Mobile UI: one new component + +`MobileCardCapture` in the shell top bar, **left of `MobileQuickLog`** (the cluster at +`index.html:14609`). Reuses ``, `.sheet-input`, `StageChip`, and the New-investor +sheet patterns (8g). + +- **Trigger:** a camera-icon button + a hidden `` — + **omit `capture`** so iOS shows the native action sheet (Take Photo / Photo Library / Browse), + satisfying "take a photo OR use an existing photo." (`capture="environment"` would force the camera + and skip the library — not what we want.) +- **On select:** read the file as a data URL; **downscale via `` to ~2000px max dimension** + before base64 (native, no library) — keeps the payload under the StartOS reverse-proxy body cap + (the matrix-intake guide's known `413` risk) and the model downscales to ~2 MP anyway. Don't + over-shrink (hurts OCR). Show a "📇 Reading the card…" loading state. +- **POST** to `/api/intake/card`; on `unreadable`/`vision_unavailable` show a "try a clearer, + well-lit, fill-the-frame photo" message with a Retake button (same UX as the bot's `📇` replies). +- **Approval sheet** (the New-investor sheet, pre-filled from `proposal`): editable investor name + + contact name/email/title/city/phone/mobile/LinkedIn; a small "Existing investor?" block when + `match`/`candidates` came back (pick a candidate row, or "Add as new"); **Approve** → the + log-communication write; **Reject/Retake** → discard. +- **Icon:** reuse the proven `.quicklog-btn svg` sizing fix (explicit CSS `width/height` + `flex:none`) + so the new camera SVG doesn't hit the same iOS sole-flex-child collapse. + +## Scope / effort / risk + +- **Server:** ~1 endpoint + the `parse`/`spark` import guard + an `app_card` source tag. Small. +- **Frontend:** ~1 component (camera button + capture input + canvas downscale + approval sheet) + + CSS for the button. Medium (the approval sheet is the New-investor sheet pre-filled). +- **No** migration, **no** new dependency, **no** Matrix-bot change. +- **Tests:** an endpoint test that stubs `spark.transcribe_card` (like `test_spark.py` does) and + asserts the transcribe→parse→match shape + the email-integrity passthrough; the real vision/OCR + path stays **live-smoke only** (same as Matrix M3). Frontend is inspection + on-device. +- **Risks:** (1) large-image upload vs the reverse-proxy cap — mitigated by client-side canvas + downscale; (2) iOS file-input behavior across Safari/standalone-PWA — verify on-device; (3) OCR + accuracy is the **shared** Matrix limitation (resolution-bound; "fill the frame"), not new here; + (4) the `parse`/`spark` import must be lazy/guarded so a dev `./start.sh` without Spark reachable + still boots (the endpoint just 502s) — mirror the existing lazy `import llm` at `server.py:6485`. + +## Open questions for Grant (resolve before building) + +1. **Provenance tag** — `source="app_card"` OK (distinct from `matrix_card`)? +2. **v1 = form-field edits only** (no conversational NL-edit on the card) — agreed? NL-edit can + follow if wanted. +3. **Who can use it** — any authenticated member (recommended), or restrict? +4. **Ships in the s9pk** (server endpoint + frontend) — so it needs a version bump + build + install, + unlike the bot-only M3. Confirm that's the intended delivery path. diff --git a/frontend/index.html b/frontend/index.html index fc0387f..244e8be 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -2281,6 +2281,14 @@ .sort-row-label { font-size: 15px; font-weight: 500; color: var(--text-primary); } .sort-row-hint { font-family: 'IBM Plex Mono', monospace; font-size: 11px; color: var(--text-subtle); } .sort-row-check { flex: none; color: var(--accent); font-size: 15px; } + /* Reversible-direction control under a selected sort row (e.g. Staleness oldest/newest). */ + .sort-dir { display: flex; gap: 8px; padding: 0 4px 2px; } + .sort-dir-opt { + flex: 1; cursor: pointer; font-family: inherit; font-size: 13px; font-weight: 500; + min-height: 40px; padding: 0 12px; border-radius: 9px; + border: 1px solid var(--border); background: var(--bg-input); color: var(--text-secondary); + } + .sort-dir-opt.active { border-color: var(--accent); color: var(--accent); background: var(--bg-panel-elevated); } /* A–Z directory: sticky letter headers over a card list. */ .az-header { @@ -2560,6 +2568,11 @@ display: inline-flex; align-items: center; justify-content: center; } .quicklog-btn:active { background: var(--bg-hover); } + /* iOS Safari collapses an inline that is the sole, centered flex child of a + fixed-size flex button when the svg carries only width/height *attributes* (it renders + as a ~1px dot). Explicit CSS dimensions + flex:none fix it — the same reason the working + .bottom-tab-icon (sized box) and .sort-pill (flex:none + text) icons render. */ + .quicklog-btn svg { width: 18px; height: 18px; flex: none; } .quicklog-hint { font-size: 13px; color: var(--text-subtle); line-height: 1.5; margin: 0 0 12px; } .quicklog-pool { display: flex; flex-direction: column; gap: 8px; margin-top: 12px; } .quicklog-empty { font-size: 13px; color: var(--text-subtle); padding: 16px 4px; } @@ -4445,19 +4458,39 @@ {label} ); - const SortSheet = ({ open, onClose, title, options, value, onPick }) => ( + // Optional `dir`/`onToggleDir` add a reversible-direction control under the selected option + // when that option is `reversible` (e.g. the grid's Staleness — oldest vs most-recent first). + // Callers that omit onToggleDir behave exactly as before (every row taps-and-closes). + const SortSheet = ({ open, onClose, title, options, value, onPick, dir, onToggleDir }) => (
- {options.map((o) => ( - - ))} + {options.map((o) => { + const reversible = o.reversible && !!onToggleDir; + return ( + + + {value === o.id && reversible && ( +
+ + +
+ )} +
+ ); + })}
); @@ -4468,7 +4501,8 @@ { id: 'name', pill: 'Name', label: 'Name', hint: 'A → Z' }, { id: 'stage', pill: 'Stage', label: 'Pipeline stage', hint: 'Lead → Commitment' }, { id: 'committed', pill: 'Committed', label: 'Committed', hint: 'Most first' }, - { id: 'staleness', pill: 'Staleness', label: 'Last contact', hint: 'Most stale first' }, + { id: 'staleness', pill: 'Staleness', label: 'Last contact', hint: 'Most stale first', reversible: true, + dirLabels: { desc: 'Most stale first', asc: 'Most recent first' } }, { id: 'priority', pill: 'Priority', label: 'Priority', hint: 'Flagged first' }, ]; // Pipeline sorts within a stage (dc PipelineApp:580). "Staleness" uses the opp's updated_at as @@ -10166,6 +10200,7 @@ const [selectedId, setSelectedId] = useState(null); const [sheet, setSheet] = useState(null); // 'view' | 'create' | 'note' | 'stage' | 'reminder' | 'sort' const [sortKey, setSortKey] = useState('name'); // GRID_SORTS + const [sortDir, setSortDir] = useState('desc'); // only the reversible sorts (Staleness) read this const [busy, setBusy] = useState(false); const [createForm, setCreateForm] = useState({ name: '', contactName: '', contactEmail: '', note: '', priority: false, stage: '', reminderTitle: '', reminderDue: '' }); const [reminderForm, setReminderForm] = useState({ title: '', due_date: '', details: '' }); @@ -10263,6 +10298,8 @@ const byName = (a, b) => String(a.investor_name || '') .localeCompare(String(b.investor_name || ''), undefined, { sensitivity: 'base' }); const staleDays = (r) => { const d = daysSince(r.last_activity_at); return d == null ? Number.MAX_SAFE_INTEGER : d; }; + // Direction multiplier for the reversible sorts; 'desc' keeps each key's natural default. + const dirMul = sortDir === 'asc' ? -1 : 1; const cmp = { name: byName, stage: (a, b) => { @@ -10270,11 +10307,13 @@ return (oi(a) - oi(b)) || byName(a, b); }, committed: (a, b) => (gridRollup(b, fundColumnIds) - gridRollup(a, fundColumnIds)) || byName(a, b), - staleness: (a, b) => (staleDays(b) - staleDays(a)) || byName(a, b), // larger days first = most stale first + // default (desc): larger days first = most stale first; asc flips to most-recent first. + // Name stays the ascending tiebreak regardless of direction. + staleness: (a, b) => (dirMul * (staleDays(b) - staleDays(a))) || byName(a, b), priority: (a, b) => ((b.priority ? 1 : 0) - (a.priority ? 1 : 0)) || byName(a, b), // row.priority: boolean }; return [...searched].sort(cmp[sortKey] || byName); - }, [rows, activeViewObj, columns, fundColumnIds, search, sortKey]); + }, [rows, activeViewObj, columns, fundColumnIds, search, sortKey, sortDir]); const selectedRow = useMemo(() => rows.find((r) => r.id === selectedId) || null, [rows, selectedId]); const closeSheet = () => setSheet(null); @@ -10533,7 +10572,9 @@ )} + options={GRID_SORTS} value={sortKey} + onPick={(id) => { setSortKey(id); setSortDir('desc'); }} + dir={sortDir} onToggleDir={setSortDir} /> {views.map((v) => ( diff --git a/start9/0.4/startos/utils.ts b/start9/0.4/startos/utils.ts index 6c90d45..f3e763e 100644 --- a/start9/0.4/startos/utils.ts +++ b/start9/0.4/startos/utils.ts @@ -63,8 +63,9 @@ export const PACKAGE_TITLE = 'Ten31 Database' // * 0.1.0:95 (mobile-first redesign goes live + installable PWA. The Grid, Pipeline, Reminders & Contacts screens are touch-native on phones below 768px [safe-area bottom-tab nav, card lists, drag-dismiss bottom sheets, swipe actions, full-screen Grid detail, SVG tab icons + ·Ten31· wordmark], with an app-wide light theme + toggle. Installable home-screen PWA: manifest.webmanifest [standalone display, #0b1118 theme] + square/apple-touch icons + a pre-auth /manifest.webmanifest route; iOS-first, no service worker. Pipeline funnel v2: 4-stage lead→engaged→diligence→commitment [in-app migration 0007] with derived grid signals [pipeline_stage/existing_investor/recency] injected-on-GET, stripped-on-write. Desktop UI unchanged; no LLM path. Bundles the previously deploy-pending mobile Phases 0–8 + drag-reorder views + the PWA) // * 0.1.0:96 (login page mobile/PWA conformance — the one surface the v95 mobile redesign skipped. CSS-only: 100vh→100dvh [dynamic viewport, fixes the centered card tucking under the iOS standalone status bar], a <768px media query adding 16px screen gutters + env[safe-area-inset] top/bottom clearance + touch-sized fields [inputs 46px/15px, button 46px], full-bleed card on small phones, and the §4 card depth shadow on the login card to match .section. No markup/JS/schema change; desktop login unchanged) // * 0.1.0:97 (mobile top-bar polish + native zoom behaviour. Viewport meta gains maximum-scale=1 + user-scalable=no: kills pinch-zoom AND the iOS auto-zoom-on-focus that jerked the page in on every <16px input tap [app-wide, not just login]; OS accessibility zoom still works. Top-bar account initial now flex-centered + dc-aligned [IBM Plex Mono, accent-light, 13px — was defaulting to inline/baseline, off-center]. Quick-log pencil bumped --text-muted→--text-secondary for real affordance [the dc t3 grey thin-outline read as empty next to the color sun emoji on-device]. CSS-only; no JS/schema change) -// * Current: 0.1.0:98 (business-card intake [Matrix bot] captures a contact's phone, mobile/cell, city + LinkedIn from a scanned card onto the contact record — cell to Mobile, office to Phone, fax skipped. Server half: _upsert_contact_from_fundraising now accepts phone+mobile on the contact dict [city+linkedin already worked]. The bot's transcription/extraction/card changes ship on the Spark [git pull + rebuild]. No schema change [contacts columns already exist]; no user-facing CRM change) -export const PACKAGE_VERSION = '0.1.0:98' +// * 0.1.0:98 (business-card intake [Matrix bot] captures a contact's phone, mobile/cell, city + LinkedIn from a scanned card onto the contact record — cell to Mobile, office to Phone, fax skipped. Server half: _upsert_contact_from_fundraising now accepts phone+mobile on the contact dict [city+linkedin already worked]. The bot's transcription/extraction/card changes ship on the Spark [git pull + rebuild]. No schema change [contacts columns already exist]; no user-facing CRM change) +// * Current: 0.1.0:99 (Grant device-test round 2, CRM half: intake fuzzy match scores DISTINCTIVE tokens only [no more "Investment Group"/"Capital"/"Family Office" false look-alikes]; mobile grid "Last contact"/staleness sort is reversible; mobile Edit-investor prefills a contact's email [GET /api/fundraising/state heals a blank grid pill from the linked classic contact, fill-only]; mobile quick-log pencil icon renders [CSS sizing on the sole flex-child svg]. The Matrix intake thread-redaction change ships on the Spark, not here. No schema change; no migration) +export const PACKAGE_VERSION = '0.1.0:99' export const DATA_MOUNT_PATH = '/data' export const WEB_PORT = 8080 diff --git a/start9/0.4/startos/versions/index.ts b/start9/0.4/startos/versions/index.ts index c9a7a49..40246ca 100644 --- a/start9/0.4/startos/versions/index.ts +++ b/start9/0.4/startos/versions/index.ts @@ -61,6 +61,6 @@ import { v_0_1_0_97 } from './v0.1.0.97' import { v_0_1_0_98 } from './v0.1.0.98' export const versionGraph = VersionGraph.of({ - current: v_0_1_0_98, - other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73, v_0_1_0_74, v_0_1_0_75, v_0_1_0_76, v_0_1_0_77, v_0_1_0_78, v_0_1_0_79, v_0_1_0_80, v_0_1_0_81, v_0_1_0_82, v_0_1_0_83, v_0_1_0_84, v_0_1_0_85, v_0_1_0_86, v_0_1_0_87, v_0_1_0_88, v_0_1_0_89, v_0_1_0_90, v_0_1_0_91, v_0_1_0_92, v_0_1_0_93, v_0_1_0_94, v_0_1_0_95, v_0_1_0_96, v_0_1_0_97], + current: v_0_1_0_99, + other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73, v_0_1_0_74, v_0_1_0_75, v_0_1_0_76, v_0_1_0_77, v_0_1_0_78, v_0_1_0_79, v_0_1_0_80, v_0_1_0_81, v_0_1_0_82, v_0_1_0_83, v_0_1_0_84, v_0_1_0_85, v_0_1_0_86, v_0_1_0_87, v_0_1_0_88, v_0_1_0_89, v_0_1_0_90, v_0_1_0_91, v_0_1_0_92, v_0_1_0_93, v_0_1_0_94, v_0_1_0_95, v_0_1_0_96, v_0_1_0_97, v_0_1_0_98], }) diff --git a/start9/0.4/startos/versions/v0.1.0.99.ts b/start9/0.4/startos/versions/v0.1.0.99.ts new file mode 100644 index 0000000..18936ef --- /dev/null +++ b/start9/0.4/startos/versions/v0.1.0.99.ts @@ -0,0 +1,22 @@ +import { VersionInfo } from '@start9labs/start-sdk' + +// Grant's real-phone device-test round 2 — four in-app fixes (CRM half; the Matrix intake +// thread-redaction change ships on the Spark, not here): +// - Intake fuzzy match no longer over-indexes on generic firm words ("Investment Group", +// "Capital", "Family Office"): _name_similarity scores DISTINCTIVE tokens only. +// - Mobile grid "Last contact" (staleness) sort is reversible (most-stale / most-recent first). +// - Mobile "Edit investor" prefills a contact's email: GET /api/fundraising/state heals a blank +// grid pill email from the linked classic contact (fill-only; next save persists it). +// - Mobile quick-log pencil icon renders (CSS sizing on the sole flex-child svg — iOS fix). +// No schema change — no migration. +export const v_0_1_0_99 = VersionInfo.of({ + version: '0.1.0:99', + releaseNotes: { + en_US: [ + 'Mobile polish + intake fixes: reversible "Last contact" sort, Edit-investor now shows a', + "contact's saved email, the quick-log pencil icon renders, and business-card/intake", + 'duplicate-matching ignores generic firm words like "Investment Group" / "Capital".', + ].join(' '), + }, + migrations: { up: async () => {}, down: async () => {} }, +})