docs: triage Matrix inbox; lead Next with contact_type retirement

- ROADMAP: backlog backup-history collapse, tab-on-refresh, email cc-dedup (P2)
- ROADMAP: mark in-app card intake done & live (v0.1.0:100); remove stale plan doc
- AGENTS.md: reorder Current-state Next to lead with contact_type retirement
This commit is contained in:
Keysat
2026-06-20 21:44:16 -05:00
parent 8f82721a54
commit b23c48bf7a
3 changed files with 10 additions and 136 deletions
+1 -1
View File
@@ -115,5 +115,5 @@ _**Box live at v0.1.0:105 (deployed + verified 2026-06-20)** — clean StartOS m
- **New (v0.1.0:104):** admin-only **Purge Deleted Data** (Settings → Admin) — guarded, type-to-confirm hard-delete of soft-deleted rows; see the soft-delete convention + `test_purge_soft_deleted.py`. - **New (v0.1.0:104):** admin-only **Purge Deleted Data** (Settings → Admin) — guarded, type-to-confirm hard-delete of soft-deleted rows; see the soft-delete convention + `test_purge_soft_deleted.py`.
- **Verification:** **45/45** backend, render-smoke green, reviewer-agent APPROVE after fixing **1 blocker** (contact purge left a dangling `reminders.contact_id` — now NULLed + test-guarded). New UI behavior is **live-smoke / on-device only** (jsdom can't drive touch). - **Verification:** **45/45** backend, render-smoke green, reviewer-agent APPROVE after fixing **1 blocker** (contact purge left a dangling `reminders.contact_id` — now NULLed + test-guarded). New UI behavior is **live-smoke / on-device only** (jsdom can't drive touch).
- **Bug A — Grant is handling:** `odell/marty/finance/ten31@` can't enroll for email capture ("could not resolve user_id") because the enroll flow requires a CRM `users` row; Grant is creating user accounts for those mailboxes. - **Bug A — Grant is handling:** `odell/marty/finance/ten31@` can't enroll for email capture ("could not resolve user_id") because the enroll flow requires a CRM `users` row; Grant is creating user accounts for those mailboxes.
- **Next:** (A) confirm the two stuck mailboxes pulled current + Grant's 4 new mailbox users enroll; (B) **retire `contact_type`** — replace the Contacts Investors/Prospects tabs + TYPE badge with grid-derived `existing_investor`/`pipeline_stage`, then drop the column (see ROADMAP); (C) **contacts ↔ `fundraising_contacts` consolidation** — census-first (count A/linked, B/contacts-only, C/pill-only on the box; see ROADMAP). **A TEMPORARY admin census is LIVE on the box (v0.1.0:105): Settings → Admin → "Run census" (or `GET /api/admin/contacts-census`) reports A/B/C — run it, capture the numbers, then DELETE the endpoint + handler + route + button** (all tagged `TEMPORARY` in code; mirrors `backend/scripts/contacts_census.sql`). (D) carried: bell approve-on-phone → Matrix-thread-clears round-trip spot-check. - **Next:** (A) **retire `contact_type`** (the next build) — replace the Contacts Investors/Prospects tabs + TYPE badge with grid-derived `existing_investor`/`pipeline_stage`, repoint the dashboard `total_lps`/`total_prospects` counts, then drop the column (live UI change → its own small design pass; see ROADMAP); (B) **contacts ↔ `fundraising_contacts` consolidation** — capture A/B/C from the live census (Settings → Admin → "Run census", or `GET /api/admin/contacts-census`), then **DELETE the TEMPORARY census endpoint + handler + route + button** (all tagged `TEMPORARY`; mirrors `backend/scripts/contacts_census.sql`); (C) confirm the two stuck mailboxes pulled current + Grant's 4 new mailbox users enroll; (D) carried: bell approve-on-phone → Matrix-thread-clears round-trip spot-check.
- **Open / risks:** the Contacts pagination, the purge, and the email-sync auto-recovery are **live-smoke / not yet device-confirmed**. Carried: **Claude/Architect path unverified live on the box**; vision OCR small-in-frame misread (`mara.com→marac.com`); doc drift — `crm-overview.md` narrative + `EVALUATION.md` still describe `lp_profiles` (the active API/schema claims were fixed; the deeper Phase-0 narrative is deferred to a doc pass). - **Open / risks:** the Contacts pagination, the purge, and the email-sync auto-recovery are **live-smoke / not yet device-confirmed**. Carried: **Claude/Architect path unverified live on the box**; vision OCR small-in-frame misread (`mara.com→marac.com`); doc drift — `crm-overview.md` narrative + `EVALUATION.md` still describe `lp_profiles` (the active API/schema claims were fixed; the deeper Phase-0 narrative is deferred to a doc pass).
+9 -2
View File
@@ -88,6 +88,13 @@
- **Consolidate `contacts``fundraising_contacts` into one linked model.** Goal (Grant): everyone in `contacts` maps to a `fundraising_investors` row (an individual maps to their own row). Today `contacts` is the canonical person directory (FK target for `communications`/`opportunities`); `fundraising_contacts.contact_id` (migration `0004`) points INTO it; the mobile Contacts page reads `contacts`. Three populations: **A** linked (grid pill ↔ contact), **B** `contacts`-only (imported prospects / manual adds — need a grid row), **C** pill-only (`fundraising_contacts.contact_id IS NULL` — need a contact row). **Census-first:** before designing any migration, count A/B/C on the box — Grant runs the SQL himself (he is **not** providing a DB copy), so hand him a counts-only script. The census decides whether this is a ~20-row cleanup or a ~300-row structural migration with `communications`/`opportunities` repointing. Then Grant reconciles B (add grid rows/pills) and C (add contact rows) and ensures all are linked. **(v0.1.0:105) A TEMPORARY admin census ships to read A/B/C off the box without shell access: `GET /api/admin/contacts-census` (`handle_contacts_census`) + a Settings → Admin "Run census" button, mirroring `backend/scripts/contacts_census.sql` (counts only). DELETE the endpoint + route + button after the numbers are captured — all tagged `TEMPORARY` in code.** - **Consolidate `contacts``fundraising_contacts` into one linked model.** Goal (Grant): everyone in `contacts` maps to a `fundraising_investors` row (an individual maps to their own row). Today `contacts` is the canonical person directory (FK target for `communications`/`opportunities`); `fundraising_contacts.contact_id` (migration `0004`) points INTO it; the mobile Contacts page reads `contacts`. Three populations: **A** linked (grid pill ↔ contact), **B** `contacts`-only (imported prospects / manual adds — need a grid row), **C** pill-only (`fundraising_contacts.contact_id IS NULL` — need a contact row). **Census-first:** before designing any migration, count A/B/C on the box — Grant runs the SQL himself (he is **not** providing a DB copy), so hand him a counts-only script. The census decides whether this is a ~20-row cleanup or a ~300-row structural migration with `communications`/`opportunities` repointing. Then Grant reconciles B (add grid rows/pills) and C (add contact rows) and ensures all are linked. **(v0.1.0:105) A TEMPORARY admin census ships to read A/B/C off the box without shell access: `GET /api/admin/contacts-census` (`handle_contacts_census`) + a Settings → Admin "Run census" button, mirroring `backend/scripts/contacts_census.sql` (counts only). DELETE the endpoint + route + button after the numbers are captured — all tagged `TEMPORARY` in code.**
### Captured tweaks (Matrix, 2026-06-18/20)
*Small UI/UX + capture-quality items captured via Matrix; not yet scheduled.*
- **[P2] Backup history (Settings) defaults minimized, chevron-expand, pinned to the bottom** — it's rarely viewed, so it shouldn't take prime space. Frontend-only (`frontend/index.html`). (2026-06-18)
- **[P2] Preserve the active tab across a page refresh** — a refresh currently snaps back to the top/default tab. Persist the selected tab (e.g. localStorage / URL hash) and rehydrate on load. Frontend-only. (2026-06-18)
- **[P2] Email capture matches an investor on `To:`/`From:` only, not `Cc:`** — today if an investor's address appears anywhere on a message landing in a team mailbox (including when a teammate is merely cc'd on an outbound reply to the investor), it logs a spurious "received from investor" entry. Restrict the investor-link match to the to/from headers so a cc doesn't create a phantom inbound note. `backend/email_integration/` matching (see `docs/guides/email.md`). (2026-06-20)
### Follow-ups/reminders + NL search + bot grid-mutations (agreed plan, 2026-06-18) ### Follow-ups/reminders + NL search + bot grid-mutations (agreed plan, 2026-06-18)
*Agreed with Grant 2026-06-18. Three workstreams, sequenced **W1 → W2 → W3**. **Overarching constraint (Grant):** the dominant risk is **leaking LP data (names, $, notes, contacts) to third-party LLMs — NOT write-safety.** A wrong number is recoverable; investor substance reaching Claude is not. Consequences: W2 keeps LP rows off Claude (only the question text + schema vocabulary leave the box; entity names resolved locally); W3 keeps bot mutation-parsing on local Qwen. Because this DB *logs* commitments/pipeline but doesn't move money, a bot mutation is low-stakes → **any team member may approve one in Matrix**; the guardrail is "the bot can't silently mass-change numbers," enforced by the per-mutation human approval gate, not a tight money gate.* *Agreed with Grant 2026-06-18. Three workstreams, sequenced **W1 → W2 → W3**. **Overarching constraint (Grant):** the dominant risk is **leaking LP data (names, $, notes, contacts) to third-party LLMs — NOT write-safety.** A wrong number is recoverable; investor substance reaching Claude is not. Consequences: W2 keeps LP rows off Claude (only the question text + schema vocabulary leave the box; entity names resolved locally); W3 keeps bot mutation-parsing on local Qwen. Because this DB *logs* commitments/pipeline but doesn't move money, a bot mutation is low-stakes → **any team member may approve one in Matrix**; the guardrail is "the bot can't silently mass-change numbers," enforced by the per-mutation human approval gate, not a tight money gate.*
@@ -131,8 +138,8 @@ Use the **matrix-bridge** repo's pattern to listen on a dedicated ten31-database
**Long-term — extract the intake bot to its own repo (recommended, not yet done).** Containerizing from this monorepo is the pragmatic now-state, but the bot is a genuinely separate deployable (own process, own `matrix-nio` dep, own lifecycle); its only CRM coupling is the HTTP API (a clean network contract) plus ~40 lines of stdlib Spark client (cheap to vendor). The tell: the spark-control Update button would run `git reset --hard origin/main` on the **whole CRM clone** — wrong blast radius. `matrix-bridge` is already a dedicated repo; mirror it. The extraction is a migration (new Gitea repo, move code + tests + guide, vendor the client, re-point the Spark deploy), so it's deferred until worth the lift — do it *before* wiring the spark-control card if both land in the same push. **Long-term — extract the intake bot to its own repo (recommended, not yet done).** Containerizing from this monorepo is the pragmatic now-state, but the bot is a genuinely separate deployable (own process, own `matrix-nio` dep, own lifecycle); its only CRM coupling is the HTTP API (a clean network contract) plus ~40 lines of stdlib Spark client (cheap to vendor). The tell: the spark-control Update button would run `git reset --hard origin/main` on the **whole CRM clone** — wrong blast radius. `matrix-bridge` is already a dedicated repo; mirror it. The extraction is a migration (new Gitea repo, move code + tests + guide, vendor the client, re-point the Spark deploy), so it's deferred until worth the lift — do it *before* wiring the spark-control card if both land in the same push.
### In-app camera business-card intake — PLAN written, awaiting go-ahead (Grant, 2026-06-20) ### In-app camera business-card intake — DONE & live (v0.1.0:100, device-confirmed 2026-06-20)
*A camera button in the mobile top bar (left of the quick-log pencil) → take/choose a photo → the same vision-transcribe → parse → fuzzy-match → edit/approve/reject flow the Matrix card intake (M3) runs, surfaced as an inline mobile sheet. **Detailed plan: `docs/handoffs/in-app-card-intake-plan.md`.*** Key finding: the reusable core is nio-free and already reachable from the CRM (`server.py` imports `llm`; `matrix_intake/parse.py`+`spark.py` import no `matrix-nio`), so it's **one endpoint** (`POST /api/intake/card`) + **one mobile component**, no bot refactor / new dep / migration. Decisions LOCKED (Grant 2026-06-20): `source="app_card"`, form-field edits only (no NL-edit) for v1, any-authenticated-member access, ships in the s9pk. Ready to build on go-ahead. Reuses the New-investor sheet pre-filled + the proven `.quicklog-btn svg` icon-sizing fix for the camera button. *Shipped: a camera button in the mobile top bar (left of the quick-log pencil) → take/choose a photo → vision-transcribe → parse → fuzzy-match → edit/approve/reject, surfaced as an inline mobile sheet (`source="app_card"`, form-field edits only, any-authenticated-member). The reusable core is nio-free and already reachable from the CRM (`server.py` imports `llm`; `matrix_intake/parse.py`+`spark.py` import no `matrix-nio`), so it landed as **one endpoint** (`POST /api/intake/card`) + **one mobile component** no bot refactor / new dep / migration; reuses the New-investor sheet pre-filled + the `.quicklog-btn svg` icon-sizing fix. History: commits `463f624` (feature) / `622d454` (handoff). (Plan doc removed — git history is the record.)*
### Scoped service-credential auth path for automated CRM writers ### Scoped service-credential auth path for automated CRM writers
*Surfaced 2026-06-17 while deploying the Matrix intake bot. **Decision: defer — the bot uses a dedicated member username/password for now.** The CRM has no API-key/service-token path; its only auth is username+password → JWT. A dedicated **member** login is appropriately scoped against what matters operationally (no admin: can't manage users, reset data, or change settings) and unblocks the live smoke today.* *Surfaced 2026-06-17 while deploying the Matrix intake bot. **Decision: defer — the bot uses a dedicated member username/password for now.** The CRM has no API-key/service-token path; its only auth is username+password → JWT. A dedicated **member** login is appropriately scoped against what matters operationally (no admin: can't manage users, reset data, or change settings) and unblocks the live smoke today.*
-133
View File
@@ -1,133 +0,0 @@
# Plan: in-app business-card intake (mobile camera → transcribe → approve)
**Status:** PLAN — decisions locked 2026-06-20 (see end); awaiting Grant's go-ahead to build.
**Goal:** a camera button in the mobile top bar (left of the quick-log pencil) → take a photo or
pick one from the library → the **same** vision-transcribe → parse → fuzzy-match → edit/approve/reject
flow the Matrix card intake runs (M3), surfaced as an inline mobile sheet instead of a Matrix thread.
## Why this is cheaper than it looks
The reusable core is already nio-free and already reachable from the CRM monolith:
- `server.py` **already** imports the Spark client: `sys.path.insert(.../ingest); import llm`
(`server.py:6485`). `llm.chat_vision` (multimodal → Spark Control passthrough) is the exact call
the bot's card OCR uses.
- `backend/matrix_intake/parse.py` imports only `json`, `re`, `spark`; `spark.py` imports only
`os`, `sys`, `llm`. **Neither imports `matrix-nio`.** So `server.py` can import `parse` +
`spark` the same way it imports `llm`, with **no bot refactor and no matrix-nio in the CRM**.
- Match (`find_intake_match` / `find_intake_candidates`) and the write
(`/api/fundraising/log-communication`, create-if-missing + contact upsert + audit) are **already**
in `server.py`. The fuzzy matcher just got the generic-word fix (v-next).
- The whole `backend/` tree (incl. `ingest/` + `matrix_intake/`) is `COPY`'d into the s9pk image
(`start9/0.4/Dockerfile:62`), so the imports resolve on the box at runtime.
So this is **one new endpoint + one new mobile component**, reusing everything downstream. No
migration, no new dependency, no change to the live Matrix bot.
## Architecture decision
**Recommended (Option A): `server.py` imports `matrix_intake.parse` + `matrix_intake.spark`
directly.** Add `backend/matrix_intake` to `sys.path` (guarded like the existing `ingest` insert)
and `import parse, spark`. Pros: zero bot churn, reuses the tested transcribe + email-integrity
parse verbatim, no divergence. Con: the CRM now imports two modules out of the bot package — a mild
coupling, but they're pure (no nio, no CRM HTTP; we do NOT import the bot's `crm_client`).
**Alternative (Option B): extract a shared `backend/intake_core/`** (`vision.py` = transcribe,
`extract.py` = parse/normalize/revise) that both the bot and `server.py` import. Cleaner ownership,
but it refactors the live, working bot for no functional gain right now. **Defer** unless the
coupling in A bites later (e.g. if the bot's `parse` grows nio-coupled helpers).
> Module-name caution (from the matrix-intake guide): the bot imports by **bare name** (`import
> spark`, `import llm`, `import config`). `server.py` already has `ingest/` on its path (so `llm`,
> `config`, `http_util` resolve). Adding `matrix_intake/` lets `parse`/`spark` resolve with **no
> name overlap** against `ingest`'s modules — verified. Import only `parse` + `spark` (not
> `crm_client`/`settings`/`bot`).
## Server: one new endpoint
`POST /api/intake/card` — authenticated **member+** (it's a UI feature for the team; NOT bot-gated,
NOT admin-only). Body is JSON, image as base64 (no multipart parsing in the stdlib handler):
```
{ "image_b64": "<base64>", "mime": "image/jpeg" }
→ 200 { "transcription": "...",
"proposal": { investor_name, contact_name, contact_email, contact_title,
city, linkedin_url, phone, mobile, note },
"match": { id, name } | null,
"candidates": [ { id, name, score, matched_on }, ... ] }
```
Handler flow (mirrors `bot.handle_card``handle_intake`, minus Matrix):
1. Auth (reuse the standard authenticated gate).
2. Decode + size-guard the base64; default mime `image/jpeg`.
3. `transcription = spark.transcribe_card(image_b64, mime)` (Spark Control vision). On error → 502
`{ ok:false, reason:"vision_unavailable" }`; on `<5` chars → 200 `{ ok:false, reason:"unreadable" }`.
4. `proposal = parse.parse_message("New investor — from a business card:\n"+transcription, roster=None)`
— the **email-integrity rule rides along** (an address/phone/LinkedIn is kept only if it literally
appears in the transcription, never minted), exactly as in Matrix. `roster=None` for v1 (the
team-roster framing is a Spark-side env; the box can pass None — prior parse behavior).
5. `match = find_intake_match(...)`, `candidates = find_intake_candidates(...)` from the proposal's
`investor_name`/`contact_email`.
6. Return `{ transcription, proposal, match, candidates }`.
**Approve** reuses the existing write — the mobile sheet POSTs the (possibly-edited) proposal to
`POST /api/fundraising/log-communication` with `create_investor_if_missing` (new) or the picked
`_match_id`/candidate row (existing), tagged **`source="app_card"`** (a new provenance value
distinct from `matrix_card`/`matrix_intake`). **Reject** is client-only (discard the sheet) — no
server call, nothing written.
**No NL-edit for v1.** In a form UI, editing the fields directly is easier than chatting "change the
email to…". The fields are inline-editable in the sheet. (A `/api/intake/card/revise` wrapping
`parse.revise` is a possible later enhancement if Grant wants conversational corrections.)
## Mobile UI: one new component
`MobileCardCapture` in the shell top bar, **left of `MobileQuickLog`** (the cluster at
`index.html:14609`). Reuses `<BottomSheet>`, `.sheet-input`, `StageChip`, and the New-investor
sheet patterns (8g).
- **Trigger:** a camera-icon button + a hidden `<input type="file" accept="image/*">`
**omit `capture`** so iOS shows the native action sheet (Take Photo / Photo Library / Browse),
satisfying "take a photo OR use an existing photo." (`capture="environment"` would force the camera
and skip the library — not what we want.)
- **On select:** read the file as a data URL; **downscale via `<canvas>` to ~2000px max dimension**
before base64 (native, no library) — keeps the payload under the StartOS reverse-proxy body cap
(the matrix-intake guide's known `413` risk) and the model downscales to ~2 MP anyway. Don't
over-shrink (hurts OCR). Show a "📇 Reading the card…" loading state.
- **POST** to `/api/intake/card`; on `unreadable`/`vision_unavailable` show a "try a clearer,
well-lit, fill-the-frame photo" message with a Retake button (same UX as the bot's `📇` replies).
- **Approval sheet** (the New-investor sheet, pre-filled from `proposal`): editable investor name +
contact name/email/title/city/phone/mobile/LinkedIn; a small "Existing investor?" block when
`match`/`candidates` came back (pick a candidate row, or "Add as new"); **Approve** → the
log-communication write; **Reject/Retake** → discard.
- **Icon:** reuse the proven `.quicklog-btn svg` sizing fix (explicit CSS `width/height` + `flex:none`)
so the new camera SVG doesn't hit the same iOS sole-flex-child collapse.
## Scope / effort / risk
- **Server:** ~1 endpoint + the `parse`/`spark` import guard + an `app_card` source tag. Small.
- **Frontend:** ~1 component (camera button + capture input + canvas downscale + approval sheet) +
CSS for the button. Medium (the approval sheet is the New-investor sheet pre-filled).
- **No** migration, **no** new dependency, **no** Matrix-bot change.
- **Tests:** an endpoint test that stubs `spark.transcribe_card` (like `test_spark.py` does) and
asserts the transcribe→parse→match shape + the email-integrity passthrough; the real vision/OCR
path stays **live-smoke only** (same as Matrix M3). Frontend is inspection + on-device.
- **Risks:** (1) large-image upload vs the reverse-proxy cap — mitigated by client-side canvas
downscale; (2) iOS file-input behavior across Safari/standalone-PWA — verify on-device; (3) OCR
accuracy is the **shared** Matrix limitation (resolution-bound; "fill the frame"), not new here;
(4) the `parse`/`spark` import must be lazy/guarded so a dev `./start.sh` without Spark reachable
still boots (the endpoint just 502s) — mirror the existing lazy `import llm` at `server.py:6485`.
## Decisions (locked, Grant 2026-06-20)
1. **Provenance tag = `source="app_card"`** — distinct from `matrix_card` / `matrix_intake`.
2. **v1 = editable form fields only** — no conversational NL-edit on the card (no
`/api/intake/card/revise`); the user taps a field and fixes it in the sheet. NL-edit can follow
later if it proves wanted.
3. **Access = any authenticated member** — the camera button + `POST /api/intake/card` use the
standard authenticated gate (not admin-only, not bot-gated); a human still approves every write.
4. **Delivery = the s9pk** (server endpoint + frontend) — a normal version bump → build → box
install, like v99 (NOT bot-only like the Matrix M3).
Build scope is otherwise as specified above. Mobile-only (top-bar button); same captured fields +
literal-in-source integrity rule as the Matrix card flow.