Remove Instructions/Feedback + lp_profiles; sync retry, purge, mobile fixes (v0.1.0:104)

Removals (net -570 lines):
- Delete the Instructions and Feedback (feature_requests) pages + backend.
- Retire lp_profiles + investor_type across server, ingest, and seeds; migration
  0008 drops both empty tables (a sanctioned one-off exception to
  never-hard-delete). 0001's lp_profiles ALTER is removed so a fresh DB doesn't
  break the migration chain (live DBs already applied it).

Fixes:
- Email sync: a transient timeout no longer terminally parks a mailbox; the
  scheduler retries 'retrying' each cycle and re-includes errored accounts on an
  hourly backoff, so stuck mailboxes self-heal.
- Mobile Contacts: page through the full directory (server caps 500/page) -- one
  fetch silently truncated at 720, hiding people from the list and from search.
- Mobile email review: clock icon to set a reminder inline; approval cards show
  date/time.

New:
- Admin-only purge of soft-deleted rows (Settings -> Admin; type-to-confirm,
  refuses any row still linked to live data).

Tests: 45/45 (adds test_sync_ready + test_purge_soft_deleted). Reviewer pass
applied (NULL reminders.contact_id on contact purge). Bumped to v0.1.0:104.
This commit is contained in:
Keysat
2026-06-20 20:06:11 -05:00
parent 985cba3c81
commit 1564c087bf
21 changed files with 629 additions and 694 deletions
+11 -10
View File
@@ -1,6 +1,6 @@
# Ten31 Venture CRM + Agentic System — AGENTS.md
**The foundation is a self-hosted venture-fund CRM** — a purpose-built fundraising tool that replaced Airtable to (1) keep sensitive LP/prospect data off third-party servers, (2) drop subscription cost, and (3) fit the fund's workflow: managing ~150 existing LPs, tracking 250+ prospects, and running the capital-raise pipeline. Core CRM domain: contacts (investor/prospect/advisor), organizations, opportunities (the deal pipeline), and communications; investor commitments live in the canonical `fundraising_*` grid (the legacy single-fund `lp_profiles` table was retired in v0.1.0:78). The fund (Ten31, ~$200M AUM, bitcoin/energy/AI thesis) runs it on a Start9 box, accessed over ClearNet (StartOS StartTunnel) with app-level user auth by a team of ~5 (Tailscale is not in use). Schema/API tour: `docs/crm-overview.md`.
**The foundation is a self-hosted venture-fund CRM** — a purpose-built fundraising tool that replaced Airtable to (1) keep sensitive LP/prospect data off third-party servers, (2) drop subscription cost, and (3) fit the fund's workflow: managing ~150 existing LPs, tracking 250+ prospects, and running the capital-raise pipeline. Core CRM domain: contacts (investor/prospect/advisor), organizations, opportunities (the deal pipeline), and communications; investor commitments live in the canonical `fundraising_*` grid (the legacy single-fund `lp_profiles` table was retired in v0.1.0:78 and dropped in v0.1.0:104). The fund (Ten31, ~$200M AUM, bitcoin/energy/AI thesis) runs it on a Start9 box, accessed over ClearNet (StartOS StartTunnel) with app-level user auth by a team of ~5 (Tailscale is not in use). Schema/API tour: `docs/crm-overview.md`.
**The agentic system is new functionality built on top of that CRM** — an in-house AI layer to widen the fundraising funnel, sharpen the thesis, and automate outreach drafting. Frontier reasoning runs on Claude (Agent SDK/API); privacy-sensitive and bulk work runs on local DGX Spark models via the **Spark Control** gateway. **Phase 0/1 — no live outward-facing agents; agents draft, humans send.**
@@ -70,8 +70,8 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude
## Conventions
- **Investor model — the grid is canonical (since v0.1.0:78).** The `fundraising_*` grid is the **system of record**: an investor entity (row) → many contact "pills" → per-fund commitments. The classic `contacts` table is a **read-only per-person directory**, auto-populated from the grid — create/edit people in the grid, not the Contacts page. Email capture rolls multiple people up to one investor. The legacy single-fund `lp_profiles` model is **retired** (empty table kept, per never-hard-delete). Reconciling grid ↔ classic `contacts` to canonical IDs is the core entity-resolution task — see `docs/crm-overview.md`. **Derived read-only columns** (`pipeline`, `pipeline_stage`, `opportunity_id`, `reminder_status`, `existing_investor`, `last_activity_at`, `staleness`) are computed live and **injected on GET, never persisted** — any new one MUST be added to BOTH strip points (`server.py` `_computed_row_values` + frontend `stripComputedRows`) or it dirties the autosave / leaks into the blob. **Exception — the contact-pill email-heal** (`fundraising_contact_emails_by_row`, injected in `handle_get_fundraising_state`, v0.1.0:99): it fills a *blank* pill `email` from the linked classic contact and deliberately has **NO** strip point, because `email` is a real blob field, not a computed column — the next one-row save legitimately persists the recovered value (it's a self-healing backfill; don't "fix" it by adding a strip point). Pipeline stage is the 4-stage funnel `lead→engaged→diligence→commitment` (`PIPELINE_STAGES`), terminal at commitment.
- **Soft-delete only:** `deleted_at` and/or `status='retired'`; never hard-delete. Every READ path must filter `deleted_at IS NULL` — list handlers, get-by-id, nested related-data sub-selects, **and aggregate sub-selects (`COUNT`/`SUM`/`MAX`)**. Audits found leaks in all of these (2026-06-12 detail + nested; 2026-06-13 list-view `contact_count`/`total_funded`/`comm_count`); the **opportunities/pipeline** aggregates were fixed in v0.1.0:87 (`handle_pipeline_report` + dashboard pipeline metrics now filter `deleted_at`), but the **reports** subsystem's **communications-side** aggregates (dashboard `recent_comms`/`comms_this_month`/`meetings_this_month`, activity report) still leak (see Current state). Regression-guarded by `backend/test_soft_delete_reads.py` (+ `test_reminders.py` for the reminders read paths, incl. the recency rollup whose email-activity liveness signal is `email_account_messages.deleted_at`, not `emails`). (Thesis has a subtlety here — see the thesis guide.)
- **Investor model — the grid is canonical (since v0.1.0:78).** The `fundraising_*` grid is the **system of record**: an investor entity (row) → many contact "pills" → per-fund commitments. The classic `contacts` table is a **read-only per-person directory**, auto-populated from the grid — create/edit people in the grid, not the Contacts page. Email capture rolls multiple people up to one investor. The legacy single-fund `lp_profiles` model is **retired and dropped** — the (empty) table was physically removed in v0.1.0:104 via migration `0008_drop_retired_tables`, a deliberate, documented one-off exception to never-hard-delete. The in-app **Instructions** and **Feedback** (`feature_requests`) pages were removed in the same release (the `feature_requests` table was dropped too). Reconciling grid ↔ classic `contacts` to canonical IDs is the core entity-resolution task — see `docs/crm-overview.md`. **Derived read-only columns** (`pipeline`, `pipeline_stage`, `opportunity_id`, `reminder_status`, `existing_investor`, `last_activity_at`, `staleness`) are computed live and **injected on GET, never persisted** — any new one MUST be added to BOTH strip points (`server.py` `_computed_row_values` + frontend `stripComputedRows`) or it dirties the autosave / leaks into the blob. **Exception — the contact-pill email-heal** (`fundraising_contact_emails_by_row`, injected in `handle_get_fundraising_state`, v0.1.0:99): it fills a *blank* pill `email` from the linked classic contact and deliberately has **NO** strip point, because `email` is a real blob field, not a computed column — the next one-row save legitimately persists the recovered value (it's a self-healing backfill; don't "fix" it by adding a strip point). Pipeline stage is the 4-stage funnel `lead→engaged→diligence→commitment` (`PIPELINE_STAGES`), terminal at commitment.
- **Soft-delete only:** `deleted_at` and/or `status='retired'`; never hard-delete. Every READ path must filter `deleted_at IS NULL` — list handlers, get-by-id, nested related-data sub-selects, **and aggregate sub-selects (`COUNT`/`SUM`/`MAX`)**. Audits found leaks in all of these (2026-06-12 detail + nested; 2026-06-13 list-view `contact_count`/`total_funded`/`comm_count`); the **opportunities/pipeline** aggregates were fixed in v0.1.0:87 (`handle_pipeline_report` + dashboard pipeline metrics now filter `deleted_at`), but the **reports** subsystem's **communications-side** aggregates (dashboard `recent_comms`/`comms_this_month`/`meetings_this_month`, activity report) still leak (see Current state). Regression-guarded by `backend/test_soft_delete_reads.py` (+ `test_reminders.py` for the reminders read paths, incl. the recency rollup whose email-activity liveness signal is `email_account_messages.deleted_at`, not `emails`). (Thesis has a subtlety here — see the thesis guide.) **The ONE sanctioned hard-delete is the admin purge** (Settings → Admin "Purge Deleted Data"; `GET/POST /api/admin/soft-deleted[/purge]`, `handle_purge_soft_deleted`, v0.1.0:104): a guarded, type-to-confirm maintenance tool for clearing dummy/test data that hard-deletes ONLY `deleted_at IS NOT NULL` rows across contacts/orgs/opps/comms and **refuses (409) any contact/org whose `ON DELETE CASCADE`/`SET NULL` would touch a LIVE row** (and NULLs the bare logical-FK back-refs `fundraising_contacts.contact_id` + `reminders.contact_id`). Guarded by `backend/test_purge_soft_deleted.py`. It does **not** reach blank *live* grid rows (the grid blob has no soft-delete axis) — that's a separate cleanup.
- **Env:** secrets in `.env` (gitignored); names in `.env.example`. Verified names: `ANTHROPIC_API_KEY`, `SPARK_CONTROL_URL`, `SPARK_CONTROL_VERIFY_TLS`, `QDRANT_URL`, `X_API_KEY`, `CRM_DB_PATH`, `CRM_DEV_DB_PATH`. Also used: `CRM_SECRET_KEY` (beta/prod), `CRM_HOST`/`CRM_PORT`, `CRM_DATA_DIR`; digest mailer: `CRM_DIGEST_SENDER` (DWD impersonation sender) + `SMTP_HOST`/`SMTP_PORT`/`SMTP_SECURITY`/`SMTP_FROM`/`SMTP_USERNAME`/`SMTP_PASSWORD` (SMTP fallback); daily digest (Phase B): `CRM_DIGEST_ENABLED` + `CRM_DIGEST_SEND_HOUR` **only seed the first-boot default** — the live control is the DB policy (`app_settings.digest_policy`, set in Settings → Admin).
- **Config placement:** operational/feature toggles live in the **admin panel**, DB-backed via `app_settings` (read-merge through a `load_*_policy(conn)` helper shared by the API + any scheduler; precedence DB-row → env-seed → default), so they're discoverable and take effect live. Reserve StartOS actions / env for **secrets and deploy-time config** (SMTP creds, API keys, DWD sender). Precedent: `digest_policy` (`GET/PATCH /api/admin/digest/policy`), `fundraising_backup_policy`.
- **Agent/bot API access — three roles now (`admin`/`member`/`bot`).** `require_admin` is the only hard gate; everything else is "authenticated" (member, admin, *and* bot all pass). The **`bot` role** (added v0.1.0:89) is authenticated-but-never-admin: `require_bot_or_admin` gates agent-facing endpoints (e.g. `/api/intake/email-proposals*`) so a bot credential reaches *only* what it needs, never user-management/settings/security. Provision it via Settings → Admin edit-user dropdown (kept out of the teammate-invite form). **Two axes to keep separate as more agent capability lands:** the role controls *reach* (which endpoints); the per-feature human draft→approve gate controls *autonomy* (acting unattended). Money/merge/delete mutations stay behind the approval gate regardless of role. Don't build a finer capability/scope system until real NL-mutation endpoints exist to scope against.
@@ -108,11 +108,12 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude
## Current state
_**Box live at v0.1.0:103 (deployed + verified 2026-06-20)** — clean migration chain (…→103, all no-op/frontend-only), server up on :8080. This session = a **mobile-UX feedback batch from Grant's device testing** (101 #15, 102 #6 email bell) + **103 reminders-require-a-date** (mobile + desktop); **Grant device-confirmed the mobile items + the date behavior on-device.** **The fundraising grid + email capture is the canonical system of record.** History: git log + `start9/0.4/startos/versions/`._
_**Box live at v0.1.0:104 (deployed + verified 2026-06-20)** — clean StartOS migration chain (…→104) and the in-app SQL chain through `0008_drop_retired_tables` (`lp_profiles` + `feature_requests` physically dropped on the box), server up on :8080. This session = a **removal + bug-fix + feature batch** (below). **The fundraising grid + email capture is the canonical system of record.** History: git log + `start9/0.4/startos/versions/`._
- **Mobile UX batch (Grant device feedback) — BUILT + LIVE (v0.1.0:101102, 2026-06-20), on-device pass pending.** Six items (durable detail in the Design bullet → "Post-8 mobile-feedback primitives"): [1] ✕-clear on search/picker fields (`ClearableInput`); [2] tappable Grid contact pills (name→Contacts deep-link, email→mailto); [3] grid search already matched contact names — verified, no change; [4a] full-height Pipeline swipe area with bottom-pinned dots; [4b] editable pipeline `expected_amount` (add-to-pipeline + card detail, `PUT /api/opportunities/{id}`); [5] bottom sheets lift above the keyboard (visualViewport); [6] **`MobileEmailBell`** — admin-only email-approval bell, a third surface over `email_activity_proposals` that auto-syncs with the web panel + Matrix room.
- **Reminders require a due date — BUILT + LIVE (v0.1.0:103, deployed + verified 2026-06-20).** **Every** create surface (mobile add-investor / standalone Reminders / Grid-detail, **and desktop** Reminders page + grid modal) pre-fills the date to +1 week (editable) and blocks an empty save (`reminderDefaultDue()`); edit paths pre-fill it for legacy date-less reminders too. Detail in the Design bullet.
- **Verification: render-smoke green** (build-gated — JSX transforms + app mounts), reviewer-agent **APPROVE, no blockers** across all batches + a holistic pass (nits applied: ClearableInput conditional padding, bell `busyRef` double-submit guard, disabled-button dimming, reminder edit-path default-fill). All new work is **frontend-only — no schema / migration / dependency change**, so backend is untouched (43/43 backend tests still green from v100). New UI behavior is **live-smoke / on-device only** (jsdom can't drive touch/keyboard/mailto).
- **On-device — CONFIRMED (Grant, 2026-06-20):** the v101102 mobile items (✕ clears, tappable contacts name→contact & email→mail, Pipeline swipe + bottom dots, amount round-trips, keyboard-lifted picker, the email bell) + the v103 date requirement all look good on his phone.
- **Next:** (A) one remaining spot-check on the bell — the **approve-on-phone → Matrix-thread-clears** round-trip (the UI works; confirm the bidirectional sync end-to-end with a real proposal). (B) Carried from v100: #7 real-card spot-checks + the standing mobile light/dark + PWA-install gate.
- **Open / risks:** `.pipeline-screen { height:100% }` leans on the `.content` flex chain for a definite height — confirm the swipe area fills + scrolls on Grant's iOS (resolves on iOS 16+; no speculative patch applied). Bell + amount-edit are admin/live-smoke only. Carried: **Claude/Architect path unverified live on the box**; vision OCR can misread a small-in-frame card (`mara.com→marac.com`, temp 0); phone/LinkedIn land on the contact record, not the grid pill; PWA iOS status bar fixed `black` in light theme; doc drift — `crm-overview.md`/`EVALUATION.md` still call `lp_profiles` live.
- **Removed (v0.1.0:104):** the **Instructions** + **Feedback** (`feature_requests`) pages + backend, and `lp_profiles` + `investor_type` (across server / ingest / seeds). Migration `0008` drops both empty tables (a sanctioned one-off exception to never-hard-delete); `0001`'s `lp_profiles` ALTER was removed so a fresh DB doesn't break the migration chain. Net 570 lines.
- **Fixes (v0.1.0:104):** [B] email sync no longer terminally parks a mailbox on a transient timeout — `'retrying'` retries every cycle, `'error'` re-included on an hourly backoff, so **Grant's & Jonathan's stuck mailboxes self-heal on this deploy** (`test_sync_ready.py`). [C] clock icon on the mobile email Review-log sets a reminder inline. [D] email-approval cards show date/time. **[Contacts 500-cap]** the mobile Contacts directory now pages through ALL contacts (was truncated at 500 of 720 — hid people from the list *and* search).
- **New (v0.1.0:104):** admin-only **Purge Deleted Data** (Settings → Admin) — guarded, type-to-confirm hard-delete of soft-deleted rows; see the soft-delete convention + `test_purge_soft_deleted.py`.
- **Verification:** **45/45** backend, render-smoke green, reviewer-agent APPROVE after fixing **1 blocker** (contact purge left a dangling `reminders.contact_id` — now NULLed + test-guarded). New UI behavior is **live-smoke / on-device only** (jsdom can't drive touch).
- **Bug A — Grant is handling:** `odell/marty/finance/ten31@` can't enroll for email capture ("could not resolve user_id") because the enroll flow requires a CRM `users` row; Grant is creating user accounts for those mailboxes.
- **Next:** (A) confirm the two stuck mailboxes pulled current + Grant's 4 new mailbox users enroll; (B) **retire `contact_type`** — replace the Contacts Investors/Prospects tabs + TYPE badge with grid-derived `existing_investor`/`pipeline_stage`, then drop the column (see ROADMAP); (C) **contacts ↔ `fundraising_contacts` consolidation** — census-first (count A/linked, B/contacts-only, C/pill-only on the box; see ROADMAP); (D) carried: bell approve-on-phone → Matrix-thread-clears round-trip spot-check.
- **Open / risks:** the Contacts pagination, the purge, and the email-sync auto-recovery are **live-smoke / not yet device-confirmed**. Carried: **Claude/Architect path unverified live on the box**; vision OCR small-in-frame misread (`mara.com→marac.com`); doc drift — `crm-overview.md` narrative + `EVALUATION.md` still describe `lp_profiles` (the active API/schema claims were fixed; the deeper Phase-0 narrative is deferred to a doc pass).