From 1564c087bf44cd18c7824d0bf390346aee8b8d55 Mon Sep 17 00:00:00 2001 From: Keysat Date: Sat, 20 Jun 2026 20:06:11 -0500 Subject: [PATCH] Remove Instructions/Feedback + lp_profiles; sync retry, purge, mobile fixes (v0.1.0:104) Removals (net -570 lines): - Delete the Instructions and Feedback (feature_requests) pages + backend. - Retire lp_profiles + investor_type across server, ingest, and seeds; migration 0008 drops both empty tables (a sanctioned one-off exception to never-hard-delete). 0001's lp_profiles ALTER is removed so a fresh DB doesn't break the migration chain (live DBs already applied it). Fixes: - Email sync: a transient timeout no longer terminally parks a mailbox; the scheduler retries 'retrying' each cycle and re-includes errored accounts on an hourly backoff, so stuck mailboxes self-heal. - Mobile Contacts: page through the full directory (server caps 500/page) -- one fetch silently truncated at 720, hiding people from the list and from search. - Mobile email review: clock icon to set a reminder inline; approval cards show date/time. New: - Admin-only purge of soft-deleted rows (Settings -> Admin; type-to-confirm, refuses any row still linked to live data). Tests: 45/45 (adds test_sync_ready + test_purge_soft_deleted). Reviewer pass applied (NULL reminders.contact_id on contact purge). Bumped to v0.1.0:104. --- AGENTS.md | 21 +- ROADMAP.md | 10 +- backend/email_integration/db.py | 12 +- backend/email_integration/sync.py | 10 + backend/email_integration/test_sync_ready.py | 72 ++ backend/ingest/chunking.py | 9 +- backend/ingest/entity_resolution.py | 9 +- backend/ingest/sync.py | 4 +- backend/ingest/test_entity_resolution.py | 21 +- backend/migrations/0001_phase0_foundation.sql | 6 +- .../0008_drop_retired_tables.down.sql | 42 ++ .../migrations/0008_drop_retired_tables.sql | 15 + backend/scripts/seed_synthetic.py | 21 +- backend/server.py | 232 +++---- backend/test_purge_soft_deleted.py | 157 +++++ docs/crm-overview.md | 8 +- docs/guides/migrations.md | 1 + frontend/index.html | 639 +++++------------- start9/0.4/startos/utils.ts | 5 +- start9/0.4/startos/versions/index.ts | 5 +- start9/0.4/startos/versions/v0.1.0.104.ts | 24 + 21 files changed, 629 insertions(+), 694 deletions(-) create mode 100644 backend/email_integration/test_sync_ready.py create mode 100644 backend/migrations/0008_drop_retired_tables.down.sql create mode 100644 backend/migrations/0008_drop_retired_tables.sql create mode 100644 backend/test_purge_soft_deleted.py create mode 100644 start9/0.4/startos/versions/v0.1.0.104.ts diff --git a/AGENTS.md b/AGENTS.md index 6a74245..96df096 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # Ten31 Venture CRM + Agentic System — AGENTS.md -**The foundation is a self-hosted venture-fund CRM** — a purpose-built fundraising tool that replaced Airtable to (1) keep sensitive LP/prospect data off third-party servers, (2) drop subscription cost, and (3) fit the fund's workflow: managing ~150 existing LPs, tracking 250+ prospects, and running the capital-raise pipeline. Core CRM domain: contacts (investor/prospect/advisor), organizations, opportunities (the deal pipeline), and communications; investor commitments live in the canonical `fundraising_*` grid (the legacy single-fund `lp_profiles` table was retired in v0.1.0:78). The fund (Ten31, ~$200M AUM, bitcoin/energy/AI thesis) runs it on a Start9 box, accessed over ClearNet (StartOS StartTunnel) with app-level user auth by a team of ~5 (Tailscale is not in use). Schema/API tour: `docs/crm-overview.md`. +**The foundation is a self-hosted venture-fund CRM** — a purpose-built fundraising tool that replaced Airtable to (1) keep sensitive LP/prospect data off third-party servers, (2) drop subscription cost, and (3) fit the fund's workflow: managing ~150 existing LPs, tracking 250+ prospects, and running the capital-raise pipeline. Core CRM domain: contacts (investor/prospect/advisor), organizations, opportunities (the deal pipeline), and communications; investor commitments live in the canonical `fundraising_*` grid (the legacy single-fund `lp_profiles` table was retired in v0.1.0:78 and dropped in v0.1.0:104). The fund (Ten31, ~$200M AUM, bitcoin/energy/AI thesis) runs it on a Start9 box, accessed over ClearNet (StartOS StartTunnel) with app-level user auth by a team of ~5 (Tailscale is not in use). Schema/API tour: `docs/crm-overview.md`. **The agentic system is new functionality built on top of that CRM** — an in-house AI layer to widen the fundraising funnel, sharpen the thesis, and automate outreach drafting. Frontier reasoning runs on Claude (Agent SDK/API); privacy-sensitive and bulk work runs on local DGX Spark models via the **Spark Control** gateway. **Phase 0/1 — no live outward-facing agents; agents draft, humans send.** @@ -70,8 +70,8 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude ## Conventions -- **Investor model — the grid is canonical (since v0.1.0:78).** The `fundraising_*` grid is the **system of record**: an investor entity (row) → many contact "pills" → per-fund commitments. The classic `contacts` table is a **read-only per-person directory**, auto-populated from the grid — create/edit people in the grid, not the Contacts page. Email capture rolls multiple people up to one investor. The legacy single-fund `lp_profiles` model is **retired** (empty table kept, per never-hard-delete). Reconciling grid ↔ classic `contacts` to canonical IDs is the core entity-resolution task — see `docs/crm-overview.md`. **Derived read-only columns** (`pipeline`, `pipeline_stage`, `opportunity_id`, `reminder_status`, `existing_investor`, `last_activity_at`, `staleness`) are computed live and **injected on GET, never persisted** — any new one MUST be added to BOTH strip points (`server.py` `_computed_row_values` + frontend `stripComputedRows`) or it dirties the autosave / leaks into the blob. **Exception — the contact-pill email-heal** (`fundraising_contact_emails_by_row`, injected in `handle_get_fundraising_state`, v0.1.0:99): it fills a *blank* pill `email` from the linked classic contact and deliberately has **NO** strip point, because `email` is a real blob field, not a computed column — the next one-row save legitimately persists the recovered value (it's a self-healing backfill; don't "fix" it by adding a strip point). Pipeline stage is the 4-stage funnel `lead→engaged→diligence→commitment` (`PIPELINE_STAGES`), terminal at commitment. -- **Soft-delete only:** `deleted_at` and/or `status='retired'`; never hard-delete. Every READ path must filter `deleted_at IS NULL` — list handlers, get-by-id, nested related-data sub-selects, **and aggregate sub-selects (`COUNT`/`SUM`/`MAX`)**. Audits found leaks in all of these (2026-06-12 detail + nested; 2026-06-13 list-view `contact_count`/`total_funded`/`comm_count`); the **opportunities/pipeline** aggregates were fixed in v0.1.0:87 (`handle_pipeline_report` + dashboard pipeline metrics now filter `deleted_at`), but the **reports** subsystem's **communications-side** aggregates (dashboard `recent_comms`/`comms_this_month`/`meetings_this_month`, activity report) still leak (see Current state). Regression-guarded by `backend/test_soft_delete_reads.py` (+ `test_reminders.py` for the reminders read paths, incl. the recency rollup whose email-activity liveness signal is `email_account_messages.deleted_at`, not `emails`). (Thesis has a subtlety here — see the thesis guide.) +- **Investor model — the grid is canonical (since v0.1.0:78).** The `fundraising_*` grid is the **system of record**: an investor entity (row) → many contact "pills" → per-fund commitments. The classic `contacts` table is a **read-only per-person directory**, auto-populated from the grid — create/edit people in the grid, not the Contacts page. Email capture rolls multiple people up to one investor. The legacy single-fund `lp_profiles` model is **retired and dropped** — the (empty) table was physically removed in v0.1.0:104 via migration `0008_drop_retired_tables`, a deliberate, documented one-off exception to never-hard-delete. The in-app **Instructions** and **Feedback** (`feature_requests`) pages were removed in the same release (the `feature_requests` table was dropped too). Reconciling grid ↔ classic `contacts` to canonical IDs is the core entity-resolution task — see `docs/crm-overview.md`. **Derived read-only columns** (`pipeline`, `pipeline_stage`, `opportunity_id`, `reminder_status`, `existing_investor`, `last_activity_at`, `staleness`) are computed live and **injected on GET, never persisted** — any new one MUST be added to BOTH strip points (`server.py` `_computed_row_values` + frontend `stripComputedRows`) or it dirties the autosave / leaks into the blob. **Exception — the contact-pill email-heal** (`fundraising_contact_emails_by_row`, injected in `handle_get_fundraising_state`, v0.1.0:99): it fills a *blank* pill `email` from the linked classic contact and deliberately has **NO** strip point, because `email` is a real blob field, not a computed column — the next one-row save legitimately persists the recovered value (it's a self-healing backfill; don't "fix" it by adding a strip point). Pipeline stage is the 4-stage funnel `lead→engaged→diligence→commitment` (`PIPELINE_STAGES`), terminal at commitment. +- **Soft-delete only:** `deleted_at` and/or `status='retired'`; never hard-delete. Every READ path must filter `deleted_at IS NULL` — list handlers, get-by-id, nested related-data sub-selects, **and aggregate sub-selects (`COUNT`/`SUM`/`MAX`)**. Audits found leaks in all of these (2026-06-12 detail + nested; 2026-06-13 list-view `contact_count`/`total_funded`/`comm_count`); the **opportunities/pipeline** aggregates were fixed in v0.1.0:87 (`handle_pipeline_report` + dashboard pipeline metrics now filter `deleted_at`), but the **reports** subsystem's **communications-side** aggregates (dashboard `recent_comms`/`comms_this_month`/`meetings_this_month`, activity report) still leak (see Current state). Regression-guarded by `backend/test_soft_delete_reads.py` (+ `test_reminders.py` for the reminders read paths, incl. the recency rollup whose email-activity liveness signal is `email_account_messages.deleted_at`, not `emails`). (Thesis has a subtlety here — see the thesis guide.) **The ONE sanctioned hard-delete is the admin purge** (Settings → Admin "Purge Deleted Data"; `GET/POST /api/admin/soft-deleted[/purge]`, `handle_purge_soft_deleted`, v0.1.0:104): a guarded, type-to-confirm maintenance tool for clearing dummy/test data that hard-deletes ONLY `deleted_at IS NOT NULL` rows across contacts/orgs/opps/comms and **refuses (409) any contact/org whose `ON DELETE CASCADE`/`SET NULL` would touch a LIVE row** (and NULLs the bare logical-FK back-refs `fundraising_contacts.contact_id` + `reminders.contact_id`). Guarded by `backend/test_purge_soft_deleted.py`. It does **not** reach blank *live* grid rows (the grid blob has no soft-delete axis) — that's a separate cleanup. - **Env:** secrets in `.env` (gitignored); names in `.env.example`. Verified names: `ANTHROPIC_API_KEY`, `SPARK_CONTROL_URL`, `SPARK_CONTROL_VERIFY_TLS`, `QDRANT_URL`, `X_API_KEY`, `CRM_DB_PATH`, `CRM_DEV_DB_PATH`. Also used: `CRM_SECRET_KEY` (beta/prod), `CRM_HOST`/`CRM_PORT`, `CRM_DATA_DIR`; digest mailer: `CRM_DIGEST_SENDER` (DWD impersonation sender) + `SMTP_HOST`/`SMTP_PORT`/`SMTP_SECURITY`/`SMTP_FROM`/`SMTP_USERNAME`/`SMTP_PASSWORD` (SMTP fallback); daily digest (Phase B): `CRM_DIGEST_ENABLED` + `CRM_DIGEST_SEND_HOUR` **only seed the first-boot default** — the live control is the DB policy (`app_settings.digest_policy`, set in Settings → Admin). - **Config placement:** operational/feature toggles live in the **admin panel**, DB-backed via `app_settings` (read-merge through a `load_*_policy(conn)` helper shared by the API + any scheduler; precedence DB-row → env-seed → default), so they're discoverable and take effect live. Reserve StartOS actions / env for **secrets and deploy-time config** (SMTP creds, API keys, DWD sender). Precedent: `digest_policy` (`GET/PATCH /api/admin/digest/policy`), `fundraising_backup_policy`. - **Agent/bot API access — three roles now (`admin`/`member`/`bot`).** `require_admin` is the only hard gate; everything else is "authenticated" (member, admin, *and* bot all pass). The **`bot` role** (added v0.1.0:89) is authenticated-but-never-admin: `require_bot_or_admin` gates agent-facing endpoints (e.g. `/api/intake/email-proposals*`) so a bot credential reaches *only* what it needs, never user-management/settings/security. Provision it via Settings → Admin edit-user dropdown (kept out of the teammate-invite form). **Two axes to keep separate as more agent capability lands:** the role controls *reach* (which endpoints); the per-feature human draft→approve gate controls *autonomy* (acting unattended). Money/merge/delete mutations stay behind the approval gate regardless of role. Don't build a finer capability/scope system until real NL-mutation endpoints exist to scope against. @@ -108,11 +108,12 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude ## Current state -_**Box live at v0.1.0:103 (deployed + verified 2026-06-20)** — clean migration chain (…→103, all no-op/frontend-only), server up on :8080. This session = a **mobile-UX feedback batch from Grant's device testing** (101 #1–5, 102 #6 email bell) + **103 reminders-require-a-date** (mobile + desktop); **Grant device-confirmed the mobile items + the date behavior on-device.** **The fundraising grid + email capture is the canonical system of record.** History: git log + `start9/0.4/startos/versions/`._ +_**Box live at v0.1.0:104 (deployed + verified 2026-06-20)** — clean StartOS migration chain (…→104) and the in-app SQL chain through `0008_drop_retired_tables` (`lp_profiles` + `feature_requests` physically dropped on the box), server up on :8080. This session = a **removal + bug-fix + feature batch** (below). **The fundraising grid + email capture is the canonical system of record.** History: git log + `start9/0.4/startos/versions/`._ -- **Mobile UX batch (Grant device feedback) — BUILT + LIVE (v0.1.0:101–102, 2026-06-20), on-device pass pending.** Six items (durable detail in the Design bullet → "Post-8 mobile-feedback primitives"): [1] ✕-clear on search/picker fields (`ClearableInput`); [2] tappable Grid contact pills (name→Contacts deep-link, email→mailto); [3] grid search already matched contact names — verified, no change; [4a] full-height Pipeline swipe area with bottom-pinned dots; [4b] editable pipeline `expected_amount` (add-to-pipeline + card detail, `PUT /api/opportunities/{id}`); [5] bottom sheets lift above the keyboard (visualViewport); [6] **`MobileEmailBell`** — admin-only email-approval bell, a third surface over `email_activity_proposals` that auto-syncs with the web panel + Matrix room. -- **Reminders require a due date — BUILT + LIVE (v0.1.0:103, deployed + verified 2026-06-20).** **Every** create surface (mobile add-investor / standalone Reminders / Grid-detail, **and desktop** Reminders page + grid modal) pre-fills the date to +1 week (editable) and blocks an empty save (`reminderDefaultDue()`); edit paths pre-fill it for legacy date-less reminders too. Detail in the Design bullet. -- **Verification: render-smoke green** (build-gated — JSX transforms + app mounts), reviewer-agent **APPROVE, no blockers** across all batches + a holistic pass (nits applied: ClearableInput conditional padding, bell `busyRef` double-submit guard, disabled-button dimming, reminder edit-path default-fill). All new work is **frontend-only — no schema / migration / dependency change**, so backend is untouched (43/43 backend tests still green from v100). New UI behavior is **live-smoke / on-device only** (jsdom can't drive touch/keyboard/mailto). -- **On-device — CONFIRMED (Grant, 2026-06-20):** the v101–102 mobile items (✕ clears, tappable contacts name→contact & email→mail, Pipeline swipe + bottom dots, amount round-trips, keyboard-lifted picker, the email bell) + the v103 date requirement all look good on his phone. -- **Next:** (A) one remaining spot-check on the bell — the **approve-on-phone → Matrix-thread-clears** round-trip (the UI works; confirm the bidirectional sync end-to-end with a real proposal). (B) Carried from v100: #7 real-card spot-checks + the standing mobile light/dark + PWA-install gate. -- **Open / risks:** `.pipeline-screen { height:100% }` leans on the `.content` flex chain for a definite height — confirm the swipe area fills + scrolls on Grant's iOS (resolves on iOS 16+; no speculative patch applied). Bell + amount-edit are admin/live-smoke only. Carried: **Claude/Architect path unverified live on the box**; vision OCR can misread a small-in-frame card (`mara.com→marac.com`, temp 0); phone/LinkedIn land on the contact record, not the grid pill; PWA iOS status bar fixed `black` in light theme; doc drift — `crm-overview.md`/`EVALUATION.md` still call `lp_profiles` live. +- **Removed (v0.1.0:104):** the **Instructions** + **Feedback** (`feature_requests`) pages + backend, and `lp_profiles` + `investor_type` (across server / ingest / seeds). Migration `0008` drops both empty tables (a sanctioned one-off exception to never-hard-delete); `0001`'s `lp_profiles` ALTER was removed so a fresh DB doesn't break the migration chain. Net −570 lines. +- **Fixes (v0.1.0:104):** [B] email sync no longer terminally parks a mailbox on a transient timeout — `'retrying'` retries every cycle, `'error'` re-included on an hourly backoff, so **Grant's & Jonathan's stuck mailboxes self-heal on this deploy** (`test_sync_ready.py`). [C] clock icon on the mobile email Review-log sets a reminder inline. [D] email-approval cards show date/time. **[Contacts 500-cap]** the mobile Contacts directory now pages through ALL contacts (was truncated at 500 of 720 — hid people from the list *and* search). +- **New (v0.1.0:104):** admin-only **Purge Deleted Data** (Settings → Admin) — guarded, type-to-confirm hard-delete of soft-deleted rows; see the soft-delete convention + `test_purge_soft_deleted.py`. +- **Verification:** **45/45** backend, render-smoke green, reviewer-agent APPROVE after fixing **1 blocker** (contact purge left a dangling `reminders.contact_id` — now NULLed + test-guarded). New UI behavior is **live-smoke / on-device only** (jsdom can't drive touch). +- **Bug A — Grant is handling:** `odell/marty/finance/ten31@` can't enroll for email capture ("could not resolve user_id") because the enroll flow requires a CRM `users` row; Grant is creating user accounts for those mailboxes. +- **Next:** (A) confirm the two stuck mailboxes pulled current + Grant's 4 new mailbox users enroll; (B) **retire `contact_type`** — replace the Contacts Investors/Prospects tabs + TYPE badge with grid-derived `existing_investor`/`pipeline_stage`, then drop the column (see ROADMAP); (C) **contacts ↔ `fundraising_contacts` consolidation** — census-first (count A/linked, B/contacts-only, C/pill-only on the box; see ROADMAP); (D) carried: bell approve-on-phone → Matrix-thread-clears round-trip spot-check. +- **Open / risks:** the Contacts pagination, the purge, and the email-sync auto-recovery are **live-smoke / not yet device-confirmed**. Carried: **Claude/Architect path unverified live on the box**; vision OCR small-in-frame misread (`mara.com→marac.com`); doc drift — `crm-overview.md` narrative + `EVALUATION.md` still describe `lp_profiles` (the active API/schema claims were fixed; the deeper Phase-0 narrative is deferred to a doc pass). diff --git a/ROADMAP.md b/ROADMAP.md index 34300b9..49b0daf 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -12,9 +12,6 @@ - `GET /api/fundraising/backups` - `GET/PATCH /api/fundraising/backup-policy` - `GET /api/fundraising/relational-summary` - - `GET /api/feature-requests` - - `POST /api/feature-requests` - - `PATCH /api/feature-requests/:id` - New DB tables: - `fundraising_state` - `fundraising_investors` @@ -22,7 +19,6 @@ - `fundraising_funds` - `fundraising_commitments` - `fundraising_views` - - `feature_requests` - `app_settings` - Grid saves/restores now sync into relational fundraising tables automatically. - Formula engine is now sandboxed (no `eval`/`new Function`) with expanded function support. @@ -86,6 +82,12 @@ ## Backlog (post-Phase-1 agentic) +### Data-model cleanups (deferred from the v0.1.0:104 session) + +- **Retire `contacts.contact_type`** (the Contacts Investors/Prospects tabs + TYPE badge). It's a legacy binary that's set mechanically — `'investor'` just means "exists in the grid" (stamped unconditionally by `_upsert_contact_from_fundraising`), `'prospect'` means "imported/added, not in the grid" — and is superseded by the grid-derived signals `contact_grid_signals()` already injects (`existing_investor`/`committed`, `pipeline_stage`). Plan: replace the tabs + TYPE badge with those signals, repoint the dashboard `total_lps`/`total_prospects` counts, then drop the column. Live UI change → its own small design pass. (Grant: "I want to delete it, next session.") + +- **Consolidate `contacts` ↔ `fundraising_contacts` into one linked model.** Goal (Grant): everyone in `contacts` maps to a `fundraising_investors` row (an individual maps to their own row). Today `contacts` is the canonical person directory (FK target for `communications`/`opportunities`); `fundraising_contacts.contact_id` (migration `0004`) points INTO it; the mobile Contacts page reads `contacts`. Three populations: **A** linked (grid pill ↔ contact), **B** `contacts`-only (imported prospects / manual adds — need a grid row), **C** pill-only (`fundraising_contacts.contact_id IS NULL` — need a contact row). **Census-first:** before designing any migration, count A/B/C on the box — Grant runs the SQL himself (he is **not** providing a DB copy), so hand him a counts-only script. The census decides whether this is a ~20-row cleanup or a ~300-row structural migration with `communications`/`opportunities` repointing. Then Grant reconciles B (add grid rows/pills) and C (add contact rows) and ensures all are linked. + ### Follow-ups/reminders + NL search + bot grid-mutations (agreed plan, 2026-06-18) *Agreed with Grant 2026-06-18. Three workstreams, sequenced **W1 → W2 → W3**. **Overarching constraint (Grant):** the dominant risk is **leaking LP data (names, $, notes, contacts) to third-party LLMs — NOT write-safety.** A wrong number is recoverable; investor substance reaching Claude is not. Consequences: W2 keeps LP rows off Claude (only the question text + schema vocabulary leave the box; entity names resolved locally); W3 keeps bot mutation-parsing on local Qwen. Because this DB *logs* commitments/pipeline but doesn't move money, a bot mutation is low-stakes → **any team member may approve one in Matrix**; the guardrail is "the bot can't silently mass-change numbers," enforced by the per-mutation human approval gate, not a tight money gate.* diff --git a/backend/email_integration/db.py b/backend/email_integration/db.py index 1a89c99..6c8f9ee 100644 --- a/backend/email_integration/db.py +++ b/backend/email_integration/db.py @@ -57,10 +57,20 @@ def _json(v) -> str: # ------------------------------------------------------------------ email_accounts def list_sync_ready_accounts(conn: sqlite3.Connection) -> list[sqlite3.Row]: + # Ready = healthy ('pending'/'active') + transient-failing ('retrying', retried every + # cycle for fast recovery) + 'error' accounts whose last attempt was over an hour ago. + # The hour-backoff on 'error' means a terminal failure (auth/permanent) self-heals once + # the operator fixes it WITHOUT hammering Google, and un-sticks any mailbox parked by the + # pre-v0.1.0:104 bug where one timeout dark-listed it forever. (last_synced_at is stamped + # on every attempt, success or fail, so it doubles as the last-attempt clock here.) cur = conn.cursor() cur.execute( "SELECT * FROM email_accounts " - "WHERE sync_enabled = 1 AND sync_status IN ('pending','active') " + "WHERE sync_enabled = 1 AND (" + " sync_status IN ('pending','active','retrying') " + " OR (sync_status = 'error' AND (last_synced_at IS NULL " + " OR last_synced_at < datetime('now','-1 hour')))" + ") " "ORDER BY last_synced_at IS NOT NULL, last_synced_at" ) return cur.fetchall() diff --git a/backend/email_integration/sync.py b/backend/email_integration/sync.py index a1e8c82..0593366 100644 --- a/backend/email_integration/sync.py +++ b/backend/email_integration/sync.py @@ -23,6 +23,7 @@ import logging import sqlite3 import traceback from typing import Optional +from urllib.error import URLError from . import attachments as _attach from . import config as _cfg @@ -112,6 +113,15 @@ def sync_account(conn_factory, credential_provider, account, error_str = "history expired; fallback to date backfill" status = "partial" _fallback_date_backfill(conn_factory, client, account, index, run_stats) + except (_errors.RateLimitError, _errors.TransientError, URLError, TimeoutError) as e: + # A network / 5xx / rate-limit error that outlived the in-pass retry loop. + # This is TRANSIENT, not terminal: park it as 'retrying' (which the scheduler + # still picks up every cycle) instead of 'error' (which it excludes). Fixes the + # v<=0.1.0:103 bug where a single timeout dark-listed a mailbox until a manual + # kick. Terminal causes (auth, permanent, unexpected) still fall through to 'error'. + error_str = f"transient: {type(e).__name__}: {e}" + status = "retrying" + log.warning("transient error during sync of %s: %s", email_addr, e) except Exception as e: error_str = f"unexpected: {type(e).__name__}: {e}" status = "error" diff --git a/backend/email_integration/test_sync_ready.py b/backend/email_integration/test_sync_ready.py new file mode 100644 index 0000000..97cc548 --- /dev/null +++ b/backend/email_integration/test_sync_ready.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +"""Regression test for list_sync_ready_accounts (v0.1.0:104). + +Guards the Bug-B fix: a transient network timeout used to flip a mailbox to terminal +sync_status='error', which the old `IN ('pending','active')` filter excluded forever — +so a single blip dark-listed a mailbox until a manual kick. The new filter: + * always includes 'pending' / 'active' / 'retrying' (the transient-retry state), and + * re-includes 'error' accounts whose last attempt was over an hour ago (gentle backoff, + so a fixed credential self-heals and the pre-fix stuck mailboxes recover on deploy). +Synthetic data only (guardrail #9). +Run: cd backend && python3 email_integration/test_sync_ready.py +""" +import os +import sqlite3 +import sys + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from email_integration import db as edb # noqa: E402 + +FAILS = [] + + +def check(cond, msg): + print((" PASS " if cond else " FAIL ") + msg) + if not cond: + FAILS.append(msg) + + +def make_account(conn, email, status, *, enabled=1, last_synced_sql="NULL"): + aid = edb.upsert_account(conn, user_id="u-" + email, + email_address=email, auth_method="dwd") + conn.execute( + f"UPDATE email_accounts SET sync_status=?, sync_enabled=?, " + f"last_synced_at={last_synced_sql} WHERE id=?", + (status, enabled, aid), + ) + conn.commit() + return aid + + +def main(): + conn = sqlite3.connect(":memory:") + conn.row_factory = sqlite3.Row + edb.apply_migrations(conn.cursor()) + + make_account(conn, "active@t.xyz", "active", last_synced_sql="datetime('now','-5 minutes')") + make_account(conn, "retrying@t.xyz", "retrying", last_synced_sql="datetime('now','-5 minutes')") + make_account(conn, "pending@t.xyz", "pending", last_synced_sql="NULL") + make_account(conn, "error_stale@t.xyz", "error", last_synced_sql="datetime('now','-2 hours')") + make_account(conn, "error_recent@t.xyz", "error", last_synced_sql="datetime('now','-5 minutes')") + make_account(conn, "error_neversync@t.xyz", "error", last_synced_sql="NULL") + make_account(conn, "disabled@t.xyz", "active", enabled=0, last_synced_sql="datetime('now','-5 minutes')") + + ready = {r["email_address"] for r in edb.list_sync_ready_accounts(conn)} + + check("active@t.xyz" in ready, "healthy 'active' is ready") + check("retrying@t.xyz" in ready, "transient 'retrying' is ready (fast retry)") + check("pending@t.xyz" in ready, "'pending' is ready") + check("error_stale@t.xyz" in ready, "'error' last attempted >1h ago is ready (backoff elapsed → recovers stuck mailbox)") + check("error_neversync@t.xyz" in ready, "'error' never synced (NULL last attempt) is ready") + check("error_recent@t.xyz" not in ready, "'error' attempted <1h ago is NOT ready (gentle backoff, no hammering)") + check("disabled@t.xyz" not in ready, "sync_enabled=0 is never ready") + + print() + if FAILS: + print(f"{len(FAILS)} FAILED") + sys.exit(1) + print("ALL PASS (sync ready filter)") + + +if __name__ == "__main__": + main() diff --git a/backend/ingest/chunking.py b/backend/ingest/chunking.py index 341c199..eb9317c 100644 --- a/backend/ingest/chunking.py +++ b/backend/ingest/chunking.py @@ -4,7 +4,7 @@ Maps each CRM record type to one or more chunks per docs/EMBEDDINGS.md: * one chunk per communications row (doc_type = the comm type) * one chunk per MATCHED email (doc_type = email; body only when matched) * one chunk per fundraising_investors notes LINE (the outreach log; split per line) - * one chunk each for free-text fields: contacts.notes, lp_profiles.notes, + * one chunk each for free-text fields: contacts.notes, opportunities (description + next_step), organizations.description Each chunk carries a canonical `lp_id` (resolved via entity_links) and a `date_ts` @@ -104,13 +104,6 @@ def build_chunks(conn): chunks.append(_mk(f"contacts.notes:{r['id']}", lp, lp_name, person, "contact_note", to_epoch(r["updated_at"]), r["notes"], "contacts", r["id"])) - # lp_profiles.notes - for r in conn.execute("""SELECT lp.id, lp.contact_id, lp.notes, lp.updated_at - FROM lp_profiles lp WHERE lp.notes IS NOT NULL AND lp.notes <> '' AND lp.deleted_at IS NULL"""): - lp, lp_name, person = _contact_lp(r["contact_id"], person_canon, org_canon, name, contact_org) - chunks.append(_mk(f"lp_profiles.notes:{r['id']}", lp, lp_name, person, - "lp_note", to_epoch(r["updated_at"]), r["notes"], "lp_profiles", r["id"])) - # opportunities (description + next_step) for r in conn.execute("""SELECT id, contact_id, name, description, next_step, updated_at FROM opportunities WHERE deleted_at IS NULL"""): diff --git a/backend/ingest/entity_resolution.py b/backend/ingest/entity_resolution.py index 1162c3b..cdeccc9 100644 --- a/backend/ingest/entity_resolution.py +++ b/backend/ingest/entity_resolution.py @@ -8,7 +8,6 @@ layer created by migration 0001: fundraising_investors ─┴─► canonical_entities (entity_kind = lp | organization) contacts ─┐ fundraising_contacts ─┴─► canonical_entities (entity_kind = person) - lp_profiles ───► linked to its contact's person entity Every source row is recorded in `entity_links` so any name variant resolves to one canonical id. This is the DETERMINISTIC tier — it merges only what we can @@ -184,7 +183,7 @@ def resolve_people(conn, org_canon_by_orgid, org_canon_by_fundinv, merge_map=Non people — each is matched to a contact-person and recorded only as a member_of edge to its investor entity (the grid's 'Contacts' column says who belongs to which investor). This is what stops the double-count. - Returns contact_id -> person canonical id (for lp_profiles).""" + Returns contact_id -> person canonical id.""" merge_map = merge_map or {} contact_to_person = {} person_meta = {} @@ -245,12 +244,6 @@ def resolve_people(conn, org_canon_by_orgid, org_canon_by_fundinv, merge_map=Non _link(conn, cid, "fundraising_contacts", r["id"], email or name_norm, mk, 0.95 if mk == "grid_link" else 0.9) _member_of(conn, cid, inv_canon) - # lp_profiles -> the person entity of its contact - for r in conn.execute("SELECT id, contact_id FROM lp_profiles WHERE deleted_at IS NULL"): - cid = contact_to_person.get(r["contact_id"]) - if cid: - _link(conn, cid, "lp_profiles", r["id"], r["contact_id"], "contact_fk", 1.0) - return person_meta diff --git a/backend/ingest/sync.py b/backend/ingest/sync.py index e31c976..1b7a373 100644 --- a/backend/ingest/sync.py +++ b/backend/ingest/sync.py @@ -34,7 +34,7 @@ import entity_resolution as er import qdrant_io _CHANGE_TABLES = [("communications", "communications"), ("contacts", "contacts"), - ("lp_profiles", "lp_profiles"), ("opportunities", "opportunities"), + ("opportunities", "opportunities"), ("organizations", "organizations"), ("fundraising_investors", "fundraising_investors")] @@ -63,7 +63,7 @@ def _state_set(conn, key, value): def _deleted_source_ids(conn, since): """CRM records soft-deleted since the watermark — their chunks get pruned.""" ids = set() - for tbl in ("contacts", "organizations", "opportunities", "communications", "lp_profiles"): + for tbl in ("contacts", "organizations", "opportunities", "communications"): try: for r in conn.execute(f"SELECT id FROM {tbl} WHERE deleted_at IS NOT NULL AND deleted_at > ?", (since,)): ids.add(r["id"]) diff --git a/backend/ingest/test_entity_resolution.py b/backend/ingest/test_entity_resolution.py index 6707488..f81307a 100644 --- a/backend/ingest/test_entity_resolution.py +++ b/backend/ingest/test_entity_resolution.py @@ -12,7 +12,7 @@ Asserts the SAFE fix: 3. a grid contact that can't be PROVABLY matched mints NOTHING (no duplicate person, no cross-firm name guess) — the count stays correct, 4. targeted cleanup soft-deletes a stale grid-only "twin" (person with no - contacts link) and a superseded 'lp'/'organization' row, with no enrichment, + contacts link), with no enrichment, 5. cleanup PRESERVES a grid-only person that carries enrichment (guardrail #3), 6. a re-emitted id is UN-tombstoned (no permanent burial), 7. re-running is idempotent. @@ -58,10 +58,9 @@ CREATE TABLE contacts ( CREATE TABLE organizations (id TEXT PRIMARY KEY, name TEXT, email TEXT); CREATE TABLE fundraising_investors (id TEXT PRIMARY KEY, investor_name TEXT); CREATE TABLE fundraising_contacts (id TEXT PRIMARY KEY, full_name TEXT, email TEXT, investor_id TEXT, contact_id TEXT); -CREATE TABLE lp_profiles (id TEXT PRIMARY KEY, contact_id TEXT, deleted_at TEXT); """ -SEEDED = ("per_TWIN", "per_ENR", "lp_OLD") +SEEDED = ("per_TWIN", "per_ENR") def seed(db): @@ -94,16 +93,14 @@ def seed(db): "('per_ENR','person','Enriched Orphan','entity_resolution','warm')") c.execute("INSERT INTO entity_links (id, canonical_id, source_model, source_id, match_value, match_kind, confidence, created_at) " "VALUES ('l_enr','per_ENR','fundraising_contacts','gy','enr','name_org',0.8,'t')") - # Superseded pre-:48 kind -> prune - c.execute("INSERT INTO canonical_entities (id, entity_kind, display_name, source) VALUES " - "('lp_OLD','lp','Old LP Row','entity_resolution')") c.commit() c.close() def resolved_persons(db): c = sqlite3.connect(db) - q = "SELECT COUNT(*) FROM canonical_entities WHERE entity_kind='person' AND deleted_at IS NULL AND id NOT IN (?,?,?)" + ph = ",".join("?" * len(SEEDED)) + q = f"SELECT COUNT(*) FROM canonical_entities WHERE entity_kind='person' AND deleted_at IS NULL AND id NOT IN ({ph})" n = c.execute(q, SEEDED).fetchone()[0] c.close() return n @@ -127,10 +124,11 @@ def grid_match_kinds(db): def minted_from_grid(db): """Persons minted directly from a grid row (the bug). Should be 0 after the fix.""" c = sqlite3.connect(db) - n = c.execute("""SELECT COUNT(DISTINCT l.canonical_id) FROM entity_links l + ph = ",".join("?" * len(SEEDED)) + n = c.execute(f"""SELECT COUNT(DISTINCT l.canonical_id) FROM entity_links l JOIN canonical_entities ce ON ce.id=l.canonical_id AND ce.deleted_at IS NULL WHERE l.source_model='fundraising_contacts' AND l.match_kind IN ('name_org','exact_email') - AND l.canonical_id NOT IN (?,?,?)""", SEEDED).fetchone()[0] + AND l.canonical_id NOT IN ({ph})""", SEEDED).fetchone()[0] c.close() return n @@ -162,12 +160,11 @@ def main(): check(mk.get("grid_assoc", 0) == 2, f"two grid contacts matched back via grid_assoc (got {mk.get('grid_assoc',0)})") check(mk.get("grid_link", 0) == 1, f"one grid contact linked via explicit contact_id (grid_link==1, got {mk.get('grid_link',0)})") - # Targeted cleanup: stale grid-only twin + superseded 'lp' row tombstoned... + # Targeted cleanup: stale grid-only twin tombstoned... check(deleted_at(db, "per_TWIN") is not None, "stale grid-only twin 'per_TWIN' tombstoned") - check(deleted_at(db, "lp_OLD") is not None, "superseded 'lp' row 'lp_OLD' tombstoned") # ...enriched grid-only person preserved. check(deleted_at(db, "per_ENR") is None, "enriched grid-only person 'per_ENR' PRESERVED (has segment)") - check(counts1.get("pruned_stale", 0) == 2, f"exactly 2 stale rows pruned (got {counts1.get('pruned_stale')})") + check(counts1.get("pruned_stale", 0) == 1, f"exactly 1 stale row pruned (got {counts1.get('pruned_stale')})") # Un-tombstone: soft-delete a real contact-person, then re-run -> it comes back. alice = er._eid("per", "e|alice@x.com") diff --git a/backend/migrations/0001_phase0_foundation.sql b/backend/migrations/0001_phase0_foundation.sql index ef247b0..8792a4e 100644 --- a/backend/migrations/0001_phase0_foundation.sql +++ b/backend/migrations/0001_phase0_foundation.sql @@ -113,4 +113,8 @@ ALTER TABLE contacts ADD COLUMN deleted_at TEXT; ALTER TABLE organizations ADD COLUMN deleted_at TEXT; ALTER TABLE opportunities ADD COLUMN deleted_at TEXT; ALTER TABLE communications ADD COLUMN deleted_at TEXT; -ALTER TABLE lp_profiles ADD COLUMN deleted_at TEXT; +-- lp_profiles ALTER removed (v0.1.0:104): the lp_profiles table is dropped in +-- 0008_drop_retired_tables.sql and is no longer created by init_db(), so this +-- ALTER would fail "no such table" on a fresh install. Live DBs already applied +-- this migration (with the original ALTER) before lp_profiles was dropped, so +-- removing the line here only affects fresh DBs — same end state either way. diff --git a/backend/migrations/0008_drop_retired_tables.down.sql b/backend/migrations/0008_drop_retired_tables.down.sql new file mode 100644 index 0000000..7e824c2 --- /dev/null +++ b/backend/migrations/0008_drop_retired_tables.down.sql @@ -0,0 +1,42 @@ +-- 0008_drop_retired_tables.down.sql (manual rollback only — never auto-applied) +-- +-- Recreates the two dropped tables as EMPTY shells, matching the schema that existed +-- immediately before 0008 (lp_profiles includes the deleted_at column that migration +-- 0001 had added). Data is not restored — both tables were empty when dropped. +CREATE TABLE IF NOT EXISTS lp_profiles ( + id TEXT PRIMARY KEY, + contact_id TEXT NOT NULL UNIQUE REFERENCES contacts(id) ON DELETE CASCADE, + commitment_amount REAL DEFAULT 0, + funded_amount REAL DEFAULT 0, + commitment_date TEXT, + fund_name TEXT, + investor_type TEXT, + accredited INTEGER DEFAULT 0, + legal_docs_signed INTEGER DEFAULT 0, + signed_date TEXT, + wire_received INTEGER DEFAULT 0, + wire_date TEXT, + k1_sent INTEGER DEFAULT 0, + preferred_communication TEXT DEFAULT 'email', + notes TEXT, + created_at TEXT DEFAULT (datetime('now')), + updated_at TEXT DEFAULT (datetime('now')), + deleted_at TEXT +); +CREATE INDEX IF NOT EXISTS idx_lp_profiles_contact ON lp_profiles(contact_id); + +CREATE TABLE IF NOT EXISTS feature_requests ( + id TEXT PRIMARY KEY, + title TEXT NOT NULL, + description TEXT, + page TEXT, + category TEXT DEFAULT 'general', + priority TEXT DEFAULT 'medium', + status TEXT DEFAULT 'new', + requested_by TEXT, + requested_by_user_id TEXT REFERENCES users(id), + created_at TEXT DEFAULT (datetime('now')), + updated_at TEXT DEFAULT (datetime('now')) +); +CREATE INDEX IF NOT EXISTS idx_feature_requests_status ON feature_requests(status); +CREATE INDEX IF NOT EXISTS idx_feature_requests_created_at ON feature_requests(created_at); diff --git a/backend/migrations/0008_drop_retired_tables.sql b/backend/migrations/0008_drop_retired_tables.sql new file mode 100644 index 0000000..00c181d --- /dev/null +++ b/backend/migrations/0008_drop_retired_tables.sql @@ -0,0 +1,15 @@ +-- 0008_drop_retired_tables.sql (v0.1.0:104) +-- +-- ONE-OFF DESTRUCTIVE EXCEPTION to the never-hard-delete rule, explicitly approved. +-- Both tables are EMPTY and fully removed from the application code: +-- * lp_profiles — the legacy single-fund LP model, retired v0.1.0:78; the +-- fundraising_* grid is the canonical commitment record now. +-- * feature_requests — backed the in-app Feedback page, which was removed. +-- +-- The never-hard-delete policy STILL STANDS for all real CRM and thesis data — this +-- is a deliberate, documented exception for two empty, retired tables so they don't +-- linger as dead schema. init_db() no longer creates either table, and migration +-- 0001's lp_profiles ALTER was removed, so a fresh DB never creates them and this +-- DROP is a harmless no-op there; on the live box it removes the existing empties. +DROP TABLE IF EXISTS lp_profiles; +DROP TABLE IF EXISTS feature_requests; diff --git a/backend/scripts/seed_synthetic.py b/backend/scripts/seed_synthetic.py index 45f4262..7ea9829 100644 --- a/backend/scripts/seed_synthetic.py +++ b/backend/scripts/seed_synthetic.py @@ -11,8 +11,8 @@ What it builds (into a SEPARATE dev DB, never crm.db): core migration (backend/migrations/), so the canonical/interaction/graph tables exist. * A classic-model dataset: organizations, contacts (investors + prospects), - opportunities across pipeline stages, communications with entity-rich prose - notes, and lp_profiles. + opportunities across pipeline stages, and communications with entity-rich + prose notes. * A fundraising grid (fundraising_state.grid_json) populated via the real sync_fundraising_relational() code path, so the normalized mirror + the grid->classic bridge behave exactly as in production. @@ -179,7 +179,7 @@ def main(): f"Prospect sourced via {random.choice(['X DM', 'warm intro', 'podcast'])}.", uid, now())) contacts.append((cid, first, last, org_name, "prospect")) - # ── opportunities + lp_profiles + communications ── + # ── opportunities + communications ── stages = server.PIPELINE_STAGES for idx, (cid, first, last, org_name, ctype) in enumerate(contacts): person = f"{first} {last}" @@ -199,19 +199,6 @@ def main(): random.choice(["Send deck", "Schedule call", "Await IC", "Send subdocs"]), uid, random.choice(["low", "medium", "high"]), now())) - # lp_profile for ~closed investors - if ctype == "investor" and idx % 2 == 0: - amt = random.choice(AMOUNTS) - conn.execute( - "INSERT INTO lp_profiles (id, contact_id, commitment_amount, funded_amount, commitment_date, " - "fund_name, investor_type, accredited, legal_docs_signed, wire_received, k1_sent, notes, updated_at) " - "VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)", - (gen(), cid, amt, amt if idx % 4 == 0 else 0, past(120), - random.choice(list(FUND_LABELS.values())), - random.choice(["family_office", "institutional", "endowment", "individual"]), - 1, 1 if idx % 3 else 0, 1 if idx % 4 == 0 else 0, 0, - f"Closed LP. Accreditation on file. Primary contact {person}.", now())) - # 2-4 communications each, entity-rich prose for k in range(random.randint(2, 4)): ctype_comm, subj, body = random.choice(COMM_TEMPLATES) @@ -275,7 +262,7 @@ def main(): print(f"\nSynthetic dev DB written to: {db}") print(" Classic model:") - for t in ("organizations", "contacts", "opportunities", "communications", "lp_profiles"): + for t in ("organizations", "contacts", "opportunities", "communications"): print(f" {t:<24} {count(t)}") print(" Fundraising grid (after real sync):") for t in ("fundraising_investors", "fundraising_contacts", "fundraising_funds", diff --git a/backend/server.py b/backend/server.py index 2ac3cf8..b80e120 100644 --- a/backend/server.py +++ b/backend/server.py @@ -215,26 +215,6 @@ def init_db(): updated_at TEXT DEFAULT (datetime('now')) ); - CREATE TABLE IF NOT EXISTS lp_profiles ( - id TEXT PRIMARY KEY, - contact_id TEXT NOT NULL UNIQUE REFERENCES contacts(id) ON DELETE CASCADE, - commitment_amount REAL DEFAULT 0, - funded_amount REAL DEFAULT 0, - commitment_date TEXT, - fund_name TEXT, - investor_type TEXT, - accredited INTEGER DEFAULT 0, - legal_docs_signed INTEGER DEFAULT 0, - signed_date TEXT, - wire_received INTEGER DEFAULT 0, - wire_date TEXT, - k1_sent INTEGER DEFAULT 0, - preferred_communication TEXT DEFAULT 'email', - notes TEXT, - created_at TEXT DEFAULT (datetime('now')), - updated_at TEXT DEFAULT (datetime('now')) - ); - CREATE TABLE IF NOT EXISTS custom_fields ( id TEXT PRIMARY KEY, name TEXT NOT NULL, @@ -273,20 +253,6 @@ def init_db(): created_at TEXT DEFAULT (datetime('now')) ); - CREATE TABLE IF NOT EXISTS feature_requests ( - id TEXT PRIMARY KEY, - title TEXT NOT NULL, - description TEXT, - page TEXT, - category TEXT DEFAULT 'general', - priority TEXT DEFAULT 'medium', - status TEXT DEFAULT 'new', - requested_by TEXT, - requested_by_user_id TEXT REFERENCES users(id), - created_at TEXT DEFAULT (datetime('now')), - updated_at TEXT DEFAULT (datetime('now')) - ); - CREATE TABLE IF NOT EXISTS fundraising_state ( id TEXT PRIMARY KEY, grid_json TEXT NOT NULL, @@ -422,9 +388,6 @@ def init_db(): CREATE INDEX IF NOT EXISTS idx_communications_contact ON communications(contact_id); CREATE INDEX IF NOT EXISTS idx_communications_date ON communications(communication_date); CREATE INDEX IF NOT EXISTS idx_audit_entity ON audit_log(entity_type, entity_id); - CREATE INDEX IF NOT EXISTS idx_lp_profiles_contact ON lp_profiles(contact_id); - CREATE INDEX IF NOT EXISTS idx_feature_requests_status ON feature_requests(status); - CREATE INDEX IF NOT EXISTS idx_feature_requests_created_at ON feature_requests(created_at); CREATE INDEX IF NOT EXISTS idx_fr_investor_name ON fundraising_investors(investor_name); CREATE INDEX IF NOT EXISTS idx_fr_investor_lead ON fundraising_investors(lead); CREATE INDEX IF NOT EXISTS idx_fr_contacts_investor ON fundraising_contacts(investor_id); @@ -2381,10 +2344,6 @@ class CRMHandler(BaseHTTPRequestHandler): if path == '/api/export/contacts': return self.handle_export_contacts(user, params) - # Feature requests - if path == '/api/feature-requests': - return self.handle_list_feature_requests(user, params) - # Fundraising grid state if path == '/api/fundraising/state': return self.handle_get_fundraising_state(user) @@ -2398,6 +2357,8 @@ class CRMHandler(BaseHTTPRequestHandler): return self.handle_get_backup_policy(user) if path == '/api/admin/digest/policy': return self.handle_get_digest_policy(user) + if path == '/api/admin/soft-deleted': + return self.handle_list_soft_deleted(user) if path == '/api/fundraising/relational-summary': return self.handle_get_fundraising_relational_summary(user) if path == '/api/fundraising/automations': @@ -2503,8 +2464,6 @@ class CRMHandler(BaseHTTPRequestHandler): return self.handle_create_communication(user, body) if path == '/api/import/csv': return self.handle_import_csv(user, body) - if path == '/api/feature-requests': - return self.handle_create_feature_request(user, body) if path == '/api/fundraising/log-communication': return self.handle_log_fundraising_communication(user, body) if path == '/api/fundraising/update-row': @@ -2525,6 +2484,8 @@ class CRMHandler(BaseHTTPRequestHandler): return self.handle_admin_create_user(user, body) if path == '/api/admin/reset-all-data': return self.handle_admin_reset_all_data(user, body) + if path == '/api/admin/soft-deleted/purge': + return self.handle_purge_soft_deleted(user, body) if path == '/api/admin/digest/test-email': return self.handle_admin_send_test_email(user, body) if path == '/api/admin/digest/send-now': @@ -2623,9 +2584,6 @@ class CRMHandler(BaseHTTPRequestHandler): if re.match(r'^/api/opportunities/[^/]+/stage$', path): opp_id = path.split('/')[-2] return self.handle_update_stage(user, opp_id, body) - if re.match(r'^/api/feature-requests/[^/]+$', path): - fr_id = path.split('/')[-1] - return self.handle_update_feature_request(user, fr_id, body) if re.match(r'^/api/reminders/[^/]+$', path): return self.handle_update_reminder(user, path.split('/')[-1], body) if re.match(r'^/api/admin/users/[^/]+$', path): @@ -2963,12 +2921,11 @@ class CRMHandler(BaseHTTPRequestHandler): _sync_contact_to_fundraising_state(conn, row_to_dict(existing), actor_user_id=user['user_id'], remove=True) # Soft-delete (guardrail #3 — never hard-delete): mark deleted_at and - # cascade to the contact's opportunities, communications, and lp_profile. + # cascade to the contact's opportunities and communications. _ts = now() conn.execute("UPDATE contacts SET deleted_at = ?, updated_at = ? WHERE id = ?", (_ts, _ts, contact_id)) conn.execute("UPDATE opportunities SET deleted_at = ? WHERE contact_id = ? AND deleted_at IS NULL", (_ts, contact_id)) conn.execute("UPDATE communications SET deleted_at = ? WHERE contact_id = ? AND deleted_at IS NULL", (_ts, contact_id)) - conn.execute("UPDATE lp_profiles SET deleted_at = ? WHERE contact_id = ? AND deleted_at IS NULL", (_ts, contact_id)) log_audit(conn, user['user_id'], 'contact', contact_id, 'delete') conn.commit() conn.close() @@ -4821,6 +4778,96 @@ class CRMHandler(BaseHTTPRequestHandler): finally: conn.close() + # ─── Soft-deleted purge (admin maintenance) ────────────────────────────────── + # Lists soft-deleted rows and HARD-deletes them — a deliberate, admin-only, type-to-confirm + # exception to the never-hard-delete rule, for clearing out dummy/test data. A purge can only + # ever touch a soft-deleted row, and refuses any contact/org whose delete would CASCADE or + # SET NULL onto LIVE data, so it can never remove or mutate a live record. + _PURGE_TABLES = ('contacts', 'organizations', 'opportunities', 'communications') + + def handle_list_soft_deleted(self, user): + if not require_admin(user): + return self.send_error_json("Admin required", 403) + conn = get_db() + try: + groups = {} + groups['contacts'] = [ + {"id": r["id"], + "label": (f"{r['first_name'] or ''} {r['last_name'] or ''}".strip() or r["email"] or r["id"]), + "deleted_at": r["deleted_at"]} + for r in conn.execute( + "SELECT id, first_name, last_name, email, deleted_at FROM contacts " + "WHERE deleted_at IS NOT NULL ORDER BY deleted_at DESC").fetchall()] + groups['organizations'] = [ + {"id": r["id"], "label": (r["name"] or r["id"]), "deleted_at": r["deleted_at"]} + for r in conn.execute( + "SELECT id, name, deleted_at FROM organizations " + "WHERE deleted_at IS NOT NULL ORDER BY deleted_at DESC").fetchall()] + groups['opportunities'] = [ + {"id": r["id"], "label": (r["name"] or r["id"]), "deleted_at": r["deleted_at"]} + for r in conn.execute( + "SELECT id, name, deleted_at FROM opportunities " + "WHERE deleted_at IS NOT NULL ORDER BY deleted_at DESC").fetchall()] + groups['communications'] = [ + {"id": r["id"], + "label": ((r["subject"] or (r["type"] or "note")) + + (f" · {(r['communication_date'] or '')[:10]}" if r["communication_date"] else "")), + "deleted_at": r["deleted_at"]} + for r in conn.execute( + "SELECT id, type, subject, communication_date, deleted_at FROM communications " + "WHERE deleted_at IS NOT NULL ORDER BY deleted_at DESC").fetchall()] + return self.send_json({"groups": groups, "total": sum(len(v) for v in groups.values())}) + finally: + conn.close() + + def handle_purge_soft_deleted(self, user, body): + if not require_admin(user): + return self.send_error_json("Admin required", 403) + body = body or {} + table = str(body.get('table') or '').strip() + row_id = str(body.get('id') or '').strip() + if table not in self._PURGE_TABLES: # validated -> safe to interpolate below + return self.send_error_json("Unknown table", 400) + if not row_id: + return self.send_error_json("id is required", 400) + conn = get_db() + try: + row = conn.execute(f"SELECT id, deleted_at FROM {table} WHERE id = ?", (row_id,)).fetchone() + if not row: + return self.send_error_json("Not found", 404) + if not row["deleted_at"]: + return self.send_error_json("Only soft-deleted rows can be purged", 400) + # A contacts/organizations delete cascades / SET NULLs onto children. Refuse if any LIVE + # child exists so a purge can never touch live data (a soft-deleted parent's children were + # soft-deleted with it, so normally there are none). + if table == 'contacts': + live = conn.execute( + "SELECT (SELECT COUNT(*) FROM opportunities WHERE contact_id=? AND deleted_at IS NULL) + " + "(SELECT COUNT(*) FROM communications WHERE contact_id=? AND deleted_at IS NULL) AS n", + (row_id, row_id)).fetchone()["n"] + if live: + return self.send_error_json("This contact still has live communications or opportunities — cannot purge", 409) + # Drop the optional logical FKs that have no ON DELETE (so a purged contact leaves no + # dangling reference): the derived grid link (fundraising_contacts.contact_id, migration + # 0004) and any reminder's contact_id (migration 0006 — the reminder's real link is + # investor_id, which is unaffected). Both are bare TEXT columns, not declared FKs. + conn.execute("UPDATE fundraising_contacts SET contact_id = NULL WHERE contact_id = ?", (row_id,)) + conn.execute("UPDATE reminders SET contact_id = NULL WHERE contact_id = ?", (row_id,)) + elif table == 'organizations': + live = conn.execute( + "SELECT (SELECT COUNT(*) FROM contacts WHERE organization_id=? AND deleted_at IS NULL) + " + "(SELECT COUNT(*) FROM opportunities WHERE organization_id=? AND deleted_at IS NULL) AS n", + (row_id, row_id)).fetchone()["n"] + if live: + return self.send_error_json("This organization is still linked to live contacts or opportunities — cannot purge", 409) + # CASCADE removes the (soft-deleted) children for contacts; the rest are leaves. + conn.execute(f"DELETE FROM {table} WHERE id = ?", (row_id,)) + log_audit(conn, user['user_id'], table, row_id, 'purge', {"table": table}) + conn.commit() + return self.send_json({"data": {"purged": True, "table": table, "id": row_id}}) + finally: + conn.close() + def handle_decide_activity_proposal(self, user, proposal_id, decision, body): if not require_admin(user): return self.send_error_json("Admin required", 403) @@ -5393,10 +5440,8 @@ class CRMHandler(BaseHTTPRequestHandler): conn.execute("DELETE FROM communications") conn.execute("DELETE FROM opportunities") - conn.execute("DELETE FROM lp_profiles") conn.execute("DELETE FROM custom_field_values") conn.execute("DELETE FROM custom_fields") - conn.execute("DELETE FROM feature_requests") conn.execute("DELETE FROM contacts") conn.execute("DELETE FROM organizations") @@ -5882,93 +5927,6 @@ class CRMHandler(BaseHTTPRequestHandler): } }) - # ═══════════════════════════════════════════════════════════════════════════ - # FEATURE REQUESTS - # ═══════════════════════════════════════════════════════════════════════════ - - def handle_list_feature_requests(self, user, params): - conn = get_db() - query = """ - SELECT fr.*, u.full_name as requested_by_name - FROM feature_requests fr - LEFT JOIN users u ON fr.requested_by_user_id = u.id - WHERE 1=1 - """ - args = [] - - if params.get('status'): - query += " AND fr.status = ?" - args.append(params['status']) - - if params.get('search'): - search = f"%{params['search']}%" - query += " AND (fr.title LIKE ? OR fr.description LIKE ? OR fr.requested_by LIKE ?)" - args.extend([search, search, search]) - - query += " ORDER BY fr.created_at DESC" - rows = rows_to_list(conn.execute(query, args).fetchall()) - conn.close() - return self.send_json({"data": rows, "total": len(rows)}) - - def handle_create_feature_request(self, user, body): - title = str(body.get('title', '')).strip() - if not title: - return self.send_error_json("title is required") - - req_id = generate_id() - requested_by = str(body.get('requested_by') or user.get('username') or '').strip() - conn = get_db() - conn.execute(""" - INSERT INTO feature_requests ( - id, title, description, page, category, priority, status, - requested_by, requested_by_user_id - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) - """, ( - req_id, - title, - body.get('description'), - body.get('page'), - body.get('category', 'general'), - body.get('priority', 'medium'), - body.get('status', 'new'), - requested_by, - user['user_id'] - )) - log_audit(conn, user['user_id'], 'feature_request', req_id, 'create', {"title": title}) - conn.commit() - row = row_to_dict(conn.execute("SELECT * FROM feature_requests WHERE id = ?", (req_id,)).fetchone()) - conn.close() - return self.send_json({"data": row}, 201) - - def handle_update_feature_request(self, user, req_id, body): - conn = get_db() - existing = conn.execute("SELECT * FROM feature_requests WHERE id = ?", (req_id,)).fetchone() - if not existing: - conn.close() - return self.send_error_json("Feature request not found", 404) - - updatable = ['title', 'description', 'page', 'category', 'priority', 'status', 'requested_by'] - sets = [] - args = [] - for field in updatable: - if field in body: - sets.append(f"{field} = ?") - args.append(body[field]) - - if not sets: - conn.close() - return self.send_error_json("No fields to update") - - sets.append("updated_at = ?") - args.append(now()) - args.append(req_id) - conn.execute(f"UPDATE feature_requests SET {', '.join(sets)} WHERE id = ?", args) - log_audit(conn, user['user_id'], 'feature_request', req_id, 'update', body) - conn.commit() - row = row_to_dict(conn.execute("SELECT * FROM feature_requests WHERE id = ?", (req_id,)).fetchone()) - conn.close() - return self.send_json({"data": row}) - # ═══════════════════════════════════════════════════════════════════════════ # FUNDRAISING STATE (AIRTABLE-LIKE GRID) # ═══════════════════════════════════════════════════════════════════════════ diff --git a/backend/test_purge_soft_deleted.py b/backend/test_purge_soft_deleted.py new file mode 100644 index 0000000..2dd0de1 --- /dev/null +++ b/backend/test_purge_soft_deleted.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +"""Test for the admin soft-deleted purge (v0.1.0:104). + +The purge is a deliberate, admin-only, type-to-confirm exception to never-hard-delete, for +clearing dummy/test data. It must be SAFE: only ever touch a soft-deleted row, and never +remove or mutate LIVE data via a cascade/SET-NULL. This boots the real server, seeds live + +soft-deleted graphs, and drives /api/admin/soft-deleted[/purge] over HTTP. Synthetic only. + +Run: cd backend && python3 test_purge_soft_deleted.py +""" +import http.client +import json +import os +import sqlite3 +import sys +import tempfile +import threading +from http.server import ThreadingHTTPServer + +_DATA = tempfile.mkdtemp() +os.environ["CRM_DATA_DIR"] = _DATA +os.environ["CRM_DB_PATH"] = os.path.join(_DATA, "crm.db") + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import server # noqa: E402 + +FAILS = [] +DEL = "2026-06-01T00:00:00" + + +def check(cond, msg): + print((" PASS " if cond else " FAIL ") + msg) + if not cond: + FAILS.append(msg) + + +class _Quiet(server.CRMHandler): + def log_message(self, *a): + pass + + +def _post(port, path, token, payload): + conn = http.client.HTTPConnection("127.0.0.1", port, timeout=10) + conn.request("POST", path, body=json.dumps(payload), + headers={"Authorization": "Bearer " + token, "Content-Type": "application/json"}) + resp = conn.getresponse() + body = resp.read().decode("utf-8", "replace") + conn.close() + try: + return resp.status, (json.loads(body) if body else None) + except ValueError: + return resp.status, None + + +def _get(port, path, token): + conn = http.client.HTTPConnection("127.0.0.1", port, timeout=10) + conn.request("GET", path, headers={"Authorization": "Bearer " + token}) + resp = conn.getresponse() + body = resp.read().decode("utf-8", "replace") + conn.close() + try: + return resp.status, (json.loads(body) if body else None) + except ValueError: + return resp.status, None + + +def exists(table, rid): + c = sqlite3.connect(os.environ["CRM_DB_PATH"]) + n = c.execute(f"SELECT COUNT(*) FROM {table} WHERE id = ?", (rid,)).fetchone()[0] + c.close() + return n > 0 + + +def seed(): + c = sqlite3.connect(os.environ["CRM_DB_PATH"]) + c.execute("INSERT INTO users (id,username,email,password_hash,full_name,role,is_active) " + "VALUES ('u1','grant','grant@ten31.example','x','Grant','admin',1)") + # Soft-deleted contact with ONLY soft-deleted children -> purgeable; cascade should remove them. + c.execute("INSERT INTO contacts (id,first_name,last_name,deleted_at) VALUES ('cClean','Dummy','Clean',?)", (DEL,)) + c.execute("INSERT INTO opportunities (id,name,contact_id,owner_id,deleted_at) VALUES ('opC','Opp','cClean','u1',?)", (DEL,)) + c.execute("INSERT INTO communications (id,contact_id,communication_date,created_by,subject,deleted_at) VALUES ('cmC','cClean','2026-05-01','u1','note',?)", (DEL,)) + # A reminder pointing at the purge target (reminders.contact_id is a bare logical FK, no ON DELETE): + # the purge must NULL it, not leave it dangling and not delete the reminder. + c.execute("INSERT INTO reminders (id,contact_id,investor_id,title) VALUES ('remC','cClean','inv-x','Follow up dummy')") + # Soft-deleted contact WITH a live child -> must refuse (cascade would kill live data). + c.execute("INSERT INTO contacts (id,first_name,last_name,deleted_at) VALUES ('cLiveKid','Has','Livekid',?)", (DEL,)) + c.execute("INSERT INTO communications (id,contact_id,communication_date,created_by,subject) VALUES ('cmLive','cLiveKid','2026-05-02','u1','live note')") + # A live contact -> must refuse (not soft-deleted). + c.execute("INSERT INTO contacts (id,first_name,last_name) VALUES ('cLive','Real','Person')") + # Soft-deleted org with no live refs -> purgeable. + c.execute("INSERT INTO organizations (id,name,deleted_at) VALUES ('orgClean','Dead Org',?)", (DEL,)) + # Soft-deleted org referenced by a LIVE contact -> must refuse (SET NULL would mutate live data). + c.execute("INSERT INTO organizations (id,name,deleted_at) VALUES ('orgRef','Ref Org',?)", (DEL,)) + c.execute("INSERT INTO contacts (id,first_name,last_name,organization_id) VALUES ('cRef','Org','Member','orgRef')") + c.commit() + c.close() + + +def main(): + server.init_db() + seed() + token = server.create_token("u1", "grant", "admin") + httpd = ThreadingHTTPServer(("127.0.0.1", 0), _Quiet) + port = httpd.server_address[1] + threading.Thread(target=httpd.serve_forever, daemon=True).start() + try: + print("\n[list soft-deleted]") + st, body = _get(port, "/api/admin/soft-deleted", token) + groups = (body or {}).get("groups", {}) + cids = {x["id"] for x in groups.get("contacts", [])} + oids = {x["id"] for x in groups.get("organizations", [])} + check(st == 200, f"GET soft-deleted -> 200 (got {st})") + check({"cClean", "cLiveKid"} <= cids and "cLive" not in cids, f"lists soft-deleted contacts only (got {cids})") + check({"orgClean", "orgRef"} <= oids, f"lists soft-deleted orgs (got {oids})") + check("opC" in {x["id"] for x in groups.get("opportunities", [])}, "lists the soft-deleted opportunity") + + print("\n[purge guards]") + st, _ = _post(port, "/api/admin/soft-deleted/purge", token, {"table": "contacts", "id": "cLive"}) + check(st == 400, f"purge a LIVE contact -> 400 (got {st})") + check(exists("contacts", "cLive"), "live contact still present after refused purge") + st, _ = _post(port, "/api/admin/soft-deleted/purge", token, {"table": "contacts", "id": "cLiveKid"}) + check(st == 409, f"purge contact with a LIVE child -> 409 (got {st})") + check(exists("contacts", "cLiveKid") and exists("communications", "cmLive"), "contact + its live child preserved") + st, _ = _post(port, "/api/admin/soft-deleted/purge", token, {"table": "organizations", "id": "orgRef"}) + check(st == 409, f"purge org referenced by a LIVE contact -> 409 (got {st})") + check(exists("organizations", "orgRef") and exists("contacts", "cRef"), "org + its live referencing contact preserved") + st, _ = _post(port, "/api/admin/soft-deleted/purge", token, {"table": "bogus", "id": "x"}) + check(st == 400, f"unknown table -> 400 (got {st})") + st, _ = _post(port, "/api/admin/soft-deleted/purge", token, {"table": "contacts", "id": "nope"}) + check(st == 404, f"missing id -> 404 (got {st})") + + print("\n[purge happy path + cascade]") + st, _ = _post(port, "/api/admin/soft-deleted/purge", token, {"table": "contacts", "id": "cClean"}) + check(st == 200, f"purge a clean soft-deleted contact -> 200 (got {st})") + check(not exists("contacts", "cClean"), "purged contact is gone") + check(not exists("opportunities", "opC") and not exists("communications", "cmC"), + "its soft-deleted children were cascade-removed") + _rc = sqlite3.connect(os.environ["CRM_DB_PATH"]) + _rem = _rc.execute("SELECT contact_id FROM reminders WHERE id = 'remC'").fetchone() + _rc.close() + check(_rem is not None and _rem[0] is None, + "a reminder on the purged contact is KEPT but its contact_id is NULL'd (no dangling ref)") + st, _ = _post(port, "/api/admin/soft-deleted/purge", token, {"table": "organizations", "id": "orgClean"}) + check(st == 200, f"purge a clean soft-deleted org -> 200 (got {st})") + check(not exists("organizations", "orgClean"), "purged org is gone") + finally: + httpd.shutdown() + + print() + if FAILS: + print(f"{len(FAILS)} FAILED") + sys.exit(1) + print("ALL PASS (soft-deleted purge)") + + +if __name__ == "__main__": + main() diff --git a/docs/crm-overview.md b/docs/crm-overview.md index f93a3d3..cb7da1c 100644 --- a/docs/crm-overview.md +++ b/docs/crm-overview.md @@ -8,7 +8,7 @@ - **One Python file, no framework.** The whole backend is `backend/server.py` (~4,530 lines): a stdlib `http.server.ThreadingHTTPServer` with a hand-written `CRMHandler(BaseHTTPRequestHandler)` and manual path dispatch. `requirements.txt` lists FastAPI/SQLAlchemy/Alembic/Pydantic but **none are imported** — they are vestigial. - **Storage is one SQLite file** (`data/crm.db`), WAL mode, opened fresh per request. Schema is created idempotently in-code at boot. There is no Alembic; "migrations" are `CREATE TABLE IF NOT EXISTS` + best-effort `ALTER TABLE ADD COLUMN`. -- **Two parallel investor data models** coexist with no shared key: (1) the *classic* `contacts / organizations / opportunities / communications / lp_profiles` CRM, and (2) the *newer, actively-used* `fundraising_*` collaborative grid. They are bridged only by fuzzy name/email matching. **This duality is the central entity-resolution problem for Phase 0.** +- **Two parallel investor data models** coexist with no shared key: (1) the *classic* `contacts / organizations / opportunities / communications` CRM, and (2) the *newer, actively-used* `fundraising_*` collaborative grid. They are bridged only by fuzzy name/email matching. **This duality is the central entity-resolution problem for Phase 0.** *(Note: `lp_profiles`, formerly part of the classic model, was dropped in v0.1.0:104 — migration `0008`; the grid is now canonical for commitments. The `lp_profiles` mentions later in this doc are retained as historical context for the entity-resolution discussion.)* - **A real Gmail subsystem** (`backend/email_integration/`) stores threaded correspondence in `crm.db` and matches emails to investors — but is **self-disabling** (off unless a service-account key is present). - **Auth is a single scheme:** username/password → HS256 JWT (Bearer header), re-validated against the `users` table each request; two roles (`admin`/`member`). The `X_API_KEY` named in `CLAUDE.md`/`PHASE_0.md` **does not exist in the code** — it is aspirational. - **Guardrail flags:** all deletes are **hard deletes** (violates guardrail #3 as written); a destructive `POST /api/admin/reset-all-data` exists; `audit_log` is mutation-only and is *not* the append-only interaction log Phase 0 wants. @@ -73,7 +73,7 @@ PKs are **8-char truncated UUIDs** (`generate_id()` = `str(uuid.uuid4())[:8]`, ` | `organizations` | weak parent of contacts/opps | `name` (not unique), `type` (free-text, default `other`), `tags` JSON, `description`. (`backend/server.py:104`) | | `contacts` | **the hub** | `first_name`/`last_name` (req), `organization_id` (FK SET NULL), `contact_type` (free-text; load-bearing values `prospect`/`investor`), `status` (default `active`), `source`, `tags` JSON, `notes`, `linkedin_url`. (`backend/server.py:123`) | | `opportunities` | deal pipeline | `contact_id` (req, FK **CASCADE**), `stage` (allowlist `PIPELINE_STAGES` at `backend/server.py:1380`, enforced **only** on the stage endpoint), `commitment_amount`, `expected_amount`, `fund_name`, `owner_id`, `lost_reason`. (`backend/server.py:148`) | -| `lp_profiles` | closed-LP extension | 1:1 with a contact (`contact_id` UNIQUE, FK CASCADE). Holds `commitment_amount`, `funded_amount`, `accredited` (bare 0/1), `legal_docs_signed`, `wire_received`, `k1_sent`, `investor_type` (free-text). (`backend/server.py:186`) | +| `lp_profiles` | **RETIRED** | Dropped in v0.1.0:104 (migration `0008_drop_retired_tables`) — the table was empty; the `fundraising_*` grid is the canonical commitment record. Formerly a 1:1 closed-LP extension of a contact holding commitment/funded/accreditation fields. | | `custom_fields` / `custom_field_values` | EAV custom fields | **Dead**: schema exists but has **no routes/handlers**; only ever wiped by reset. Do not build on this. (`backend/server.py:206`) | | `tags` | global tag palette | `name` UNIQUE + `color`. Not FK-linked to the per-row `tags` JSON arrays; just an autocomplete source. (`backend/server.py:237`) | | `audit_log` | mutation diff trail | `user_id`, `entity_type`, `entity_id`, `action`, `changes` JSON. **Mutation-only**, no reads, no actor/agent dimension. (`backend/server.py:227`) | @@ -140,11 +140,9 @@ Full REST verbs exist (mutations are **not** tunneled through POST): `do_GET` (1 | GET/POST · GET/PUT/DELETE | `/api/opportunities[/{id}]` | Opp CRUD | Bearer | | PATCH | `/api/opportunities/{id}/stage` | Move pipeline stage (validated) | Bearer | | GET/POST · GET/PUT/DELETE | `/api/communications[/{id}]` | Comms CRUD | Bearer | -| GET/POST · GET/PUT | `/api/lp-profiles[/{id}]` | LP-profile CRUD (no delete route) | Bearer | -| GET | `/api/reports/{dashboard,pipeline,lp-breakdown,activity}` | Aggregates | Bearer | +| GET | `/api/reports/{dashboard,pipeline,activity}` | Aggregates | Bearer | | GET | `/api/export/contacts` | Export **all** contacts (returns JSON, not CSV) | Bearer | | POST | `/api/import/csv` | Bulk import from JSON rows | Bearer | -| GET/POST · PATCH | `/api/feature-requests[/{id}]` | Feature-request tracker | Bearer | | GET | `/api/users` | List users (no hashes) | Bearer | | POST · PATCH | `/api/admin/users[/{id}]` | Create / update user | **Admin** | | POST | `/api/admin/reset-all-data` | ⚠️ Wipe CRM (confirm phrase `RESET ALL DATA`) | **Admin** | diff --git a/docs/guides/migrations.md b/docs/guides/migrations.md index d0b1ce9..9233936 100644 --- a/docs/guides/migrations.md +++ b/docs/guides/migrations.md @@ -21,6 +21,7 @@ Read this before adding or editing a schema migration or a one-time seed/backfil - **Make migrations/seeders deployment-state-invariant.** Target rows **structurally**, not by transient text the same change mutates; capture prior state so a revert is exact. - *Learned the hard way:* matching old nodes by a body string the same changeset deleted broke fresh DBs. A migration must produce the same end state whether the box is empty, mid-version, or fully seeded. - **Soft-delete only** — `deleted_at` and/or `status='retired'`; never hard-delete CRM records or thesis history. +- **Dropping a table is forbidden by default — it needs explicit sign-off** (never-hard-delete). `0008_drop_retired_tables` (lp_profiles + feature_requests, v0.1.0:104) is the one sanctioned exception, for **empty** retired tables only. To actually drop one: (1) **remove its `CREATE TABLE` from `init_db()`** — `init_db()` runs every boot, so leaving it there re-creates the table right after the drop migration runs; (2) add a `DROP TABLE IF EXISTS` forward migration + a `.down.sql` recreating the empty shell; (3) **remove any `ALTER TABLE ` line from an earlier historical migration** — once `init_db()` stops creating the table, that ALTER fails `no such table` on a *fresh* DB and aborts the whole chain (it was the actual bug here). Editing that old migration is safe and deployment-state-invariant: live DBs already applied it before the drop, so the edit only affects fresh DBs, which converge to the same end state. `DROP TABLE IF EXISTS` is a no-op on a fresh DB and removes the table on the live box. ## Verify before shipping diff --git a/frontend/index.html b/frontend/index.html index db8793f..a63094a 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -2612,6 +2612,13 @@ .bell-summary { font-size: 13px; color: var(--text-secondary); line-height: 1.45; margin: 0 0 14px; padding: 10px 12px; background: var(--bg-input); border: 1px solid var(--border); border-radius: 8px; } .bell-back { width: 100%; margin-top: 10px; background: transparent; border: none; color: var(--accent-light); font-size: 14px; font-family: inherit; cursor: pointer; padding: 8px; } .bell-back:disabled { opacity: 0.5; cursor: default; } + .bell-review-head { display: flex; align-items: flex-start; justify-content: space-between; gap: 10px; } + .bell-reminder-btn { + flex: none; width: 34px; height: 34px; border-radius: 50%; padding: 0; cursor: pointer; + display: inline-flex; align-items: center; justify-content: center; + background: var(--bg-input); border: 1px solid var(--border); color: var(--text-secondary); + } + .bell-reminder-btn:active { background: var(--bg-hover); } .quicklog-hint { font-size: 13px; color: var(--text-subtle); line-height: 1.5; margin: 0 0 12px; } .quicklog-pool { display: flex; flex-direction: column; gap: 8px; margin-top: 12px; } .quicklog-empty { font-size: 13px; color: var(--text-subtle); padding: 16px 4px; } @@ -2993,17 +3000,6 @@ return '-'; }; - const loadFeatureRequests = () => { - try { - const raw = localStorage.getItem('venture_crm_feature_requests'); - if (!raw) return []; - const parsed = JSON.parse(raw); - return Array.isArray(parsed) ? parsed : []; - } catch (_) { - return []; - } - }; - const FUNDRAISING_GRID_STORAGE_KEY = 'venture_crm_fundraising_grid_v1'; const FUNDRAISING_VIEWS_STORAGE_KEY = 'venture_crm_fundraising_views_v1'; const FUNDRAISING_VERSION_STORAGE_KEY = 'venture_crm_fundraising_version_v1'; @@ -3134,27 +3130,13 @@ { id: 'm-3002', contact_id: 'c-1003', contact_name: 'David Martinez', type: 'call', subject: 'Follow-up call', body: 'Requested portfolio details.', communication_date: '2026-02-10T10:00:00Z', outcome: 'positive', next_action: 'Send updated deck', next_action_date: '2026-02-20' }, { id: 'm-3003', contact_id: 'c-1004', contact_name: 'Jennifer Taylor', type: 'email', subject: 'DDQ package', body: 'Sent due diligence package.', communication_date: '2026-02-11T09:00:00Z', outcome: 'neutral', next_action: 'Schedule IC call', next_action_date: '2026-02-22' } ], - lp_profiles: [ - { id: 'lp-4001', contact_id: 'c-1001', contact_name: 'James Chen', organization: 'Sovereign Wealth Holdings', commitment_amount: 25000000, funded_amount: 25000000, fund_name: 'Fund I', legal_docs_signed: true, wire_received: true, k1_sent: true }, - { id: 'lp-4002', contact_id: 'c-1002', contact_name: 'Sarah Williams', organization: 'Pacific Capital Partners', commitment_amount: 15000000, funded_amount: 15000000, fund_name: 'Fund I', legal_docs_signed: true, wire_received: true, k1_sent: false } - ], tags: [ { id: 't-5001', name: 'High Priority', color: '#ef4444' }, { id: 't-5002', name: 'Fund II Prospect', color: 'var(--accent)' } ], - feature_requests: loadFeatureRequests(), audit_log: [] }; - const persistFeatureRequests = () => { - if (!MOCK_MODE) return; - try { - localStorage.setItem('venture_crm_feature_requests', JSON.stringify(mockDb.feature_requests || [])); - } catch (_) { - // no-op - } - }; - const loadMockFundraisingGrid = () => { try { const raw = localStorage.getItem(FUNDRAISING_GRID_STORAGE_KEY); @@ -3222,15 +3204,6 @@ }; }); - mockDb.lp_profiles = mockDb.lp_profiles.map((lp) => { - const c = mockDb.contacts.find((x) => x.id === lp.contact_id); - return { - ...lp, - contact_name: contactName(c), - organization: c?.organization_name || c?.organization || lp.organization || '' - }; - }); - mockDb.communications = mockDb.communications.map((m) => { const c = mockDb.contacts.find((x) => x.id === m.contact_id); return { ...m, contact_name: contactName(c) }; @@ -3260,7 +3233,6 @@ const metrics = { total_lps: mockDb.contacts.filter((c) => c.contact_type === 'investor').length, total_prospects: mockDb.contacts.filter((c) => c.contact_type === 'prospect').length, - total_committed: mockDb.lp_profiles.reduce((s, lp) => s + (lp.commitment_amount || 0), 0), pipeline_value: mockDb.opportunities.filter((o) => o.stage !== 'commitment').reduce((s, o) => s + (o.expected_amount || 0), 0), active_opportunities: mockDb.opportunities.filter((o) => o.stage !== 'commitment').length, comms_this_month: mockDb.communications.length @@ -3314,8 +3286,7 @@ if (!item) throw new Error('Contact not found'); const opportunities = mockDb.opportunities.filter((o) => o.contact_id === id); const communications = mockDb.communications.filter((m) => m.contact_id === id); - const lp = mockDb.lp_profiles.find((lpRow) => lpRow.contact_id === id) || null; - return makeResult({ data: clone({ ...item, opportunities, communications, lp_profile: lp }) }); + return makeResult({ data: clone({ ...item, opportunities, communications }) }); } if (/^\/api\/contacts\/[^/]+$/.test(path) && method === 'DELETE') { @@ -3323,7 +3294,6 @@ mockDb.contacts = mockDb.contacts.filter((c) => c.id !== id); mockDb.opportunities = mockDb.opportunities.filter((o) => o.contact_id !== id); mockDb.communications = mockDb.communications.filter((m) => m.contact_id !== id); - mockDb.lp_profiles = mockDb.lp_profiles.filter((lp) => lp.contact_id !== id); return makeResult({ message: 'Contact deleted' }); } @@ -3468,38 +3438,6 @@ return makeResult({ message: 'Communication deleted' }); } - if (path === '/api/lp-profiles' && method === 'GET') { - const search = (params.get('search') || '').toLowerCase(); - let rows = [...mockDb.lp_profiles]; - if (search) rows = rows.filter((r) => `${r.contact_name || ''} ${r.organization || ''}`.toLowerCase().includes(search)); - return makeResult({ data: clone(rows), total: rows.length }); - } - - if (path === '/api/lp-profiles' && method === 'POST') { - const c = mockDb.contacts.find((x) => x.id === body.contact_id); - if (!c) throw new Error('Valid contact is required'); - const item = { - id: `lp-${Date.now()}`, - contact_id: body.contact_id, - contact_name: contactName(c), - organization: c.organization_name || c.organization || '', - commitment_amount: Number(body.commitment_amount) || 0, - funded_amount: Number(body.funded_amount) || 0, - fund_name: body.fund_name || '', - legal_docs_signed: !!body.legal_docs_signed, - wire_received: !!body.wire_received, - k1_sent: !!body.k1_sent - }; - mockDb.lp_profiles.unshift(item); - return makeResult({ data: clone(item) }, 201); - } - - if (/^\/api\/lp-profiles\/[^/]+$/.test(path) && method === 'DELETE') { - const id = path.split('/').pop(); - mockDb.lp_profiles = mockDb.lp_profiles.filter((lp) => lp.id !== id); - return makeResult({ message: 'LP deleted' }); - } - if (path === '/api/import/csv' && method === 'POST') { const rows = Array.isArray(body.data) ? body.data : []; return makeResult({ data: { created: rows.length, updated: 0, skipped: 0, errors: [] }, dry_run: !!body.dry_run }); @@ -3547,54 +3485,6 @@ return makeResult({ data: { presence: [], locks: [], lock_conflict: null } }); } - if (path === '/api/feature-requests' && method === 'GET') { - const status = params.get('status') || ''; - const search = (params.get('search') || '').toLowerCase(); - let rows = [...mockDb.feature_requests]; - if (status) rows = rows.filter((r) => r.status === status); - if (search) { - rows = rows.filter((r) => `${r.title || ''} ${r.description || ''} ${r.requested_by || ''} ${r.category || ''}`.toLowerCase().includes(search)); - } - rows.sort((a, b) => (a.created_at > b.created_at ? -1 : 1)); - return makeResult({ data: clone(rows), total: rows.length }); - } - - if (path === '/api/feature-requests' && method === 'POST') { - if (!body.title || !body.title.trim()) throw new Error('Title is required'); - const item = { - id: `fr-${Date.now()}`, - title: body.title.trim(), - description: (body.description || '').trim(), - category: body.category || 'other', - priority: body.priority || 'medium', - status: 'new', - requested_by: (body.requested_by || 'Unknown').trim(), - page: body.page || '', - created_at: new Date().toISOString(), - updated_at: new Date().toISOString() - }; - mockDb.feature_requests.unshift(item); - persistFeatureRequests(); - return makeResult({ data: clone(item) }, 201); - } - - if (/^\/api\/feature-requests\/[^/]+$/.test(path) && method === 'PATCH') { - const id = path.split('/').pop(); - const allowed = ['status', 'priority', 'category', 'page']; - mockDb.feature_requests = mockDb.feature_requests.map((r) => { - if (r.id !== id) return r; - const next = { ...r, updated_at: new Date().toISOString() }; - allowed.forEach((field) => { - if (Object.prototype.hasOwnProperty.call(body, field)) next[field] = body[field]; - }); - return next; - }); - persistFeatureRequests(); - const item = mockDb.feature_requests.find((r) => r.id === id); - if (!item) throw new Error('Feature request not found'); - return makeResult({ data: clone(item) }); - } - throw new Error(`Mock endpoint not implemented: ${method} ${path}`); }; @@ -5945,10 +5835,23 @@ (async () => { try { setLoading(true); - // One fetch of the full directory (server cap 500); tab + search + sort are - // applied client-side so switching is instant and needs no refetch. - const r = await api('/api/contacts?sort=last_name&order=asc&limit=500', {}, token); - if (!cancelled) { setContacts(r.data || []); setError(''); } + // Page through the WHOLE directory (the server caps each page at 500 — a single + // fetch silently truncated at 720 contacts, hiding everyone past ~"Pol" from the + // list AND from client-side search). Accumulate pages until the full set is in. + // tab + search + sort stay client-side so switching needs no refetch. + const PAGE = 500; + let all = []; + let offset = 0; + for (;;) { + const r = await api(`/api/contacts?sort=last_name&order=asc&limit=${PAGE}&offset=${offset}`, {}, token); + if (cancelled) return; + const batch = r.data || []; + all = all.concat(batch); + offset += PAGE; + // Stop on a short/empty page or once we've gathered the reported total. + if (batch.length < PAGE || all.length >= Number(r.total || 0)) break; + } + if (!cancelled) { setContacts(all); setError(''); } } catch (err) { if (!cancelled) setError(getErrorMessage(err, 'Failed to load contacts')); } finally { @@ -7294,284 +7197,6 @@ ); }; - const FeatureRequestsPage = ({ token, onShowToast, user }) => { - const [requests, setRequests] = useState([]); - const [loading, setLoading] = useState(true); - const [search, setSearch] = useState(''); - const [statusFilter, setStatusFilter] = useState(''); - const [showForm, setShowForm] = useState(false); - const [formError, setFormError] = useState(''); - const [formData, setFormData] = useState({ - title: '', - description: '', - category: 'ui_ux', - priority: 'medium', - page: '', - requested_by: user?.full_name || user?.username || '' - }); - - const fetchRequests = useCallback(async () => { - try { - setLoading(true); - const result = await api(`/api/feature-requests?status=${statusFilter}&search=${encodeURIComponent(search)}`, {}, token); - setRequests(result.data || []); - } catch (err) { - onShowToast(getErrorMessage(err, 'Failed to load feature requests'), 'error'); - } finally { - setLoading(false); - } - }, [statusFilter, search, token, onShowToast]); - - useEffect(() => { - fetchRequests(); - }, [fetchRequests]); - - const handleSubmit = async (e) => { - e.preventDefault(); - setFormError(''); - try { - await api('/api/feature-requests', { - method: 'POST', - body: JSON.stringify(formData) - }, token); - setShowForm(false); - setFormData({ - title: '', - description: '', - category: 'ui_ux', - priority: 'medium', - page: '', - requested_by: user?.full_name || user?.username || '' - }); - await fetchRequests(); - onShowToast('Feature request submitted', 'success'); - } catch (err) { - setFormError(getErrorMessage(err, 'Failed to submit request')); - } - }; - - const handleStatusChange = async (id, status) => { - try { - await api(`/api/feature-requests/${id}`, { - method: 'PATCH', - body: JSON.stringify({ status }) - }, token); - setRequests((prev) => prev.map((r) => (r.id === id ? { ...r, status } : r))); - } catch (err) { - onShowToast(getErrorMessage(err, 'Failed to update status'), 'error'); - } - }; - - const handlePriorityChange = async (id, priority) => { - try { - await api(`/api/feature-requests/${id}`, { - method: 'PATCH', - body: JSON.stringify({ priority }) - }, token); - setRequests((prev) => prev.map((r) => (r.id === id ? { ...r, priority } : r))); - } catch (err) { - onShowToast(getErrorMessage(err, 'Failed to update priority'), 'error'); - } - }; - - const counts = useMemo(() => ({ - total: requests.length, - newCount: requests.filter((r) => r.status === 'new').length, - planned: requests.filter((r) => r.status === 'planned').length, - done: requests.filter((r) => r.status === 'done').length - }), [requests]); - - return ( -
-
-

Feature Requests

- -
- -
-
-
Total Requests
-
{counts.total}
-
-
-
New
-
{counts.newCount}
-
-
-
Planned
-
{counts.planned}
-
-
-
Done
-
{counts.done}
-
-
- -
-
- setSearch(e.target.value)} - /> - -
- - {loading ? ( - - ) : requests.length === 0 ? ( -
No feature requests yet
- ) : ( - - - - - - - - - - - - - {requests.map((r) => ( - - - - - - - - - ))} - -
TitleRequested ByCategoryPriorityStatusSubmitted
-
{r.title}
- {r.description &&
{r.description}
} - 		{r.requested_by || '-'}{(r.category || 'other').replace('_', ' ')} - - - - {formatDateLong(r.created_at)}
- )} -
- - {showForm && ( -
-
-
Submit Feature Request
- {formError &&
{formError}
} -
-
- - setFormData({ ...formData, title: e.target.value })} - required - /> -
-
- -