Keysat
f0f1ed3bcd
DueChip + .due-chip--{bucket}/--due-*-* slots in the chip-var enumeration;
note the stacked investor-picker and the source_row_id -> investor_id
reminder-link convention (PATCH can't reassign).
118 lines
21 KiB
Markdown
118 lines
21 KiB
Markdown
# Ten31 Venture CRM + Agentic System — AGENTS.md
|
||
|
||
**The foundation is a self-hosted venture-fund CRM** — a purpose-built fundraising tool that replaced Airtable to (1) keep sensitive LP/prospect data off third-party servers, (2) drop subscription cost, and (3) fit the fund's workflow: managing ~150 existing LPs, tracking 250+ prospects, and running the capital-raise pipeline. Core CRM domain: contacts (investor/prospect/advisor), organizations, opportunities (the deal pipeline), and communications; investor commitments live in the canonical `fundraising_*` grid (the legacy single-fund `lp_profiles` table was retired in v0.1.0:78). The fund (Ten31, ~$200M AUM, bitcoin/energy/AI thesis) runs it on a Start9 box, accessed over ClearNet (StartOS StartTunnel) with app-level user auth by a team of ~5 (Tailscale is not in use). Schema/API tour: `docs/crm-overview.md`.
|
||
|
||
**The agentic system is new functionality built on top of that CRM** — an in-house AI layer to widen the fundraising funnel, sharpen the thesis, and automate outreach drafting. Frontier reasoning runs on Claude (Agent SDK/API); privacy-sensitive and bulk work runs on local DGX Spark models via the **Spark Control** gateway. **Phase 0/1 — no live outward-facing agents; agents draft, humans send.**
|
||
|
||
> **Inbox check:** At session start, if `~/Projects/standards/INBOX.md` exists, scan it for
|
||
> items tagged `(CRM)` and surface them before proposing next steps; triage with `/triage`.
|
||
|
||
## Stack (versions that matter)
|
||
|
||
- **Python 3.11, standard library only at runtime.** The CRM is one monolith, `backend/server.py` (~5k lines): a stdlib `http.server.ThreadingHTTPServer` + hand-written `CRMHandler` with manual path dispatch (`do_GET`/`do_POST`). **Not FastAPI.** `backend/requirements.txt` lists FastAPI/SQLAlchemy/Alembic/Pydantic/pytest-style deps but **none are imported at runtime** (vestigial).
|
||
- **SQLite** at `data/crm.db` (WAL, `foreign_keys=ON`), opened per-request via `get_db()`. Schema via ordered migrations.
|
||
- **Frontend:** single `frontend/index.html`, inline-Babel React. **No build step.**
|
||
- Optional runtime deps, used only if present: `bcrypt`, `PyJWT` (`jwt`), `cryptography` (Gmail module).
|
||
- **MCP + ingest** (in the Docker image, not the bare CRM): `mcp==1.2.0` (FastMCP, `backend/mcp/server.py`), `fastembed==0.4.2`, `anthropic`, `cryptography==42.0.5`.
|
||
- **Packaging:** StartOS 0.4, TypeScript SDK (`@start9labs/start-sdk`) under `start9/0.4/startos/`. Live target is `start9/0.4/`.
|
||
- **Local models** (bge-m3 embeddings, bge-reranker-v2-m3, `/api/search`, Qdrant): always via Spark Control. Contract: `docs/EMBEDDINGS.md`.
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
# Run locally (dev, port 8080; or ./start.sh <port>) — runs python3 backend/server.py
|
||
./start.sh
|
||
# Run prod-mode (beta) — requires CRM_SECRET_KEY
|
||
./start_beta.sh
|
||
# Sanity-check edits (there is no compiler/build for the CRM)
|
||
python3 -m py_compile backend/server.py
|
||
# Run ONE test (tests are standalone scripts with `if __name__ == "__main__"`; no pytest installed)
|
||
python3 backend/redaction/test_scrub_leak.py # substitute any backend/**/test_*.py
|
||
# Run all tests (aggregate runner — runs each backend/**/test_*.py in its own subprocess)
|
||
python3 backend/run_tests.py # add substrings to filter, e.g. `... soft_delete redaction`
|
||
# Build + install the s9pk — BUMP THE VERSION FIRST. See docs/guides/packaging.md.
|
||
cd start9/0.4 && make
|
||
```
|
||
|
||
- **Migrations** apply automatically at startup (`backend/core_migrations.py`, `schema_migrations` ledger). See `docs/guides/migrations.md` before adding one.
|
||
- **Lint:** none configured.
|
||
|
||
## Directory layout (day-one)
|
||
|
||
- `backend/server.py` — the CRM monolith: HTTP handler, route dispatch, `init_db()`, auth (username/password → HS256 JWT, roles admin/member/bot).
|
||
- `backend/core_migrations.py` + `backend/migrations/NNNN_*.sql` (+ paired `.down.sql`) — additive schema migrations, applied at startup.
|
||
- `backend/thesis_seed.py` — Thesis Workshop seed + idempotent `ensure_*` one-time seeders, wired in `server.init_db()`.
|
||
- `backend/thesis_review.py` — thesis version review/approval (human dual sign-off → canonical).
|
||
- `backend/mcp/` — `architect_agent.py` (Claude thesis copilot), `architect_tools.py`, `outreach_agent.py` (LP draft assistant), `architect_grounding.py`, `crm_tools.py`, `server.py` (FastMCP).
|
||
- `backend/email_integration/` — Gmail capture via domain-wide delegation + Tier-B draft creation (`compose.py`).
|
||
- `backend/redaction/` — `scrub.py` + `client.py`: the scrub→Claude→re-hydrate privacy boundary.
|
||
- `backend/ingest/` — chunk→embed→Qdrant + retrieval modes.
|
||
- `backend/entity_*.py` — entity resolution/merge (the two-investor-model reconciliation).
|
||
- `backend/nl_query/` — read-only natural-language query (W2): `intents.py` (curated parameterized query catalog), `runner.py` (slot validator = trust boundary), `translate.py` (local-Qwen question→{intent,slots}). See the nl-query guide.
|
||
- `backend/matrix_intake/` — Matrix intake bot (separate process; `matrix-nio`, isolated to this component): typed message → local-Qwen parse → in-thread approve → write via the CRM's own `log-communication`. See the matrix-intake guide.
|
||
- `frontend/index.html` — the entire UI.
|
||
- `docs/` — architecture, phase plans, contracts, runbooks (see Deeper docs). `docs/guides/` — scoped subsystem rules (see below).
|
||
- `start9/0.4/` — StartOS package (`startos/utils.ts` holds `PACKAGE_VERSION`).
|
||
- `data/crm.db` — the live DB (gitignored). `.env` / `.env.example` — config (`.env` gitignored).
|
||
|
||
## Scoped guides
|
||
|
||
Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude/rules/` symlinks (scoped by `paths:` frontmatter). **Read the guide before editing that area:**
|
||
|
||
- **Migrations or seeders** (`backend/migrations/`, `core_migrations.py`, `thesis_seed.py`) → `docs/guides/migrations.md`
|
||
- **Thesis logic** (`backend/thesis_*.py`, `backend/mcp/architect_*.py`) → `docs/guides/thesis.md`
|
||
- **Redaction or any MCP/Claude path** (`backend/redaction/`, `backend/mcp/`) → `docs/guides/redaction.md`
|
||
- **Ingest / retrieval** (`backend/ingest/`) → `docs/guides/spark-ingest.md`
|
||
- **Email capture / drafts + digest send** (`backend/email_integration/`, `backend/digest_mailer.py`, `backend/smtp_send.py`) → `docs/guides/email.md`
|
||
- **Building or deploying the s9pk** (`start9/`) → `docs/guides/packaging.md`
|
||
- **Matrix intake bot** (`backend/matrix_intake/`) → `docs/guides/matrix-intake.md`
|
||
- **Natural-language query** (`backend/nl_query/`) → `docs/guides/nl-query.md`
|
||
|
||
## Conventions
|
||
|
||
- **Investor model — the grid is canonical (since v0.1.0:78).** The `fundraising_*` grid is the **system of record**: an investor entity (row) → many contact "pills" → per-fund commitments. The classic `contacts` table is a **read-only per-person directory**, auto-populated from the grid — create/edit people in the grid, not the Contacts page. Email capture rolls multiple people up to one investor. The legacy single-fund `lp_profiles` model is **retired** (empty table kept, per never-hard-delete). Reconciling grid ↔ classic `contacts` to canonical IDs is the core entity-resolution task — see `docs/crm-overview.md`. **Derived read-only columns** (`pipeline`, `pipeline_stage`, `opportunity_id`, `reminder_status`, `existing_investor`, `last_activity_at`, `staleness`) are computed live and **injected on GET, never persisted** — any new one MUST be added to BOTH strip points (`server.py` `_computed_row_values` + frontend `stripComputedRows`) or it dirties the autosave / leaks into the blob. Pipeline stage is the 4-stage funnel `lead→engaged→diligence→commitment` (`PIPELINE_STAGES`), terminal at commitment.
|
||
- **Soft-delete only:** `deleted_at` and/or `status='retired'`; never hard-delete. Every READ path must filter `deleted_at IS NULL` — list handlers, get-by-id, nested related-data sub-selects, **and aggregate sub-selects (`COUNT`/`SUM`/`MAX`)**. Audits found leaks in all of these (2026-06-12 detail + nested; 2026-06-13 list-view `contact_count`/`total_funded`/`comm_count`); the **opportunities/pipeline** aggregates were fixed in v0.1.0:87 (`handle_pipeline_report` + dashboard pipeline metrics now filter `deleted_at`), but the **reports** subsystem's **communications-side** aggregates (dashboard `recent_comms`/`comms_this_month`/`meetings_this_month`, activity report) still leak (see Current state). Regression-guarded by `backend/test_soft_delete_reads.py` (+ `test_reminders.py` for the reminders read paths, incl. the recency rollup whose email-activity liveness signal is `email_account_messages.deleted_at`, not `emails`). (Thesis has a subtlety here — see the thesis guide.)
|
||
- **Env:** secrets in `.env` (gitignored); names in `.env.example`. Verified names: `ANTHROPIC_API_KEY`, `SPARK_CONTROL_URL`, `SPARK_CONTROL_VERIFY_TLS`, `QDRANT_URL`, `X_API_KEY`, `CRM_DB_PATH`, `CRM_DEV_DB_PATH`. Also used: `CRM_SECRET_KEY` (beta/prod), `CRM_HOST`/`CRM_PORT`, `CRM_DATA_DIR`; digest mailer: `CRM_DIGEST_SENDER` (DWD impersonation sender) + `SMTP_HOST`/`SMTP_PORT`/`SMTP_SECURITY`/`SMTP_FROM`/`SMTP_USERNAME`/`SMTP_PASSWORD` (SMTP fallback); daily digest (Phase B): `CRM_DIGEST_ENABLED` + `CRM_DIGEST_SEND_HOUR` **only seed the first-boot default** — the live control is the DB policy (`app_settings.digest_policy`, set in Settings → Admin).
|
||
- **Config placement:** operational/feature toggles live in the **admin panel**, DB-backed via `app_settings` (read-merge through a `load_*_policy(conn)` helper shared by the API + any scheduler; precedence DB-row → env-seed → default), so they're discoverable and take effect live. Reserve StartOS actions / env for **secrets and deploy-time config** (SMTP creds, API keys, DWD sender). Precedent: `digest_policy` (`GET/PATCH /api/admin/digest/policy`), `fundraising_backup_policy`.
|
||
- **Agent/bot API access — three roles now (`admin`/`member`/`bot`).** `require_admin` is the only hard gate; everything else is "authenticated" (member, admin, *and* bot all pass). The **`bot` role** (added v0.1.0:89) is authenticated-but-never-admin: `require_bot_or_admin` gates agent-facing endpoints (e.g. `/api/intake/email-proposals*`) so a bot credential reaches *only* what it needs, never user-management/settings/security. Provision it via Settings → Admin edit-user dropdown (kept out of the teammate-invite form). **Two axes to keep separate as more agent capability lands:** the role controls *reach* (which endpoints); the per-feature human draft→approve gate controls *autonomy* (acting unattended). Money/merge/delete mutations stay behind the approval gate regardless of role. Don't build a finer capability/scope system until real NL-mutation endpoints exist to scope against.
|
||
- **Design:** before building or changing any user-facing UI, read `design/DESIGN.md` and `design/tokens.tokens.json` and conform to them. The **mobile-first redesign landed** (Claude Design round-trip distilled into the contract 2026-06-19): the authority for mobile/responsive work is **`DESIGN.md` §8** + the tokens `mobile` and `color.light` groups; `design/BRIEF.md` is the input brief and `design/_imports/2026-06-19/` the provenance + per-surface interaction reference (the comps are Claude Design runtime prototypes — re-author each surface in the app's React idiom + real API, not drop-in; **the design source of truth is each `*.dc.html` at its DEFAULT `data-props` (compact/dark/plex/earmark — see `GridApp.dc.html` `data-props`), NOT the `screenshots/` PNGs, which are option-history (rejected/stale combos: INVESTOR/PROSPECT disposition badges, 6-stage MEETING/FUNDED funnel, star flag). Don't anchor on the screenshots** (cost a re-scope 2026-06-19; general learning in `standards/guides/design.md` Phase C)). A **light theme is built (P6)**: it lives in `:root[data-theme="light"]` (set by a pre-paint boot script from `localStorage.venture_crm_theme`; dark is the default), with an app-wide toggle in the desktop sidebar footer + the mobile top bar. **Colors are theme vars now — any new UI color MUST use a `:root` var (grow the set if needed), never a literal, or it won't flip in light** (chips/badges flip via `.stage-chip--{stage}`/`.due-chip--{overdue,today,week,later}` + the `--chip-*`/`--note-*`/`--badge-priority-*`/`--rem-*`/`--due-{overdue,today,week,later}-*`/`--money`/`--recency-*`/`--due-soon` slots; authoritative dark+light pairs are in the Claude Design export `design/_imports/2026-06-19_zip-file/` `store.js` + `*App.dc.html`). Mobile light is complete; desktop has known unthemed shades (Phase 7). (Note: inline `style={{}}` objects can't respond to media queries; responsive layout belongs in the CSS `<style>` block. The **mobile foundation primitives are built** — CSS: `.bottom-tab-bar`, the `.bottom-sheet` primitive, `.mobile-only`/`.desktop-only`, `:root` mobile vars; React (Phase 2): **`<BottomSheet>`** (scrim/Escape/drag-to-dismiss) + **`useIsMobile()`** (768px) + the **`MobileDetailRow`**/`.fs-detail` full-screen-detail + `.contact-card`/`.az-header` list patterns — **build new mobile surfaces on these** (P3 Grid reuses them directly; swap surfaces via a rules-of-hooks-safe `useIsMobile()` wrapper that mounts a `Mobile*`/`Desktop*` pair, never a per-component hook toggle). The inline-style→CSS migration is **scoped, per-surface** (~114 styles across 4 surfaces+shell, not ~1,300), folded into each surface's build; see `ROADMAP.md`.) **Phase 8 card/detail primitives (reuse these, don't reinvent):** `EarmarkCorner` (existing-LP corner triangle; `inline` variant for the org card), the `priority-pill`/`lp-pill` text pills, `StageChip` (+ `sm`), `NoteTimeline`, `LogCommunicationSheet`, **`SortPill`/`SortSheet`** (mono pill + label+hint option sheet, 8d — drive with a per-surface `{id,pill,label,hint}` table, e.g. `GRID_SORTS`/`PIPELINE_SORTS`/`SORT_OPTIONS`), **`MobileQuickLog`** (shell top-bar quick-log pencil), **`DueChip`** (8e — urgency-bucketed reminder due pill, `.due-chip--{bucket}` colors); `<BottomSheet>` takes a **`stacked`** prop to layer a sheet opened over another sheet (e.g. the Log sheet over a detail, or the 8e investor-picker over the add sheet). **Mobile reminder writes link a real investor via `source_row_id` → server-resolved `investor_id`** (the 8e add-flow picker; create POSTs `source_row_id`, never a free-text name — the old label never linked); **`PATCH /api/reminders` can't reassign the investor** (edit shows it read-only). The mobile **Contacts + Pipeline detail surfaces are drag-dismiss bottom sheets** (8b) that log via **`POST /api/communications`**; the **Grid detail stays full-screen** (its dc default) and reads its **notes timeline** via **`GET /api/communications?source_row_id=<grid row id>`** (investor-level: maps grid row → `fundraising_investors.source_row_id` → `fundraising_contacts.contact_id` → comms, soft-delete-respecting). **The Contacts read path injects derived read-only `committed` + `pipeline_stage` + `priority`** (`contact_grid_signals()` — existing-LP ring + stage pill + Priority sort) on **both** `GET /api/contacts` and `/api/contacts/{id}`; this needs no strip-point (the directory is read-only, never written back as a row) — unlike the grid's injected columns.
|
||
- **Commit style:** imperative subject, concise body explaining the *why*; put the package version in the subject (`… (v0.1.0:NN)`) for shippable changes. **No AI co-author / attribution trailers** — commits are authored by the user.
|
||
|
||
## Always
|
||
|
||
- **Verify before shipping:** `python3 -m py_compile` the edited files; for DB logic, run the change against a **copy** of `data/crm.db`, never production.
|
||
- **Keep real LP data out of Claude:** develop only on code/schema/synthetic-or-locally-redacted data; route any real record substance through `backend/redaction` first.
|
||
- **Get explicit user authorization before any production deploy/install** to `$START9_BOX_HOST`.
|
||
|
||
## Never
|
||
|
||
- **Never treat Qdrant (or any derived index) as source of truth** — the CRM/SQLite is canonical and rebuildable-from.
|
||
- **Never hard-delete** CRM records or thesis history — soft-delete/archive only.
|
||
- **Never let an agent send email, post, or contact an LP autonomously** — agents draft; a human approves and sends.
|
||
- **Never set a `thesis_version` canonical from code/seeds** — that is human dual sign-off.
|
||
- **Never call a Spark directly** — go through Spark Control (`SPARK_CONTROL_URL`).
|
||
- **Never commit secrets, `data/crm.db`, `.env`, or `data/backups/`** (all gitignored). Scan staged files before committing. (`.claude/` *is* tracked — `launch.json` and `rules/` symlinks ship with the repo; keep local-only settings in `.claude/settings.local.json`.)
|
||
- **Never bulk-export the LP list** to any third party; send only minimal non-sensitive context to Claude.
|
||
- **Never assume FastAPI / SQLAlchemy / pytest** are in play — they sit in `requirements.txt` unused; runtime is stdlib + SQLite.
|
||
- **Never add a `Co-Authored-By` / "Generated with" trailer** to commits or PRs — commits are the user's.
|
||
|
||
## Deeper docs
|
||
|
||
- Full constitution + guardrails: `docs/ten31-constitution.md`
|
||
- Architecture & rationale: `docs/Ten31_Agentic_Build_Plan.md`
|
||
- Retrieval/embeddings contract: `docs/EMBEDDINGS.md`
|
||
- CRM schema/API tour: `docs/crm-overview.md`
|
||
- Current thesis handoff: `docs/thesis-handoff.md`
|
||
- Operations & runbooks: `docs/OPERATIONS.md`, `docs/go-live-runbook.md`, `docs/gmail-enablement-runbook.md`
|
||
|
||
## Current state
|
||
|
||
_**Box live at v0.1.0:94**; `main` ahead by mobile Phases 0–7 + P3b + drag-reorder + **8a + 8b + 8c + 8d + 8e** — **all deploy-pending** (no s9pk built). **The fundraising grid + email capture is the canonical system of record.** Active thread: **mobile-first redesign → Phase 8**, building to `design/phase8-conformance.md` (the 8a–8i spec, anchored on each `*.dc.html` DEFAULT `data-props` — NOT the `screenshots/` PNGs). **Plan (Grant, 2026-06-19): finish features first → then Grant device-tests + deploys** (nothing verified on a real phone). History: git log + `start9/0.4/startos/versions/`._
|
||
|
||
- **Mobile redesign — 4 core surfaces built (Grid · Contacts · Pipeline · Reminders), each a rules-of-hooks-safe `useIsMobile()` → `Mobile*`/`Desktop*` pair (desktop untouched).** Foundation: bottom-tab bar + `:root` mobile vars; 4-stage enum; derived grid signals injected-on-GET/stripped-on-write at both points; mobile writes use **one-row endpoints only** (log-communication, pipeline link/stage, reminders, `update-row`) — never whole-grid PUT.
|
||
- **Phase 8 done so far (8a–8e):** cards re-authored (existing-LP **earmark**/avatar **ring**, PRIORITY/Existing-LP pills, 4-stage chip, recency; disposition badges dropped); **Contacts + Pipeline detail → drag-dismiss bottom sheets** (email-copy pill, Log/Email, Org card, stat tiles, inline move-stage, notes timeline); **Grid-detail notes timeline** + shared `LogCommunicationSheet`; **top-bar quick-log pencil** (`MobileQuickLog`, all tabs); **sort controls** on Grid/Pipeline/Contacts (shared `SortPill`/`SortSheet`; Contacts type-tabs dropped + Priority sort); **8e Reminders re-bucketed** — Active/Done/All tabs dropped → title + urgency **summary line** + gradient add; **4 buckets** Overdue/Today/This-week/Later each with a colored **dot**; urgency-colored **`DueChip`** pill (`--due-*` theme vars) replaces plain due text; collapsible **Completed** section (done+cancelled, strikethrough, tap-check reopens); card now note + org + due-chip (assignee dropped from card, still in edit sheet); **swipe-right → snooze preset sheet** (+1/+3/+7/+14d, dc-style) replacing the old fixed +7d; **add-flow investor picker** — a stacked searchable sheet over the canonical grid investors that writes **`source_row_id` → a real server-resolved `investor_id` link** (replaces the old free-text label that never actually linked; "team task" = no investor). Per-phase detail in git log + the Design convention's primitives list.
|
||
- **Live (deployed):** W2 NL query (v94); W1 reminders (v93); grid Pipeline (v88); Matrix intake + Gmail capture (DWD) + daily digest; Thesis/Architect (dual-approval); outreach — all draft-only.
|
||
- **Tests:** **39/39 backend green** (`python3 backend/run_tests.py`; +`test_grid_comm_timeline.py` for the 8c timeline filter, +priority assertions in `test_contacts_grid_signals.py`), `py_compile` clean; 8c+8d+8e surfaces interaction-verified via throwaway 375px jsdom harnesses (deleted after). 8e was frontend-only — no backend touched, so the 39/39 count is unchanged (the reminders read path was untouched: the list endpoint with no `status` param already returns all statuses, which 8e now splits client-side).
|
||
- **Next — Phase 8, in order, build to `design/phase8-conformance.md`:** **8f** Pipeline card (earmark/Priority/recency + horizontal-scroll stage pills + dots) → **8g** add-investor stage+priority → **8h** loose ends (incl. Grid detail G4/G5/G6 stage-card/reminder-card/timeline; "Open-in-Grid" deep-link-to-investor) → **8i** shell SVG icons + `·Ten31·` wordmark. **Skip Pipeline accordion** (Grant). **Then (after feature-complete):** deploy P0–P8 + P3b in one s9pk (**authorize + version-bump first**) and device-test light/dark on a phone.
|
||
- **Open / risks:** all mobile work + light theme **built but never deployed or device-tested** (smoke/jsdom only); `MobileDetailRow` now unused-but-retained (legacy-usage sweep); Pipeline detail "Committed" tile shows grid-committed not deal-expected (deal forecast in a footnote); W2 happy-path only; **Claude/Architect path unverified live on the box**; v2.0 reserve-asset spine **not canonical**; doc drift — `crm-overview.md`/`EVALUATION.md` still call `lp_profiles` live.
|