ten31-database/AGENTS.md

Ten31 Venture CRM + Agentic System — AGENTS.md

The foundation is a self-hosted venture-fund CRM — a purpose-built fundraising tool that replaced Airtable to (1) keep sensitive LP/prospect data off third-party servers, (2) drop subscription cost, and (3) fit the fund's workflow: managing ~150 existing LPs, tracking 250+ prospects, and running the capital-raise pipeline. Core CRM domain: contacts (investor/prospect/advisor), organizations, opportunities (the deal pipeline), and communications; investor commitments live in the canonical fundraising_* grid (the legacy single-fund lp_profiles table was retired in v0.1.0:78). The fund (Ten31, ~$200M AUM, bitcoin/energy/AI thesis) runs it on a Start9 box, accessed over ClearNet (StartOS StartTunnel) with app-level user auth by a team of ~5 (Tailscale is not in use). Schema/API tour: docs/crm-overview.md.

The agentic system is new functionality built on top of that CRM — an in-house AI layer to widen the fundraising funnel, sharpen the thesis, and automate outreach drafting. Frontier reasoning runs on Claude (Agent SDK/API); privacy-sensitive and bulk work runs on local DGX Spark models via the Spark Control gateway. Phase 0/1 — no live outward-facing agents; agents draft, humans send.

Inbox check: At session start, if ~/Projects/standards/INBOX.md exists, scan it for items tagged (CRM) and surface them before proposing next steps; triage with /triage.

Stack (versions that matter)

• Python 3.11, standard library only at runtime. The CRM is one monolith, backend/server.py (~5k lines): a stdlib http.server.ThreadingHTTPServer + hand-written CRMHandler with manual path dispatch (do_GET/do_POST). Not FastAPI. backend/requirements.txt lists FastAPI/SQLAlchemy/Alembic/Pydantic/pytest-style deps but none are imported at runtime (vestigial).
• SQLite at data/crm.db (WAL, foreign_keys=ON), opened per-request via get_db(). Schema via ordered migrations.
• Frontend: single frontend/index.html, inline-Babel React. No build step.
• Optional runtime deps, used only if present: bcrypt, PyJWT (jwt), cryptography (Gmail module).
• MCP + ingest (in the Docker image, not the bare CRM): mcp==1.2.0 (FastMCP, backend/mcp/server.py), fastembed==0.4.2, anthropic, cryptography==42.0.5.
• Packaging: StartOS 0.4, TypeScript SDK (@start9labs/start-sdk) under start9/0.4/startos/. Live target is start9/0.4/.
• Local models (bge-m3 embeddings, bge-reranker-v2-m3, /api/search, Qdrant): always via Spark Control. Contract: docs/EMBEDDINGS.md.

Commands

# Run locally (dev, port 8080; or ./start.sh <port>) — runs python3 backend/server.py
./start.sh
# Run prod-mode (beta) — requires CRM_SECRET_KEY
./start_beta.sh
# Sanity-check edits (there is no compiler/build for the CRM)
python3 -m py_compile backend/server.py
# Run ONE test (tests are standalone scripts with `if __name__ == "__main__"`; no pytest installed)
python3 backend/redaction/test_scrub_leak.py        # substitute any backend/**/test_*.py
# Run all tests (aggregate runner — runs each backend/**/test_*.py in its own subprocess)
python3 backend/run_tests.py                         # add substrings to filter, e.g. `... soft_delete redaction`
# Build + install the s9pk — BUMP THE VERSION FIRST. See docs/guides/packaging.md.
cd start9/0.4 && make

• Migrations apply automatically at startup (backend/core_migrations.py, schema_migrations ledger). See docs/guides/migrations.md before adding one.
• Lint: none configured.

Directory layout (day-one)

• backend/server.py — the CRM monolith: HTTP handler, route dispatch, init_db(), auth (username/password → HS256 JWT, roles admin/member/bot).
• backend/core_migrations.py + backend/migrations/NNNN_*.sql (+ paired .down.sql) — additive schema migrations, applied at startup.
• backend/thesis_seed.py — Thesis Workshop seed + idempotent ensure_* one-time seeders, wired in server.init_db().
• backend/thesis_review.py — thesis version review/approval (human dual sign-off → canonical).
• backend/mcp/ — architect_agent.py (Claude thesis copilot), architect_tools.py, outreach_agent.py (LP draft assistant), architect_grounding.py, crm_tools.py, server.py (FastMCP).
• backend/email_integration/ — Gmail capture via domain-wide delegation + Tier-B draft creation (compose.py).
• backend/redaction/ — scrub.py + client.py: the scrub→Claude→re-hydrate privacy boundary.
• backend/ingest/ — chunk→embed→Qdrant + retrieval modes.
• backend/entity_*.py — entity resolution/merge (the two-investor-model reconciliation).
• backend/matrix_intake/ — Matrix intake bot (separate process; matrix-nio, isolated to this component): typed message → local-Qwen parse → in-thread approve → write via the CRM's own log-communication. See the matrix-intake guide.
• frontend/index.html — the entire UI.
• docs/ — architecture, phase plans, contracts, runbooks (see Deeper docs). docs/guides/ — scoped subsystem rules (see below).
• start9/0.4/ — StartOS package (startos/utils.ts holds PACKAGE_VERSION).
• data/crm.db — the live DB (gitignored). .env / .env.example — config (.env gitignored).

Scoped guides

Subsystem rules live in docs/guides/ and lazy-load in Claude Code via .claude/rules/ symlinks (scoped by paths: frontmatter). Read the guide before editing that area:

• Migrations or seeders (backend/migrations/, core_migrations.py, thesis_seed.py) → docs/guides/migrations.md
• Thesis logic (backend/thesis_*.py, backend/mcp/architect_*.py) → docs/guides/thesis.md
• Redaction or any MCP/Claude path (backend/redaction/, backend/mcp/) → docs/guides/redaction.md
• Ingest / retrieval (backend/ingest/) → docs/guides/spark-ingest.md
• Email capture / drafts + digest send (backend/email_integration/, backend/digest_mailer.py, backend/smtp_send.py) → docs/guides/email.md
• Building or deploying the s9pk (start9/) → docs/guides/packaging.md
• Matrix intake bot (backend/matrix_intake/) → docs/guides/matrix-intake.md

Conventions

• Investor model — the grid is canonical (since v0.1.0:78). The fundraising_* grid is the system of record: an investor entity (row) → many contact "pills" → per-fund commitments. The classic contacts table is a read-only per-person directory, auto-populated from the grid — create/edit people in the grid, not the Contacts page. Email capture rolls multiple people up to one investor. The legacy single-fund lp_profiles model is retired (empty table kept, per never-hard-delete). Reconciling grid ↔ classic contacts to canonical IDs is the core entity-resolution task — see docs/crm-overview.md.
• Soft-delete only: deleted_at and/or status='retired'; never hard-delete. Every READ path must filter deleted_at IS NULL — list handlers, get-by-id, nested related-data sub-selects, and aggregate sub-selects (COUNT/SUM/MAX). Audits found leaks in all of these (2026-06-12 detail + nested; 2026-06-13 list-view contact_count/total_funded/comm_count); the opportunities/pipeline aggregates were fixed in v0.1.0:87 (handle_pipeline_report + dashboard pipeline metrics now filter deleted_at), but the reports subsystem's communications-side aggregates (dashboard recent_comms/comms_this_month/meetings_this_month, activity report) still leak (see Current state). Regression-guarded by backend/test_soft_delete_reads.py. (Thesis has a subtlety here — see the thesis guide.)
• Env: secrets in .env (gitignored); names in .env.example. Verified names: ANTHROPIC_API_KEY, SPARK_CONTROL_URL, SPARK_CONTROL_VERIFY_TLS, QDRANT_URL, X_API_KEY, CRM_DB_PATH, CRM_DEV_DB_PATH. Also used: CRM_SECRET_KEY (beta/prod), CRM_HOST/CRM_PORT, CRM_DATA_DIR; digest mailer: CRM_DIGEST_SENDER (DWD impersonation sender) + SMTP_HOST/SMTP_PORT/SMTP_SECURITY/SMTP_FROM/SMTP_USERNAME/SMTP_PASSWORD (SMTP fallback); daily digest (Phase B): CRM_DIGEST_ENABLED + CRM_DIGEST_SEND_HOUR only seed the first-boot default — the live control is the DB policy (app_settings.digest_policy, set in Settings → Admin).
• Config placement: operational/feature toggles live in the admin panel, DB-backed via app_settings (read-merge through a load_*_policy(conn) helper shared by the API + any scheduler; precedence DB-row → env-seed → default), so they're discoverable and take effect live. Reserve StartOS actions / env for secrets and deploy-time config (SMTP creds, API keys, DWD sender). Precedent: digest_policy (GET/PATCH /api/admin/digest/policy), fundraising_backup_policy.
• Agent/bot API access — three roles now (admin/member/bot). require_admin is the only hard gate; everything else is "authenticated" (member, admin, and bot all pass). The bot role (added v0.1.0:89) is authenticated-but-never-admin: require_bot_or_admin gates agent-facing endpoints (e.g. /api/intake/email-proposals*) so a bot credential reaches only what it needs, never user-management/settings/security. Provision it via Settings → Admin edit-user dropdown (kept out of the teammate-invite form). Two axes to keep separate as more agent capability lands: the role controls reach (which endpoints); the per-feature human draft→approve gate controls autonomy (acting unattended). Money/merge/delete mutations stay behind the approval gate regardless of role. Don't build a finer capability/scope system until real NL-mutation endpoints exist to scope against.
• Commit style: imperative subject, concise body explaining the why; put the package version in the subject (… (v0.1.0:NN)) for shippable changes. No AI co-author / attribution trailers — commits are authored by the user.

Always

• Verify before shipping: python3 -m py_compile the edited files; for DB logic, run the change against a copy of data/crm.db, never production.
• Keep real LP data out of Claude: develop only on code/schema/synthetic-or-locally-redacted data; route any real record substance through backend/redaction first.
• Get explicit user authorization before any production deploy/install to $START9_BOX_HOST.

Never

• Never treat Qdrant (or any derived index) as source of truth — the CRM/SQLite is canonical and rebuildable-from.
• Never hard-delete CRM records or thesis history — soft-delete/archive only.
• Never let an agent send email, post, or contact an LP autonomously — agents draft; a human approves and sends.
• Never set a thesis_version canonical from code/seeds — that is human dual sign-off.
• Never call a Spark directly — go through Spark Control (SPARK_CONTROL_URL).
• Never commit secrets, data/crm.db, .env, or data/backups/ (all gitignored). Scan staged files before committing. (.claude/ is tracked — launch.json and rules/ symlinks ship with the repo; keep local-only settings in .claude/settings.local.json.)
• Never bulk-export the LP list to any third party; send only minimal non-sensitive context to Claude.
• Never assume FastAPI / SQLAlchemy / pytest are in play — they sit in requirements.txt unused; runtime is stdlib + SQLite.
• Never add a Co-Authored-By / "Generated with" trailer to commits or PRs — commits are the user's.

Deeper docs

• Full constitution + guardrails: docs/ten31-constitution.md
• Architecture & rationale: docs/Ten31_Agentic_Build_Plan.md
• Retrieval/embeddings contract: docs/EMBEDDINGS.md
• CRM schema/API tour: docs/crm-overview.md
• Current thesis handoff: docs/thesis-handoff.md
• Operations & runbooks: docs/OPERATIONS.md, docs/go-live-runbook.md, docs/gmail-enablement-runbook.md

Current state

Phase 0 + Phase 1 built; box live at v0.1.0:91; repo at v0.1.0:92 (v92 = reminders/follow-ups — built + tested locally 2026-06-18, deploy pending. Box deployed & verified live 2026-06-18 — installed-version=0.1.0:91, server up on :8080, clean; the StartOS version-graph traversal logs an inert down-to-39-then-up because the per-version up/down hooks are no-ops — real SQLite migrations run in-app at startup). The fundraising grid + email capture is the canonical system of record. Deploy/feature history: git log + start9/0.4/startos/versions/; longer-term backlog/debt: ROADMAP.md / EVALUATION.md.

• Reminders & follow-ups (W1) — BUILT + tested locally 2026-06-18 (repo v0.1.0:92, deploy pending). First step of the agreed reminders → NL-search → bot-mutations plan (ROADMAP.md "Follow-ups/reminders + NL search + bot grid-mutations"; overarching constraint: keep LP data off third-party LLMs — the dominant risk, above write-safety). First-class tickler tied to the grid: reminders table (in-app migration 0006; logical FK to fundraising_investors.id + denormalized name, like 0005), full CRUD (GET/POST/PATCH/DELETE /api/reminders; soft-delete; open/done/snoozed/cancelled; assignee; source human/bot/automation; accepts source_row_id so the grid stays decoupled), a read-only derived reminder_status grid column (overdue/due_soon/open — injected + stripped like pipeline_stage; filterable so a saved view can later drive the follow-up view off real reminders, not the binary follow_up checkbox), an orphan reconciler (reconcile_grid_reminders), a Reminders page + Dashboard "Reminders Due" card + "Reminders due" daily-digest section, and a per-investor last_activity_at recency rollup (shared building block for the W2 NL "not nurtured" query). Pure local CRM — no LLM path, no leak surface. Tests: test_reminders.py + digest reminders test (31/31 green, render-smoke green). Deferred fast-follow W1b = nurture-gap auto-suggested reminders. Next per plan: W2 NL→safe-query (web + Matrix), then W3 bot grid-mutations behind a Matrix approval gate (any member approves; money is low-stakes here — the concern is the bot can't silently mass-change numbers).

• Email-proposal review over Matrix + a bot role — DEPLOYED, LIVE & smoke-tested 2026-06-18 (box v0.1.0:91, Spark bot b2690c4). The CRM-drafted "proposed grid notes" gain: (1) a click-to-view inline source-email popup on the Email Capture page (GET /api/email/detail — from/to/cc/date/subject + scrollable body); (2) a CRM→Matrix review bridge — the bot pulls pending proposals (GET /api/intake/email-proposals), posts dash-framed review cards (note names who emailed whom, not "Sent/Received") to a dedicated room (MATRIX_EMAIL_REVIEW_ROOM), and relays in-thread yes/no/NL-edit (POST .../decide), kept in sync with the web panel (decide on either → the other reflects it). Decided threads are redacted whole (card + replies; the bot holds a redact/mod power level) so — with Element's "show deleted messages" OFF — the main chat and threads view clear completely (confirmed the intended UX). New bot role (authenticated, never admin; require_bot_or_admin) gates the agent endpoints; state in email_proposal_matrix (email-migration 0003). Full mechanics, deploy gotchas, and the redact_resolved.py backfill tool: docs/guides/matrix-intake.md.

• Adopt the Pipeline — LIVE & smoked (v0.1.0:88): the grid drives the deal board. "Add to Pipeline" row action → durably-linked opportunities row via opportunities.fundraising_investor_id (migration 0005); grid owns link+seed, board owns stage/probability/owner; "Remove" soft-deletes the opp, grid row intact; deleting a grid investor archives its orphaned opp. Detail + locked decisions: ROADMAP.md "Adopt the Pipeline".

• Matrix intake bot — LIVE (Spark, container matrix-intake, bot b2690c4): typed Matrix message → local-Qwen parse → in-thread human approval → write via POST /api/fundraising/log-communication (source="matrix_intake"); new-vs-existing via GET /api/intake/match. Fuzzy match + numbered disambiguation shortlist, conversational NL revise (email-integrity preserved), and the INTAKE_TEAM_ROSTER outreach frame all shipped & mostly smoked. One path still un-smoked in-room: the fuzzy disambiguation numbered-pick grammar. Runs as a docker-compose service (restart: unless-stopped). Guide: docs/guides/matrix-intake.md.

• Working (all draft-only): CRM + ingest (chunk→embed→Qdrant + retrieval) + redaction boundary; Gmail capture (DWD) + email-activity propose→approve; Thesis Workshop + Architect (Claude) with dual-approval gate; Outreach Draft Assistant + follow-up radar + per-user voice + Tier-B in-thread Gmail draft creation.

• Tests: 30/30 backend green (python3 backend/run_tests.py), py_compile clean; frontend render-smoke gates the default make build.

• Debt (P2, not deploy-blocking; full list EVALUATION.md): reports-subsystem soft-delete sweep — pipeline/opportunities aggregates fixed v87; remaining: the dashboard communications aggregates (recent_comms/comms_this_month/meetings_this_month) + activity report + report-endpoint tests; ?limit=abc crashes the request thread; auth regression test for the 3 v79-gated GETs (/api/users, /api/email/status, /api/email/accounts); scrub-gateway TLS verify off; hardcoded Spark/Qdrant IPs + oversized StartOS package icon (fix before the next s9pk upload); the 5.4k-line server.py monolith.

• Open / risks: the v2.0 reserve-asset spine is the working approved spine but not a canonical thesis_version (needs Grant + Jonathan dual sign-off; Appendix-A conviction incl. ~40% Strike stays Grant's working read, not fed to the engine); Claude/Architect path still unverified live on the box; the intake matcher reads only the grid blob (not classic contacts); doc drift — crm-overview.md + EVALUATION.md still call lp_profiles live (doc-auditor pass).

• Next: 1) spark-control intake dashboard card (separate session in the spark-control repo — handoff at docs/handoffs/add-intake-bot-to-spark-control.md), and longer-term extract the bot to its own repo (ROADMAP); 2) in-room smoke of the intake disambiguation numbered-pick grammar (the one unexercised path) — and a roster-tuning pass if any teammate name/initial still slips through; 3) NL→safe-query (search item 3 — separate, larger build); 4) Grant + Jonathan freeze v2.0 canonical; 5) reply-all for Tier-B drafts; then clear the P2 debt (reports comms-aggregate soft-delete sweep, ?limit=abc crash, auth regression test, oversized StartOS icon, etc.). Possible follow-ups if wanted: email-review — a since-floor on to_post if a large proposal backlog ever needs throttling; Pipeline — drag-and-drop stage moves, surface expected_amount × probability weighting.