ten31-database/AGENTS.md

# Ten31 Agentic System — AGENTS.md

In-house AI-agent system over a self-hosted Start9 CRM (SQLite) for a bitcoin/energy/AI investment fund: widen the fundraising funnel, sharpen the thesis, automate outreach. Frontier reasoning runs on Claude (Agent SDK/API); privacy-sensitive and bulk work runs on local DGX Spark models via the **Spark Control** gateway. **Phase 0/1 — no live outward-facing agents; agents draft, humans send.**

## Stack (versions that matter)

- **Python 3.11, standard library only at runtime.** The CRM is one monolith, `backend/server.py` (~5k lines): a stdlib `http.server.ThreadingHTTPServer` + hand-written `CRMHandler` with manual path dispatch (`do_GET`/`do_POST`). **Not FastAPI.** `backend/requirements.txt` lists FastAPI/SQLAlchemy/Alembic/Pydantic/pytest-style deps but **none are imported at runtime** (vestigial).
- **SQLite** at `data/crm.db` (WAL, `foreign_keys=ON`), opened per-request via `get_db()`. Schema via ordered migrations.
- **Frontend:** single `frontend/index.html`, inline-Babel React. **No build step.**
- Optional runtime deps, used only if present: `bcrypt`, `PyJWT` (`jwt`), `cryptography` (Gmail module).
- **MCP + ingest** (in the Docker image, not the bare CRM): `mcp==1.2.0` (FastMCP, `backend/mcp/server.py`), `fastembed==0.4.2`, `anthropic`, `cryptography==42.0.5`.
- **Packaging:** StartOS 0.4, TypeScript SDK (`@start9labs/start-sdk`) under `start9/0.4/startos/`. Live target is `start9/0.4/`.
- **Local models** (bge-m3 embeddings, bge-reranker-v2-m3, `/api/search`, Qdrant): always via Spark Control. Contract: `docs/EMBEDDINGS.md`.

## Commands

```bash
# Run locally (dev, port 8080; or ./start.sh <port>) — runs python3 backend/server.py
./start.sh
# Run prod-mode (Tailscale/beta) — requires CRM_SECRET_KEY
./start_beta.sh
# Sanity-check edits (there is no compiler/build for the CRM)
python3 -m py_compile backend/server.py
# Run ONE test (tests are standalone scripts with `if __name__ == "__main__"`; no pytest installed)
python3 backend/redaction/test_scrub_leak.py        # substitute any backend/**/test_*.py (13 exist)
# Run all tests (no aggregate runner exists)
for t in $(find backend -name 'test_*.py'); do echo "== $t"; python3 "$t" || break; done
# Build the s9pk (x86_64 only) -> ten-database_x86_64.s9pk  — BUMP THE VERSION FIRST (see Always)
cd start9/0.4 && make
# Install to the box — PRODUCTION; get explicit user OK first. TODO: confirm exact host/context.
start-cli package install -s ten-database_x86_64.s9pk   # target: immense-voyage.local
```

- **Migrations** apply automatically at startup via `backend/core_migrations.py` from `backend/migrations/NNNN_*.sql`, tracked in a `schema_migrations` ledger. Verify a new one against a **copy** of `data/crm.db`, never production.
- **Lint:** none configured.

## Directory layout (day-one)

- `backend/server.py` — the CRM monolith: HTTP handler, route dispatch, `init_db()`, auth (username/password → HS256 JWT, roles admin/member).
- `backend/core_migrations.py` + `backend/migrations/NNNN_*.sql` (+ paired `.down.sql`) — additive schema migrations, applied at startup.
- `backend/thesis_seed.py` — Thesis Workshop seed + idempotent `ensure_*` one-time seeders (interaction_log sentinels), wired in `server.init_db()`.
- `backend/thesis_review.py` — thesis version review/approval (human dual sign-off → canonical).
- `backend/mcp/` — `architect_agent.py` (Claude thesis copilot), `architect_tools.py` (thesis CRUD/versions), `outreach_agent.py` (LP draft assistant), `architect_grounding.py`, `crm_tools.py`, `server.py` (FastMCP).
- `backend/email_integration/` — Gmail capture via domain-wide delegation: `credentials.py`, `matcher.py`, `parser.py`, `db.py`, `sync.py`, `scheduler.py`, `routes.py`, `compose.py` (Tier-B draft creation), `migrations/`.
- `backend/redaction/` — `scrub.py` + `client.py`: the scrub→Claude→re-hydrate privacy boundary (`Boundary`, `SCRUB_BACKEND=local|gateway`, fail-closed).
- `backend/ingest/` — chunk→embed→Qdrant + retrieval modes (`search.py`, `embed.py`, `qdrant_io.py`, `sparse.py`, `entity_resolution.py`).
- `backend/entity_*.py` — entity resolution/merge (the two-investor-model reconciliation).
- `frontend/index.html` — the entire UI.
- `docs/` — `Ten31_Agentic_Build_Plan.md` (architecture), `PHASE_0.md`/`PHASE_1.md`, `EMBEDDINGS.md` (retrieval contract), `crm-overview.md` (schema/API tour), `thesis-handoff.md`, `ten31-constitution.md` (full constitution + guardrails).
- `start9/0.4/` — StartOS package: `startos/utils.ts` (`PACKAGE_VERSION`), `startos/versions/`, `Dockerfile`, `docker_entrypoint.sh`, `Makefile`, `s9pk.mk`.
- `data/crm.db` — the live DB (gitignored). `.env` / `.env.example` — config (`.env` gitignored).

## Conventions

- **Two coexisting investor models** (classic `contacts`/`lp_profiles` + the `fundraising_*` grid). Reconciling them to canonical IDs is the core entity-resolution task — see `docs/crm-overview.md`.
- **Migrations are additive + reversible only:** numbered `NNNN_*.sql` with a paired `NNNN_*.down.sql`. SQLite ALTER = add-column/rename only.
- **One-time seeds/backfills are idempotent** via `interaction_log` sentinels (the `ensure_*` pattern), wired into `init_db` — safe to re-run on every boot.
- **Soft-delete only:** `deleted_at` and/or `status='retired'`; never hard-delete. `_node_tree` and `create_thesis_version` filter on `deleted_at IS NULL` and **ignore status** — so to drop a node from the live agent prompt AND version snapshots you must set `deleted_at`, not just status.
- **Thesis canonical gate:** node status is `draft|candidate|approved|retired` (the working tree); a canonical `thesis_version` is frozen ONLY by human **dual** sign-off (`thesis_review`). Code/seeds never set a version canonical.
- **Env:** secrets in `.env` (gitignored); names in `.env.example`. Verified names: `ANTHROPIC_API_KEY`, `SPARK_CONTROL_URL`, `SPARK_CONTROL_VERIFY_TLS`, `QDRANT_URL`, `X_API_KEY`, `CRM_DB_PATH`, `CRM_DEV_DB_PATH`. Also used: `CRM_SECRET_KEY` (beta/prod), `CRM_HOST`/`CRM_PORT` (`start.sh`), `CRM_DATA_DIR`.
- **Commit style:** imperative subject, concise body explaining the *why*; put the package version in the subject (`… (v0.1.0:NN)`) for shippable changes. **No AI co-author / attribution trailers** — commits are authored by the user. (Older history carries a `Co-Authored-By: Claude` trailer; dropped going forward.)

## Always

- **Bump the version before building an s9pk:** edit `PACKAGE_VERSION` in `start9/0.4/startos/utils.ts`, add `start9/0.4/startos/versions/v0.1.0.NN.ts`, and register it in `versions/index.ts` (import, set `current`, move prior `current` into `other[]`). Start9 0.4.x ignores a same-version rebuild.
- **Verify before shipping:** `python3 -m py_compile` the edited files; for DB logic, run the change against a **copy** of `data/crm.db`.
- **Make migrations/seeders deployment-state-invariant and idempotent:** target rows **structurally**, not by transient text the same change mutates; capture prior state so a revert is exact. (Learned the hard way: matching old nodes by a body string the same changeset deleted broke fresh DBs.)
- **Keep real LP data out of Claude:** develop only on code/schema/synthetic-or-locally-redacted data; route any real record substance through `backend/redaction` before it reaches a Claude model.
- **Get explicit user authorization before any production deploy/install** to `immense-voyage.local`.
- **Ship a paired `.down.sql`** with every new migration.

## Never

- **Never treat Qdrant (or any derived index) as source of truth** — the CRM/SQLite is canonical and rebuildable-from.
- **Never hard-delete** CRM records or thesis history — soft-delete/archive only.
- **Never let an agent send email, post, or contact an LP autonomously** — agents draft; a human approves and sends.
- **Never set a `thesis_version` canonical from code/seeds** — that is human dual sign-off.
- **Never call a Spark directly** — go through Spark Control (`SPARK_CONTROL_URL`).
- **Never commit secrets, `data/crm.db`, `.env`, backups, or `.claude/`** (all gitignored). Scan staged files before committing.
- **Never bulk-export the LP list** to any third party; send only minimal non-sensitive context to Claude.
- **Never assume FastAPI / SQLAlchemy / pytest** are in play — they sit in `requirements.txt` unused; runtime is stdlib + SQLite.
- **Never add a `Co-Authored-By` / "Generated with" trailer** to commits or PRs — commits are the user's.

## Deeper docs

- Full constitution + guardrails: `docs/ten31-constitution.md` — TODO: consider folding its still-current content into this file and retiring the separate doc.
- Architecture & rationale: `docs/Ten31_Agentic_Build_Plan.md`
- Retrieval/embeddings contract: `docs/EMBEDDINGS.md`
- CRM schema/API tour: `docs/crm-overview.md`
- Current thesis handoff: `docs/thesis-handoff.md`