ten31-database/ROADMAP.md

# Venture CRM Roadmap (Airtable Replacement)

## Current status
- Premium Airtable-like frontend grid exists and is actively iterating.
- Backend now has production-grade APIs for:
  - `GET /api/fundraising/state`
  - `PUT /api/fundraising/state` (with optimistic version check)
  - `GET /api/fundraising/export`
  - `POST /api/fundraising/backup`
  - `POST /api/fundraising/restore-preview`
  - `POST /api/fundraising/restore`
  - `GET /api/fundraising/backups`
  - `GET/PATCH /api/fundraising/backup-policy`
  - `GET /api/fundraising/relational-summary`
  - `GET /api/feature-requests`
  - `POST /api/feature-requests`
  - `PATCH /api/feature-requests/:id`
- New DB tables:
  - `fundraising_state`
  - `fundraising_investors`
  - `fundraising_contacts`
  - `fundraising_funds`
  - `fundraising_commitments`
  - `fundraising_views`
  - `feature_requests`
  - `app_settings`
- Grid saves/restores now sync into relational fundraising tables automatically.
- Formula engine is now sandboxed (no `eval`/`new Function`) with expanded function support.
- Automation engine v1 added:
  - Rule table + toggle API
  - List memberships (`main`, `follow_up`, `graveyard`, `longshot`, `all`)
  - Automation run log
- Collaboration/reliability additions:
  - Unified activity feed API (`audit` + `automation` + `backup`)
  - Backup integrity verification API
  - Better version-conflict metadata (`updated_at`, `updated_by`)
- Security hardening additions:
  - Basic IP rate limiting (login and write APIs)
  - Configurable CORS origin (`CRM_CORS_ORIGIN`)
  - Production secret enforcement (`CRM_ENV=production` requires `CRM_SECRET_KEY`)
  - Security status API + go-live checklist (`SECURITY.md`)

## Phase 1 (Production foundation)
1. Persist grid + views on backend
- Wire frontend fundraising grid reads/writes to `/api/fundraising/state`.
- Keep localStorage only as emergency fallback.
- Add autosave debounce and conflict handling (`expected_version`).

2. Admin-invite auth model
- Disable self-register for non-admin users.
- Add admin-only invite/create-user endpoint.
- Keep role model: `admin`, `member`.

3. Deployment and remote access
- Add `docker-compose` for one-command launch.
- Reverse proxy + TLS option (Caddy/Traefik) for non-Tailscale deployments.
- Recommended for your use case: Tailscale private access to laptop host.

4. Data safety and operations
- Automated nightly SQLite backups and restore test script.
- Add `/api/fundraising/export` for JSON snapshot export.
- Add health/readiness checks.

## Phase 2 (Airtable parity)
1. Advanced views
- Multi-condition filter groups (AND/OR groups)
- Multi-column sorting
- Pinned/frozen columns
- Personal vs shared views

2. Formula engine v2
- Add functions: `SUM`, `MIN`, `MAX`, `ROUND`, `ABS`, `CONCAT` (done)
- Type-aware formulas and better errors
- Dependency graph and recalculation rules

3. Activity + audit
- Record-level change history in UI
- Last modified by / at fields
- Restore archived rows

## Phase 3 (Team workflow and automation)
1. Tasks/reminders tied to investors/contacts
2. Automation rules (graveyard/follow-up triggers)
3. Email/communication integrations (optional)
4. Granular permissions (if team grows)

## Backlog (post-Phase-1 agentic)

### Daily activity digest (email to the team)
*Requested 2026-06-15. **Phase A built + deployed** (v0.1.0:75 live on the box; send path then moved to **Gmail DWD** in v0.1.0:76, redeploy pending). Phase B (digest content + Spark summarization + daily scheduler) remains.*

**Decisions (locked 2026-06-15):** recipients = **all active admins**; summarization = **Spark-LLM narrative** (never Claude — un-anonymized substance stays local); granularity = **grouped by user** (→ per investor).

**Send transport — DECIDED 2026-06-15: Gmail domain-wide delegation** (not SMTP). The box's existing service-account grant (which powers email capture) **includes `gmail.compose`**, which authorizes `users.messages.send` — verified by a token-mint probe **and a live `messages.send` to grant**. So the digest sends through the account the CRM already uses: **no app password, no new account, no admin change.** The narrow `gmail.send` scope is *not* granted, so the sender must request `gmail.compose`.

**Phase A — DONE:** (v0.1.0:75) `configureDigestSmtp` Start9 action + `docker_entrypoint.sh` `SMTP_*` export + `backend/smtp_send.py` + admin `POST /api/admin/digest/test-email` (recipient-restricted to the admin set — not an open relay) + Settings button. (v0.1.0:76, redeploy pending) `backend/email_integration/gmail_send.py` (`users.messages.send` via DWD/compose) + `backend/digest_mailer.py` (**Gmail-DWD preferred, SMTP fallback**); the endpoint + button route through it; sender = `CRM_DIGEST_SENDER` else first active admin. Tests: `test_smtp_send.py`, `test_smtp_endpoint.py`, `test_gmail_send.py`.

**Phase B — TODO:** daily scheduler (co-locate with `email_integration/scheduler.py`); per-user→per-investor activity query (`deleted_at IS NULL` throughout); Spark-narrative summary of captured email substance; compose + send to all admins.

Have the CRM send a **daily digest email** summarizing each registered user's activity — primarily **who emailed which investors and the substance of those emails** — to the fund principal (and eventually other admins). Scales with the synced-user count: 2 users synced today, ~5 eventually.

- **Source data:** the captured email-activity already flowing through the Gmail DWD propose→approve pipeline (`backend/email_integration/`), keyed per registered user → per investor/contact. Optionally fold in other CRM activity (audit feed, automation runs, new opportunities) later.
- **Send path is NEW capability.** Today nothing leaves the box — the system only *captures* Gmail and *creates drafts*. This needs outbound SMTP. StartOS 0.4 has a system-wide SMTP account (since v0.4.0-beta.9): the user configures it once for the whole server and services read it via `sdk.getSystemSmtp(effects).const()`, which returns a `T.SmtpValue` (`host`, `port`, `from`, `username`, `password`, `security`). Wire the digest sender to that rather than hardcoding any account. *Implementation path (researched 2026-06-15, our SDK pin `^0.4.0-beta.66`):* model a `manageSMTP` action on [gitea-startos](https://github.com/Start9Labs/gitea-startos) / [vaultwarden-startos](https://github.com/Start9Labs/vaultwarden-startos) — a three-way `selection` (system / custom / disabled) built on `sdk.inputSpecConstants.smtpInputSpec`, persisted to `storeJson`, with `main.ts` injecting `SMTP_HOST/PORT/USER/PASS/FROM/SECURITY` env vars into the daemon `exec` block (same shape as the existing `setAnthropicApiKey.ts` action). The Python sender reads them via `os.environ` and opens `smtplib.SMTP`/`SMTP_SSL`. **"Custom SMTP" is a dedicated per-package account, fully independent of the server's system SMTP** — the custom branch never calls `getSystemSmtp`, so the digest can send through its own provider even on a box with no system account configured (confirmed in both reference packages). This is the likely fit here: a digest-only mailbox separate from anyone's Gmail. Note StartOS 0.4 dropped the old `Config`/`Properties` manifest spec — SMTP config is an **action + storeJson**, not a manifest config field. **SDK note (verified 2026-06-15):** our pin `^0.4.0-beta.66` resolves to exactly `0.4.0-beta.66` (caret on a prerelease stays within the `0.4.0` tuple), whose SMTP surface — `getSystemSmtp` → `T.SmtpValue {host, port, from, username, password, security}`, `inputSpecConstants.smtpInputSpec` (providers gmail/ses/sendgrid/mailgun/protonmail/other; selection disabled/system/custom), `smtpShape`, `smtpPrefill` — is **byte-identical** to the 1.5.3 reference packages (verified from published tarballs; repo `node_modules` is absent). Build against beta.66 as-is — **no SDK bump needed** (moving to 1.x is a major-track change with broad blast radius across `startos/`, and nothing about SMTP justifies it).
- **Analysis runs on Spark, never Claude.** The digest is deliberately **un-anonymized** (real LP names + email substance), so any summarization/analysis must go through **Spark Control to local models** — this is the one path that intentionally bypasses the scrub→Claude→re-hydrate boundary, because keeping the substance local is the whole point. Never route digest content to Claude.
- **Exempt from "agents draft, humans send."** That rule governs outward LP/prospect contact. This is an internal ops digest to the team's own inboxes — a different category — so an automated daily send here does not violate the draft-only guardrail. State this explicitly at build time.
- **Scheduling:** a daily cron, naturally co-located with the existing `backend/email_integration/scheduler.py` sync cadence.
- **Soft-delete:** every aggregate/read in the digest must filter `deleted_at IS NULL` (see the standing soft-delete rule).

Open design questions (Phase B detail, still to settle): fixed daily send time; "nothing happened today" suppression; whether the Spark summary is per-investor-thread or a single per-user narrative.

## Definition of done for "Airtable substitute" v1
- Team can manage all investors in one master table
- Saved views replicate current Airtable workflows
- CSV import from Airtable is reliable and repeatable
- Data persists safely and supports multi-user access
- Auth is invite-only and backups are automated