7b560c97b6
Phase C/D of the /design round-trip (Claude Design "Venture-CRM mobile redesign", 2026-06-19). Captures the cloud output and folds it into the durable design/ contract; no frontend reskin in this pass. - _imports/2026-06-19/: provenance — GridApp.dc.html (byte-exact canonical surface) + a manifest README (project URL/inventory, data model, derived- field formulas, per-surface interaction model). DesignSync can't bulk- download, so screenshots/other sources stay recoverable from the cloud URL. - DESIGN.md: §8 Responsive rewritten to the landed mobile-first system (4-tab bottom bar, card/detail, bottom sheets, swipe/snap, safe areas); §4 mobile component states; §3 15px mobile type scale; §2 stage/staleness + light-theme palette pointers. - tokens.tokens.json: new `mobile` group (type scale, radii, touch sizing, safe-area) + `motion.sheet`; `color.light` palette — light theme adopted as a planned, toggle-gated feature (dark stays default). - ROADMAP.md: mobile-first implementation backlog (contract-vs-code gap), gated on the inline-style->CSS migration and the locked pipeline spec.
325 lines
57 KiB
Markdown
325 lines
57 KiB
Markdown
# Venture CRM Roadmap (Airtable Replacement)
|
||
|
||
## Current status
|
||
- Premium Airtable-like frontend grid exists and is actively iterating.
|
||
- Backend now has production-grade APIs for:
|
||
- `GET /api/fundraising/state`
|
||
- `PUT /api/fundraising/state` (with optimistic version check)
|
||
- `GET /api/fundraising/export`
|
||
- `POST /api/fundraising/backup`
|
||
- `POST /api/fundraising/restore-preview`
|
||
- `POST /api/fundraising/restore`
|
||
- `GET /api/fundraising/backups`
|
||
- `GET/PATCH /api/fundraising/backup-policy`
|
||
- `GET /api/fundraising/relational-summary`
|
||
- `GET /api/feature-requests`
|
||
- `POST /api/feature-requests`
|
||
- `PATCH /api/feature-requests/:id`
|
||
- New DB tables:
|
||
- `fundraising_state`
|
||
- `fundraising_investors`
|
||
- `fundraising_contacts`
|
||
- `fundraising_funds`
|
||
- `fundraising_commitments`
|
||
- `fundraising_views`
|
||
- `feature_requests`
|
||
- `app_settings`
|
||
- Grid saves/restores now sync into relational fundraising tables automatically.
|
||
- Formula engine is now sandboxed (no `eval`/`new Function`) with expanded function support.
|
||
- Automation engine v1 added:
|
||
- Rule table + toggle API
|
||
- List memberships (`main`, `follow_up`, `graveyard`, `longshot`, `all`)
|
||
- Automation run log
|
||
- Collaboration/reliability additions:
|
||
- Unified activity feed API (`audit` + `automation` + `backup`)
|
||
- Backup integrity verification API
|
||
- Better version-conflict metadata (`updated_at`, `updated_by`)
|
||
- Security hardening additions:
|
||
- Basic IP rate limiting (login and write APIs)
|
||
- Configurable CORS origin (`CRM_CORS_ORIGIN`)
|
||
- Production secret enforcement (`CRM_ENV=production` requires `CRM_SECRET_KEY`)
|
||
- Security status API + go-live checklist (`SECURITY.md`)
|
||
|
||
## Phase 1 (Production foundation)
|
||
1. Persist grid + views on backend
|
||
- Wire frontend fundraising grid reads/writes to `/api/fundraising/state`.
|
||
- Keep localStorage only as emergency fallback.
|
||
- Add autosave debounce and conflict handling (`expected_version`).
|
||
|
||
2. Admin-invite auth model
|
||
- Disable self-register for non-admin users.
|
||
- Add admin-only invite/create-user endpoint.
|
||
- Keep role model: `admin`, `member`.
|
||
|
||
3. Deployment and remote access
|
||
- Add `docker-compose` for one-command launch.
|
||
- Reverse proxy + TLS option (Caddy/Traefik) for non-Tailscale deployments.
|
||
- Recommended for your use case: Tailscale private access to laptop host.
|
||
|
||
4. Data safety and operations
|
||
- Automated nightly SQLite backups and restore test script.
|
||
- Add `/api/fundraising/export` for JSON snapshot export.
|
||
- Add health/readiness checks.
|
||
|
||
## Phase 2 (Airtable parity)
|
||
1. Advanced views
|
||
- Multi-condition filter groups (AND/OR groups)
|
||
- Multi-column sorting
|
||
- Pinned/frozen columns
|
||
- Personal vs shared views
|
||
|
||
2. Formula engine v2
|
||
- Add functions: `SUM`, `MIN`, `MAX`, `ROUND`, `ABS`, `CONCAT` (done)
|
||
- Type-aware formulas and better errors
|
||
- Dependency graph and recalculation rules
|
||
|
||
3. Activity + audit
|
||
- Record-level change history in UI
|
||
- Last modified by / at fields
|
||
- Restore archived rows
|
||
|
||
## Phase 3 (Team workflow and automation)
|
||
1. Tasks/reminders tied to investors/contacts
|
||
2. Automation rules (graveyard/follow-up triggers)
|
||
3. Email/communication integrations (optional)
|
||
4. Granular permissions (if team grows)
|
||
|
||
## Backlog (post-Phase-1 agentic)
|
||
|
||
### Follow-ups/reminders + NL search + bot grid-mutations (agreed plan, 2026-06-18)
|
||
*Agreed with Grant 2026-06-18. Three workstreams, sequenced **W1 → W2 → W3**. **Overarching constraint (Grant):** the dominant risk is **leaking LP data (names, $, notes, contacts) to third-party LLMs — NOT write-safety.** A wrong number is recoverable; investor substance reaching Claude is not. Consequences: W2 keeps LP rows off Claude (only the question text + schema vocabulary leave the box; entity names resolved locally); W3 keeps bot mutation-parsing on local Qwen. Because this DB *logs* commitments/pipeline but doesn't move money, a bot mutation is low-stakes → **any team member may approve one in Matrix**; the guardrail is "the bot can't silently mass-change numbers," enforced by the per-mutation human approval gate, not a tight money gate.*
|
||
|
||
**W1 — Reminders & follow-ups — BUILT + tested locally 2026-06-18 (v0.1.0:92, deploy pending).** First-class tickler tied to the grid: `reminders` table (in-app migration `0006`; logical FK to `fundraising_investors.id` + denormalized name, like `0005`), full CRUD (`GET/POST/PATCH/DELETE /api/reminders`; soft-delete; status open/done/snoozed/cancelled; assignee; `source` human/bot/automation), a read-only **derived `reminder_status` grid column** (overdue/due_soon/open — injected like `pipeline_stage`, **filterable so the follow-up view can later key off reminders instead of the binary `follow_up` checkbox**, per Grant), an orphan reconciler (`reconcile_grid_reminders` — cancels reminders when their investor leaves the grid, the pipeline reconciler's twin), a **Reminders** nav page (filter/complete/snooze/edit/delete + create), a Dashboard **"Reminders Due"** card, a **"Reminders due"** daily-digest section, and a per-investor **`last_activity_at`** recency rollup (the shared building block W2's "not nurtured" query needs). Pure local CRM — no LLM path, no leak surface. Tests: `test_reminders.py` + digest reminders test; **31/31 suite green, render-smoke green**. **Deploy:** needs an s9pk build + install (version bumped to 92); get authorization first.
|
||
- **W1b (deferred fast-follow):** nurture-gap automation — a daily job flags "committed / in-pipeline + no activity in N days + no open reminder" → auto-suggests a reminder (`source='automation'`, human confirms). Build once the recency rollup is proven in practice.
|
||
- **Left untouched (deliberate):** the grid `follow_up` checkbox + automation list-memberships, and `communications.next_action_date` + `/api/outreach/radar` — reminders are the new richer layer; folding those into it is a later cleanup, not now.
|
||
|
||
**W2 — Natural-language query (read-only). BACKEND BUILT + tested + validated locally 2026-06-18; web/Matrix UI pending.** = the **"Email/communication search + NL query → item 3 (NL→safe structured query)"** below, now sequenced second and **redesigned** (see below). Subsystem detail: `docs/guides/nl-query.md`.
|
||
|
||
- **Approach changed from the original "Claude behind redaction + a validated filter-AST" to LOCAL-ONLY + a named-intent catalog (decided with Grant 2026-06-18).** Rationale: (a) the dominant risk is LP data reaching a vendor — running translation on the **local Qwen via Spark Control** keeps the question on the box entirely (same basis as intake/digest), so there is **no Claude path and no redaction boundary** to manage, which is both simpler and safer; (b) a generic SQL/AST compiler was over-built for the real need — instead there are **~12 curated, hand-written, parameterized "named queries"** (`backend/nl_query/intents.py`) each with typed slots, and the **slot validator** (`runner.validate`) is the whole trust boundary (no dynamic identifiers, no raw SQL). The LLM only maps a question → `{intent, slots}`; its output is still validated, so a hallucinated intent is rejected. **Results never go to any model** (deterministic local render). Both design choices were pressure-tested by independent review agents before building.
|
||
- **As built:** `backend/nl_query/` (`intents.py` catalog, `runner.py` validator/executor + audit, `translate.py` local-Qwen translator, `try_questions.py` dev harness). `POST /api/query/nl` (`{question}` or direct `{intent,slots}`) + `GET /api/query/catalog`, `require_bot_or_admin`, read-only, audited (`audit_log` `entity_type='nl_query'`). Soft-delete-correct per table (`fundraising_*` has no `deleted_at` — `graveyard` is the axis; emails via a live `email_account_messages` sighting; reminders/opps/comms via `deleted_at`). Builds on W1's `last_activity_at`. Tests: `nl_query/test_nl_query.py` + `test_translate.py` + `test_nl_query_endpoint.py` (34/34 suite green).
|
||
- **Validation:** the local Qwen translated **12/12 of Grant's real example questions** correctly (right intent + slots, incl. "3 months"→90, sent/received→direction) against the live Spark — settles local-only; Claude not needed. Translation quality on messy/typo/no-match inputs shakes out in live use.
|
||
- **Remaining:** **step 4** = web "Ask" box in the Communications tab (calls the endpoint, renders rows + the interpreted query); **step 5** = Matrix `@bot <question>` (thin client of the endpoint; the 2-admin review room means a full-book dump is acceptable, so no bulk-result cap — only a light anti-flood truncation). Reads need no approval gate. Then deploy with reminders (v92) as **v0.1.0:93**.
|
||
|
||
**W3 — Bot grid-mutations behind a Matrix approval gate.** Generalize the email-proposal scaffold (`email_proposal_matrix` + propose→post→decide→apply) into one `agent_proposals` table (kind discriminator + JSON payload + target). Bot proposes set-commitment / assign-fund / change-stage / set-reminder; a human approves/edits/rejects in Matrix (**any member**); then apply. **Surgical, version-checked mutations — never blob RMW:** stage rides the existing `opportunities` link + validated stage endpoint; reminders write the W1 table; set-commitment/assign-fund need a version-checked single-cell upsert into the grid blob. Triggers the deferred **scoped service-token** item below (per-mutation-kind allowlist on the bot credential; money/merge/delete always require human approval regardless of scope — the autonomy axis). Parse on local Qwen, not Claude.
|
||
|
||
|
||
### Matrix-bridge intake for the fundraising grid — M1+M2 BUILT (deploy + live smoke pending)
|
||
*Requested 2026-06-16. **M1 (scaffold + parse + in-thread propose) and M2 (match + write-on-approve) built, tested (26/26), not yet deployed** — code in `backend/matrix_intake/`, guide at `docs/guides/matrix-intake.md`. Remaining: install `matrix-nio` + creds on the Spark, create the CRM bot user, and a **live Matrix smoke** (can't run in CI). M3 (business-card photo) deferred until Spark Control has a vision model. Next major build after this is **Pipeline adoption** (see below).*
|
||
|
||
Use the **matrix-bridge** repo's pattern to listen on a dedicated ten31-database Matrix room. Send a message (with an optional business-card photo) and a local LLM **via Spark Control** parses it into the fundraising-grid schema and **auto-creates the investor entity + contact row**. For an existing investor, send a meeting note and it **appends an interaction-log entry**. Approval gate: the bot replies in Matrix with the proposed add/edit; the user approves / rejects / edits in-thread before the write commits (keeps the draft→human-approve guardrail).
|
||
- Fits the "grid is canonical" direction (writes land in `fundraising_*`) and the never-send-autonomously rule (in-thread human approval before any write).
|
||
|
||
**Locked design (2026-06-16, approved) — build now, M1 then M2:**
|
||
- **Separate component, shared scaffold:** new `backend/matrix_intake/` (its own process; lifts matrix-bridge's connect/prime-then-listen/threaded-reply plumbing). `matrix-nio` is isolated to this component's `requirements.txt` — it never enters the stdlib CRM runtime. Keeps the CRM write credential + LP data out of the general-purpose matrix-bridge bot (blast-radius + data-sovereignty), and lets the two iterate independently. Runs on the Spark (placement settled against `standards/guides/placement.md` at deploy).
|
||
- **v1 = text-only.** Business-card photo deferred to M3 — Spark Control fronts chat/embeddings/rerank but **no vision model** today, so photo→fields isn't buildable end-to-end yet.
|
||
- **Parse:** local Qwen via Spark Control `/v1/chat/completions` (temp 0, JSON-only), reusing the existing Spark client pattern (`backend/redaction`/`backend/ingest`).
|
||
- **Approval handshake (the one stateful piece):** in-memory pending-proposal store keyed by Matrix thread root; user replies **yes / edit field=value / no** in-thread. Satisfies never-write-autonomously; exempt from "agents draft, humans send" (internal data entry, like the digest).
|
||
- **CRM-side:** `POST /api/intake/investor` (service-auth) creates a new investor+contact **through the existing grid-save path** (so relational sync + audit + backup-on-write happen as with a UI edit; bot never does whole-blob RMW) or appends a meeting note to the interaction log for an existing investor; `GET /api/intake/match?q=` fuzzy-matches via the existing entity-resolution/email-matcher. New investor needs no fund at intake.
|
||
- **Phases:** M1 = scaffold + parse + in-thread propose, **no writes** (proves Matrix↔Spark). M2 = intake endpoint + match + write-on-approve + tests. M3 (deferred) = business-card photo.
|
||
|
||
**Post-deploy enhancement — fuzzy match + in-thread confirm (Grant, 2026-06-17). DEPLOYED & LIVE 2026-06-17 (v0.1.0:86; box migration chain …85→86 clean, `candidates` endpoint verified); Matrix live-smoke pending.** Today `find_intake_match` is **exact-after-normalization** (`_normalize_text` = lowercase+strip), so near-misses — "Charlie" vs "Charles" (same last name), "Acme Capital" vs "Acme Capital LLC", a one-character email typo — return no match and the bot proposes a **new** investor, risking a duplicate the human approves without realizing a near-match exists. The existing in-thread approval gate is useless against this because the human is never *shown* the near-match. Fix: matcher returns **ranked fuzzy candidates** (deterministic pre-filter: normalized name similarity / token overlap + email edit-distance ≤ ~2), surfaced in-thread for the human to confirm or pick, with the **local Spark LLM optionally re-ranking/judging the shortlist** (good at Charlie/Charles + legal-suffix equivalence; fed only the shortlist, never the whole LP list). Keeps the approval gate but makes it effective against duplicates. Land **after** the live smoke — net-new logic + reply grammar + tests; the current exact match is safe and its failure mode (a duplicate) is recoverable via the existing entity-merge subsystem (`backend/entity_*.py`).
|
||
- **As built:** `find_intake_candidates` in `server.py` (deterministic — stdlib `difflib` name similarity + token-set Jaccard, legal-suffix-aware via `_strip_legal_suffix`, + email Levenshtein ≤ 2; ranked, ≥0.62, top 5). `GET /api/intake/match` now returns `{match, candidates}`. Bot: a new `_stage="disambiguate"` shortlist (`proposals.render_disambiguation` / `interpret_disambiguation` / `attach_to_candidate` / `promote_to_new`) — human picks a number / `new` / `no`. **The optional LLM-judge re-rank was deliberately deferred** (the deterministic filter already surfaces the named cases; an LLM judge is the right *pruner* for shortlist noise — build if the deterministic ranking proves too noisy in practice). Tests: `test_intake_endpoints.py` (server fuzzy cases), `matrix_intake/test_proposals.py` (disambiguation grammar), `matrix_intake/test_crm_client.py` (candidate shape).
|
||
|
||
**Post-deploy enhancement — conversational (LLM-mediated) edits (Grant, 2026-06-17). DEPLOYED & LIVE 2026-06-17 (bot-side; pulled + restarted on the Spark `modelo32`); Matrix live-smoke pending.** Today an in-thread correction uses a rigid grammar (`edit field=value`). Let a free-form reply that isn't `yes`/`no`/a literal `edit …` be treated as a natural-language revision instruction: send {current proposal + the instruction} back through local Qwen (`spark.py`, the same parse leg — no Claude, no scrub) and re-render the revised proposal card for approval (e.g. "add that we met on June 14" → updated Note). Keeps the draft→human-approve gate (the human still confirms the LLM's revision) and subsumes `edit field=value` as a deterministic fast path. Thread the instruction text into `normalize`'s source so the email-integrity rule still holds (a revised email must appear in the original message or the instruction). Pairs naturally with the fuzzy-match item above — build both as one conversational-UX pass after the smoke. (Parsing of free-form *intake* messages already works today via the Qwen parse leg; this item is specifically about the *edit/refine* turn.)
|
||
- **As built:** `parse.revise` + `_apply_revision` (offline-testable; the approval-stage `else` branch in `bot.py` routes any non-yes/no/edit reply here). `parse_message` now stashes `_source_text` so revise can re-check email integrity against {instruction + original}; the model's email field is never trusted. No-op revisions are caught via `proposals.same_fields` (re-prompt, not a false "Updated"). **Known v1 limit:** revise edits fields but does not re-run the matcher on a mid-thread firm rename. Tests: `matrix_intake/test_parse.py` (revise merge + email integrity + match-id preservation).
|
||
|
||
**Managed service — DONE (container) 2026-06-17; dashboard card deferred to a spark-control session.** The bot ran as a bare `nohup` process (silently died on a Spark reboot). Now it's a **docker-compose service** (`docker-compose.yml` at the repo root + `backend/matrix_intake/Dockerfile`; `restart: unless-stopped` → survives reboot; image bundles `backend/matrix_intake` + the stdlib `backend/ingest` Spark client; `.env` mounted read-only). Cutover done on the Spark (nohup stopped, container `matrix-intake` up + listening). **Still bare `docker`/SSH-managed** — a spark-control dashboard card (Update/Restart/Stop/Logs tile like `matrix-bridge`) is a separate task in the spark-control repo: see `docs/handoffs/add-intake-bot-to-spark-control.md`.
|
||
|
||
**Parse mis-identifies the investor when the message names an internal teammate (found in live-smoke, 2026-06-17).** *"jonathan is chatting with wyoming soon about fund commitment"* → the bot picked **jonathan** (a colleague/CRM admin) as the investor and offered a Jonathan/Nathan fuzzy shortlist, when the investor is **Wyoming**. Root cause is upstream of matching: local Qwen has no notion of who's internal, and mis-read the sentence role. **Fix (cheap, high-confidence, near-term):** feed the parse prompt the ~5-person team roster + the frame *"messages are written by a team member about a prospect; a named team member is the person doing outreach, never the investor"* (roster from a config value or a small read — not the admin-gated `/api/users`, since the bot is a member). Offline-testable (stub the model). **Bigger design (deferred, needs more failure samples):** the user's idea of routing inputs through the LLM *with grid context* for entity resolution — feasible (local Qwen, same as the digest, never Claude) but feed a **bounded shortlist, not the full ~400-name grid** (a small model dilutes on a haystack); pairs with the deferred LLM-judge. Also exposes a missing concept: the **internal deal owner** (Jonathan), which the bot doesn't model. Get 3–5 more real intake messages before re-architecting; the roster fix lands regardless.
|
||
|
||
**Long-term — extract the intake bot to its own repo (recommended, not yet done).** Containerizing from this monorepo is the pragmatic now-state, but the bot is a genuinely separate deployable (own process, own `matrix-nio` dep, own lifecycle); its only CRM coupling is the HTTP API (a clean network contract) plus ~40 lines of stdlib Spark client (cheap to vendor). The tell: the spark-control Update button would run `git reset --hard origin/main` on the **whole CRM clone** — wrong blast radius. `matrix-bridge` is already a dedicated repo; mirror it. The extraction is a migration (new Gitea repo, move code + tests + guide, vendor the client, re-point the Spark deploy), so it's deferred until worth the lift — do it *before* wiring the spark-control card if both land in the same push.
|
||
|
||
### Scoped service-credential auth path for automated CRM writers
|
||
*Surfaced 2026-06-17 while deploying the Matrix intake bot. **Decision: defer — the bot uses a dedicated member username/password for now.** The CRM has no API-key/service-token path; its only auth is username+password → JWT. A dedicated **member** login is appropriately scoped against what matters operationally (no admin: can't manage users, reset data, or change settings) and unblocks the live smoke today.*
|
||
|
||
**Accepted residual risk (why this is worth revisiting):** a member credential is far broader than the bot's actual need (two endpoints: `GET /api/intake/match`, `POST /api/fundraising/log-communication`). A member can **read the entire LP/prospect database** — the exact data this system exists to keep off third-party servers — plus broad member-level *write* within the fundraising domain (could create/append on any investor). The credential lives in a `.env` on the Spark, so a Spark compromise leaks read-access to all LP data. Mitigating context: own-infra, LAN-local; the Matrix bot is the **first out-of-process API writer** (the digest runs in-process with direct DB access), so there is exactly **one** consumer today → building a token-scope framework now is premature (YAGNI).
|
||
|
||
**Right long-term design:** a hashed, revocable **service token** with a per-route **scope allowlist** (intake-match + log-communication only), minted/revoked from the admin panel, replacing the bot's member login. Revocation then kills the token without rotating a reused human password.
|
||
|
||
**Build trigger:** when a **second** out-of-process automated writer appears, OR before **any** automated writer is reachable beyond the LAN — whichever comes first. Build it once, properly, at that point.
|
||
|
||
### Admin-only vs. all-users web-UI surface — audit
|
||
*Requested 2026-06-16 (idea, P2).* Have the **explorer agent** report which web-UI functionality is visible only to admins vs. to all users (member role) — a map of the role-gated surface across `frontend/index.html` and the backend route auth checks. Useful input for the consolidation/permissions work.
|
||
|
||
### Daily activity digest (email to the team)
|
||
*Requested 2026-06-15. **Phase A deployed** (v0.1.0:76). **Phase B deployed & verified live in v0.1.0:77 (2026-06-16)** — digest content + Spark summarization + daily scheduler + by-investor section + admin-panel control + on-demand send. Auto-send defaults OFF until an admin enables it in Settings → Admin. **v0.1.0:83 (built, deploy pending): in-app windowed preview** — Settings → Admin builds a digest over a chosen window (last 24h or since a date) and shows it before sending (`POST /api/admin/digest/preview`), so the **real Spark summarizer can be verified on demand** even on a quiet day (the fixed last-24h `send-now` couldn't); manual send uses the same window and never touches the daily cursor.*
|
||
|
||
**Decisions (locked 2026-06-15):** recipients = **all active admins**; summarization = **Spark-LLM narrative** (never Claude — un-anonymized substance stays local); granularity = **grouped by user** (→ per investor).
|
||
|
||
**Send transport — DECIDED 2026-06-15: Gmail domain-wide delegation** (not SMTP). The box's existing service-account grant (which powers email capture) **includes `gmail.compose`**, which authorizes `users.messages.send` — verified by a token-mint probe **and a live `messages.send` to grant**. So the digest sends through the account the CRM already uses: **no app password, no new account, no admin change.** The narrow `gmail.send` scope is *not* granted, so the sender must request `gmail.compose`.
|
||
|
||
**Phase A — DONE:** (v0.1.0:75) `configureDigestSmtp` Start9 action + `docker_entrypoint.sh` `SMTP_*` export + `backend/smtp_send.py` + admin `POST /api/admin/digest/test-email` (recipient-restricted to the admin set — not an open relay) + Settings button. (v0.1.0:76, redeploy pending) `backend/email_integration/gmail_send.py` (`users.messages.send` via DWD/compose) + `backend/digest_mailer.py` (**Gmail-DWD preferred, SMTP fallback**); the endpoint + button route through it; sender = `CRM_DIGEST_SENDER` else first active admin. Tests: `test_smtp_send.py`, `test_smtp_endpoint.py`, `test_gmail_send.py`.
|
||
|
||
**Phase B — DONE (2026-06-15/16):** `backend/digest_builder.py` builds **two sections** — *by team member* (per-user **Spark** narrative + both directions, with a deterministic fallback) and *by investor* (team-wide, inbound + outbound, deduped per email, structured). Soft-delete filtered throughout. `backend/email_integration/digest_scheduler.py` is an always-on daily thread that re-reads a **DB-backed policy** each cycle and sends once/day at the configured hour to all active admins (window cursor in `app_settings`). Control moved out of env into the **admin panel**: `app_settings.digest_policy` + `GET/PATCH /api/admin/digest/policy` + a Settings → Admin **enable toggle + send-time dropdown** (env vars only seed the first-boot default). Plus admin `POST /api/admin/digest/send-now` + a "Send Digest Now" button. Decisions settled: **6 PM default**, **always-send** (empty-day note), **per-user narrative + by-investor section**, **in-app control** (not StartOS). Tests: `backend/test_digest_builder.py`. Detail: `docs/guides/email.md`.
|
||
|
||
Have the CRM send a **daily digest email** summarizing each registered user's activity — primarily **who emailed which investors and the substance of those emails** — to the fund principal (and eventually other admins). Scales with the synced-user count: 2 users synced today, ~5 eventually.
|
||
|
||
- **Source data:** the captured email-activity already flowing through the Gmail DWD propose→approve pipeline (`backend/email_integration/`), keyed per registered user → per investor/contact. Optionally fold in other CRM activity (audit feed, automation runs, new opportunities) later.
|
||
- **Send path is NEW capability.** Today nothing leaves the box — the system only *captures* Gmail and *creates drafts*. This needs outbound SMTP. StartOS 0.4 has a system-wide SMTP account (since v0.4.0-beta.9): the user configures it once for the whole server and services read it via `sdk.getSystemSmtp(effects).const()`, which returns a `T.SmtpValue` (`host`, `port`, `from`, `username`, `password`, `security`). Wire the digest sender to that rather than hardcoding any account. *Implementation path (researched 2026-06-15, our SDK pin `^0.4.0-beta.66`):* model a `manageSMTP` action on [gitea-startos](https://github.com/Start9Labs/gitea-startos) / [vaultwarden-startos](https://github.com/Start9Labs/vaultwarden-startos) — a three-way `selection` (system / custom / disabled) built on `sdk.inputSpecConstants.smtpInputSpec`, persisted to `storeJson`, with `main.ts` injecting `SMTP_HOST/PORT/USER/PASS/FROM/SECURITY` env vars into the daemon `exec` block (same shape as the existing `setAnthropicApiKey.ts` action). The Python sender reads them via `os.environ` and opens `smtplib.SMTP`/`SMTP_SSL`. **"Custom SMTP" is a dedicated per-package account, fully independent of the server's system SMTP** — the custom branch never calls `getSystemSmtp`, so the digest can send through its own provider even on a box with no system account configured (confirmed in both reference packages). This is the likely fit here: a digest-only mailbox separate from anyone's Gmail. Note StartOS 0.4 dropped the old `Config`/`Properties` manifest spec — SMTP config is an **action + storeJson**, not a manifest config field. **SDK note (verified 2026-06-15):** our pin `^0.4.0-beta.66` resolves to exactly `0.4.0-beta.66` (caret on a prerelease stays within the `0.4.0` tuple), whose SMTP surface — `getSystemSmtp` → `T.SmtpValue {host, port, from, username, password, security}`, `inputSpecConstants.smtpInputSpec` (providers gmail/ses/sendgrid/mailgun/protonmail/other; selection disabled/system/custom), `smtpShape`, `smtpPrefill` — is **byte-identical** to the 1.5.3 reference packages (verified from published tarballs; repo `node_modules` is absent). Build against beta.66 as-is — **no SDK bump needed** (moving to 1.x is a major-track change with broad blast radius across `startos/`, and nothing about SMTP justifies it).
|
||
- **Analysis runs on Spark, never Claude.** The digest is deliberately **un-anonymized** (real LP names + email substance), so any summarization/analysis must go through **Spark Control to local models** — this is the one path that intentionally bypasses the scrub→Claude→re-hydrate boundary, because keeping the substance local is the whole point. Never route digest content to Claude.
|
||
- **Exempt from "agents draft, humans send."** That rule governs outward LP/prospect contact. This is an internal ops digest to the team's own inboxes — a different category — so an automated daily send here does not violate the draft-only guardrail. State this explicitly at build time.
|
||
- **Scheduling:** a daily cron, naturally co-located with the existing `backend/email_integration/scheduler.py` sync cadence.
|
||
- **Soft-delete:** every aggregate/read in the digest must filter `deleted_at IS NULL` (see the standing soft-delete rule).
|
||
|
||
Open design questions (settled at build time): send time = **6 PM box-local** (configurable in the admin panel), covering the ~24h window up to send; empty days = **always send** with a "no activity" note; summary granularity = **one per-user narrative** plus a **by-investor structured section** (inbound + outbound, team-wide) added 2026-06-16; enable/time live in the **admin panel** (DB-backed), not StartOS actions.
|
||
|
||
### Email/communication search + natural-language query
|
||
*Requested 2026-06-16. Three increments, **sequenced 1 → 2 → 3** (1 and 2 first as a quick increment; 3 is a separate, larger build after). Origin: Grant asked whether we can query "emails sent to a specific investor" / "activity by user," and floated NL queries like "existing investors who have committed capital across our funds that we haven't emailed in a while."*
|
||
|
||
**Status: items 1 & 2 SHIPPED in v0.1.0:83 (built + verified locally 2026-06-16, deploy pending).** The Communications tab now has the structured activity surface (item 1: typed/fixed investor dropdown, mailbox + direction + **date-range** filters, free-text, **click-to-expand full body** via `GET /api/email/detail`) and a **"Search content"** semantic mode (item 2: `GET /api/email/search` over the Qdrant email index). The dropdown-empty bug (the facet only listed grid investors) was the v83 fix — it now mirrors the list across grid/org/contact matches. **Item 3 (NL→SQL) remains** — the larger, separate build below. Detail: `docs/guides/email.md`.
|
||
|
||
**Context — the data is captured but currently has NO front-end.** The entire Gmail email schema (`emails`, `email_threads`, `email_investor_links`, `email_account_messages`, `email_activity_proposals`, …) exists and is populated by the DWD capture pipeline, but is surfaced **nowhere** in `frontend/index.html` today (only as inputs to the daily digest). So all three items below are about making already-captured data queryable/visible. Email bodies of *matched* emails are already chunked + embedded into Qdrant with `{lp_id, lp_name, doc_type:"email", date_ts}` metadata.
|
||
|
||
**Caveat that shapes all three — the two-model join.** "Emails to an investor" link to the **fundraising grid** (`email_investor_links.fundraising_investor_id`); "committed capital" lives in the grid too (`fundraising_commitments`, multi-fund). But manually-logged `communications` and `lp_profiles` (single-fund) live in the **classic** model, and the two models are only bridged by fuzzy email/name matching (no authoritative join key). Any query spanning "committed capital" + "email recency" must reckon with this. Prefer the grid side as the higher-signal source (matcher already does).
|
||
|
||
**1. Activity query endpoints + panel — DONE (v0.1.0:83).** Delivered as the **Communications tab** rather than the originally-sketched `/api/activity` endpoints: `GET /api/email/activity` (`db.query_email_activity`) returns the actual records filterable by investor / mailbox / direction / **date range** / free-text, and `GET /api/email/detail` expands the full body. Answers "emails to investor X" and "what has mailbox Y sent" interactively. Soft-delete filtered throughout; investor identity is typed (`fund:`/`org:`/`contact:`) so org/contact-only matches resolve and are pickable. *(The `collect_user_activity()`/`collect_investor_activity()` digest helpers remain the by-user/by-investor pivot source; a dedicated per-user pivot UI was not needed for the answer Grant wanted, which the mailbox+direction filters already give.)*
|
||
|
||
**2. Email content search box — DONE (v0.1.0:83).** A **"Search content"** toggle in the Communications tab → `GET /api/email/search?q=` wraps `backend/ingest/search.py:hybrid_search` filtered to `doc_type='email'`; hits are hydrated + soft-delete-filtered against SQLite (canonical) and link back to the full body. Semantic/lexical search over email *content* ("find where we discussed the mining deal"), distinct from item 1's structured filters. 503 (clean "unavailable") when Spark/Qdrant is unreachable.
|
||
|
||
**3. Natural-language → safe structured query — SUPERSEDED & BUILT as W2 above (2026-06-18).** The design constraints below (especially "LLM = Claude behind the redaction boundary" and the validated-AST shape) were **revisited and changed** during the build: translation runs on the **local Qwen** (no Claude, no redaction), and the safe surface is a **named-intent catalog**, not a generic query AST. See the W2 entry above and `docs/guides/nl-query.md` for what shipped; the original framing is kept here for provenance. _An LLM translates a plain-English question into a **safe, read-only** DB query against the CRM, for relational/analytical questions that semantic search *cannot* answer — Grant's example ("committed across funds AND not emailed in a while") is joins + aggregates + recency, not a text-topic match. Original design constraints (locked at request time):_
|
||
- **LLM = Claude behind the redaction boundary** (better at text-to-SQL than local Qwen; the scrub→Claude→re-hydrate path already exists for the PII concern). Not Spark — Spark Control offers embeddings/rerank/RAG + local chat, but **no text-to-SQL**.
|
||
- **Safety is the hard part, not the parsing.** Do NOT hand the LLM open-ended SQL against the live DB (soft-delete leaks, injection, runaway scans). Constrain it: read-only connection/view, a curated/parameterized query surface or a validated query AST, soft-delete-filtered views, row/time caps. Treat as its own designed feature with its own tests.
|
||
- Must reckon with the two-model join caveat above (capital lives in the grid; recency from email links).
|
||
|
||
### Consolidate on the fundraising grid as canonical; retire vestigial classic-CRM surfaces
|
||
*Decided 2026-06-16. The CRM carries two stacked models: the original generic CRM (contacts / lp_profiles / opportunities / manual communications) and the fundraising grid + email capture. The team uses the grid; most classic surfaces are un-adopted (verified on the box: Pipeline + Communications empty, Contacts auto-populated from the grid). **Decision: the fundraising grid + email capture is the canonical system of record;** prune or repurpose the rest rather than maintain a parallel half-empty CRM.*
|
||
|
||
**Retire `lp_profiles` + LP Tracker — DONE & deployed live (v0.1.0:78, 2026-06-16).** 21/21 backend tests green, `py_compile` clean; installed to the box (`installed-version`→`0.1.0:78`, migration chain …77→78 clean, server up on :8080).
|
||
- Removed the orphaned `LPTrackerPage` component + the `lp-tracker`→`fundraising-grid` redirect (frontend).
|
||
- Removed the `/api/lp-profiles*` endpoints (list/get/create/update) and their handlers, the unused `lp-breakdown` report + route, the contact-dossier LP display (frontend + the `lp_profile` block in `handle_get_contact`), and the demo-seed LP block.
|
||
- **Dashboard KPIs repointed:** "Total Committed" now sums `fundraising_investors.total_invested` (the canonical grid rollup), **excluding graveyarded investors** so the headline reflects live committed capital — a deliberate divergence from `/api/fundraising/relational-summary`, which sums all rows. "Total Funded" dropped — the grid has no funded-vs-committed concept and the frontend never rendered it. (If a funded/wired status is wanted later, that's a new grid feature, not a revival of lp_profiles.) Regression-guarded by `test_dashboard_report.py`.
|
||
- **Left in place (intentional):** the empty `lp_profiles` table + index (no destructive drop, per never-hard-delete); the contact-delete soft-delete cascade; the `--reset-all-data` clear; and the inert MOCK_MODE `mockDb.lp_profiles` fixtures (dev-only fallback, never hits the backend — its dashboard mock still reads mock lp_profiles, a known dev-only divergence from the real backend). Updated `test_soft_delete_reads.py` to drop the now-removed `lp_profile` assertions (kept its org `total_funded` opportunities-aggregate checks).
|
||
|
||
**Adopt the Pipeline — wire it to the grid. — DONE: DEPLOYED & live-smoked 2026-06-18 (v0.1.0:88; migration chain …86→88 clean, `0005_grid_pipeline_link.sql` applied on the box, server up; the full +Pipeline → board → advance-stage → remove round-trip is verified on the box).** *(Was: second build after the Matrix-bridge intake.)*
|
||
- Pipeline (`opportunities`) is fully built and functional but unused. Keep it: it's the one classic surface that tracks something the grid doesn't — a forward-looking deal funnel (stage, `expected_amount × probability`, owner, close date) vs. the grid's actual committed dollars + flags.
|
||
- New idea (Grant, 2026-06-16): let users **flag an investor in the grid as a pipeline opportunity** (a grid column/control) so it **auto-creates / syncs an `opportunities` row** that loads into the Pipeline board. Design the grid↔pipeline link (which fund seeds it? what sets stage/expected amount? keep them reconciled). Turns Pipeline from a disconnected second data-entry surface into a view driven by the canonical grid.
|
||
- Revisit the stray contact-create side-door (the "Create Opportunity" modal `POST /api/contacts`) once the grid-driven flow exists.
|
||
|
||
**As built (decisions locked with Grant 2026-06-17):** UX = **row action + seed modal** ("Add to Pipeline" per grid row → captures primary contact / target fund / expected amount / stage / probability). The durable link is `opportunities.fundraising_investor_id` (**migration 0005**, additive + reversible); "is in pipeline?" / "what stage?" are **derived from a live opp join**, never a denormalized flag (no drift). **Ownership split:** the grid owns whether the link exists + the seed; the **board owns stage/probability/owner/close/next-step** — a grid save never reseeds a live opp (`POST /api/fundraising/pipeline/link` is idempotent: one live opp/investor, re-link returns the existing one). Contact is **reused from the grid's synced `fundraising_contacts.contact_id`** — the `POST /api/contacts` side-door is **gone**. Grid `lead` → opp owner (fallback acting user). Two **read-only** grid columns (Pipeline action + Pipeline Stage) injected on read; their row values are stripped on write so they never persist or dirty the autosave. **Remove from pipeline** (`POST .../unlink`) **soft-deletes the opp; the grid row is left fully intact** (Grant's explicit ask). Deleting an investor from the grid archives its orphaned opp (`reconcile_grid_pipeline_links`, called after `sync_fundraising_relational`). **Folded in:** the P2 soft-delete leak in `handle_pipeline_report` + dashboard pipeline aggregates (archived opps no longer counted). Tests: `backend/test_grid_pipeline_link.py` (link/idempotent/round-trip/guards/unlink-intact/re-link/orphan/aggregates), 28/28 suite green, render-smoke green. **Deploy:** server-side → needs an **s9pk build + install** (v87); get authorization first.
|
||
- **Follow-up (v0.1.0:88, frontend-only, DEPLOYED & verified 2026-06-18):** retired the Pipeline page's **"+ New Opportunity"** button + its create-by-contact modal — an opportunity is now born **only** from a fundraising-grid investor row ("+ Pipeline"), matching how the team works (they live in the grid). The board is now a view + stage-management surface; button replaced with a muted "Add deals from the Fundraising Grid" hint. Removed the dead handler/state + the page's unused `/api/contacts` fetch.
|
||
- **Deferred (not built):** no write-back of committed dollars into grid fund cells (grid stays canonical for committed $); a graveyarded investor with a live opp still shows its stage (deliberate — a live deal is a live deal).
|
||
|
||
**Keep the Contacts table — as the read-only per-person directory it already is.** Confirmed 2026-06-16: the grid models **investor entity → many people** correctly today. The grid "contacts" column is a multi-pill editor; each pill syncs to a `fundraising_contacts` row AND its own classic `contacts` row (5-person family office → 1 investor + 5 contacts, linked via `fundraising_contacts.contact_id`, migration 0004). The Contacts page is **read-only for creation** (header: "added from the Fundraising Grid"; no New-Contact button), edit-only via the detail slide-over — the desired flow already holds. Email capture already rolls **multiple people up to one investor** (matcher indexes each pill's email separately, all → same `fundraising_investor_id`; `email_investor_links` records both investor and specific person). No build here — future email-surfacing UI should present comms grouped by investor across all its people.
|
||
|
||
### Front-end: pre-compile JSX, drop runtime Babel (optional, larger)
|
||
*Logged 2026-06-16 during the v0.1.0:82 vendor+SRI work. The scoped fix shipped: React/ReactDOM/Babel are now vendored + SRI-pinned and served same-origin, and a jsdom render smoke check gates every build (`docs/guides/packaging.md`). This is the bigger alternative we deliberately deferred.*
|
||
|
||
Today the app ships `@babel/standalone` (~3 MB) and transforms ~5k lines of inline JSX **in the browser on every page load**. A build step that pre-compiles the JSX to plain JS would (a) eliminate the runtime-transform blank-screen class entirely (no Babel in production), and (b) load much faster. **Cost:** it introduces a build step, which contradicts the current **"No build step"** convention (single `frontend/index.html`, inline-Babel React) — so this is a real architecture change, not a tweak. Weigh only if page-load size/latency or render robustness becomes a felt problem; the render-smoke gate already de-risks the status quo. If taken: keep the source `index.html` editable, emit a compiled artifact into the s9pk, and keep the smoke check pointed at the built output.
|
||
|
||
### One-off feature batch (Grant, 2026-06-18)
|
||
*Eight one-off ideas, triaged against the backend 2026-06-18. **Cross-cutting guardrail:** anything framed as "auto-add / auto-forward / auto-suggest" lands as a **proposal surfaced for human approval** (reuse the `email_proposal_matrix` propose→Matrix→decide rails), never a silent write — per "agents draft, humans approve." #1 is built (deploy pending); #6 is a spark-control task (→ INBOX); the rest are scoped backlog. #2/#4/#7 reuse existing rails (email-proposal loop + W2 NL-query) — they're "wire a new source into an existing pipeline," not greenfield.*
|
||
|
||
- **1. Drag-reorder fundraising grid views — BUILT (frontend; deploy pending), 2026-06-18.** The sidebar view list is now drag-reorderable (HTML5 DnD mirroring the column-reorder idiom: `moveViewBefore` + `draggingViewId`/`dragOverViewId` in `frontend/index.html`). Order persists via the grid page's **existing autosave** (`views` is already in its snapshot + deps → `PUT /api/fundraising/state` → `views_json`), the same path rename/delete use — **no backend change.** Render-smoke green; the in-app drag interaction itself not yet browser-tested. **Known edge (same as existing rename/delete):** reordering while *off* the grid page only updates localStorage and is re-hydrated from the backend on next grid mount — reorder while viewing the grid. **Deploy:** needs an s9pk build + install.
|
||
|
||
- **2. [P2] Suggest new contacts from digested emails (outreach detector).** When a captured *outbound* email goes to an address not already in `contacts`/the grid and looks like outreach, propose adding it as a contact. Hangs off the existing email capture + `email_proposal_matrix` / `/api/intake/email-proposals` review rails — net-new is the detector + "looks like outreach" criteria (exclude vendors / newsletters / internal domains). Lands as a **proposal**, not an auto-add.
|
||
|
||
- **3. Pipeline stages + investor flags/labels — sharpened into a LOCKED SPEC (2026-06-19).** Was "new pipeline stages"; the design conversation collapsed it into a 4-stage per-investor funnel + auto-derived Existing-Investor flag + staleness overlay/nudge. **Full locked spec: see "Pipeline stages + investor flags/labels — LOCKED SPEC" below.**
|
||
|
||
- **4. [P2] Squarespace website form-submissions → DB (near-term, high value).** Parse `form-submission@squarespace.info` capture emails — structured Name / Email / Company / LinkedIn / Location / comments (see the website-lead screenshots, Grant 2026-06-18) — and feed them into the proposal flow. Deterministic parser (fixed format) + existing proposal rails = relatively contained. **Guardrail:** despite the "auto-added" ask, land each lead as a **Matrix proposal → one-tap approve**, not a silent insert (same pattern as email proposals). Real leads (e.g. Matt Baas, Vikrum Tatla) are currently only living in an inbox.
|
||
|
||
- **5. [P3] Matrix voice note → Spark Control transcription → intake.** matrix-nio receives an audio/voice event → download + decrypt the media → **Spark Control** transcription endpoint (Whisper-class — **confirm it exists; external dep**) → feed the text into the existing local-Qwen intake parse + disambiguation. Never call a Spark directly (Spark Control only). Larger; gated on the transcription endpoint existing.
|
||
|
||
- **6. → INBOX (spark-control repo, not this one). Dashboard card for the crm/intake bot** (Update/Restart/Stop/Logs tile like `matrix-bridge`). Already noted under the Matrix-intake "Managed service" item + `docs/handoffs/add-intake-bot-to-spark-control.md`; captured to `standards/INBOX.md` to confirm/do in a spark-control session.
|
||
|
||
- **7. [P2] Intake: "query the LLM when the name doesn't match."** Extend the disambiguation grammar (today: number / new / no — see screenshot) with a `search: <text>` option that runs the read-only **W2 NL-query** to locate the real existing investor when the typed name doesn't fuzzy-match a candidate. Builds on the existing NL-query + intake rails; keeps the human approval gate.
|
||
|
||
- **8. [P2] Email capture learns from approve/reject (scope down to rules v1).** Use the already-logged approve/reject decisions to pre-suggest a decision. **v1 = deterministic, not ML:** detect `List-Unsubscribe` / `Precedence: bulk` (newsletters) + a learned denylist of rejected sender addresses/domains → pre-mark / auto-suggest reject (e.g. recurring non-investor newsletters). Don't build a classifier until the rules prove insufficient.
|
||
|
||
### Pipeline stages + investor flags/labels — LOCKED SPEC (Grant, 2026-06-19)
|
||
*Sharpened from the inherited 6-stage funnel (lead/outreach/meeting/due_diligence/committed/funded) over a design conversation 2026-06-18/19. Supersedes one-off batch item #3. **Locked — ready to build on green-light.** Grounding (verified): the grid's only labeling today is 3 boolean flags (priority/follow_up/graveyard) + a derived longshot + the `lead` **owner** column; there is **no investor type field**; "existing investor" is implicit in `total_invested > 0`; the 6-stage pipeline lives on classic `opportunities` and only applies to rows explicitly "+Add to Pipeline"'d; saved views are driven off the flags, not stage.*
|
||
|
||
**Conceptual frame — three orthogonal axes (were conflated):**
|
||
- **A. Relationship** — existing-LP vs prospect → collapsed to a single **auto-derived "Existing Investor" flag** (below). No prospect/lead/advisor sub-types: leads become prospects fast, and there are no advisors in this grid.
|
||
- **B. Disposition flags** — keep **Priority** (the focus set) + **Graveyard** (truly dead). **Drop Longshot** — labeling something longshot is already half-giving-up, overlaps graveyard, and doesn't earn a third bucket. Everything not Priority/Graveyard is the neutral middle.
|
||
- **C. Pipeline stage** — the active-raise funnel (below), per-investor.
|
||
|
||
**1. Funnel = 4 stages, per-investor, terminal at Commitment:** `Lead → Engaged → Diligence → Commitment`
|
||
- **Lead** — identified + first contact (cold outreach, a logged first meeting, or a website inbound); one-directional so far.
|
||
- **Engaged** — a **two-way** conversation exists (they replied / there's a back-and-forth). *(Boundary confirmed with Grant: two-way, not "a second person at the firm.")*
|
||
- **Diligence** — substantive: follow-up calls/meetings or data-room access.
|
||
- **Commitment** — terminal. On commit → hand off to fund admin + record the $ in the grid fund cell; the pipeline's job is done.
|
||
- **No Funded** (fund admin owns post-commitment; the Existing-Investor flag is effectively the "closed" signal). **No Meeting** (an activity, not a position). **No Lost** stage (the Graveyard flag covers dead).
|
||
- **Start at any stage** — a known LP re-solicited for a new fund drops straight into Engaged/Diligence, not Lead.
|
||
|
||
**2. "Existing Investor" = auto-derived flag** from `total_invested > 0`, injected read-only like `pipeline_stage` (never a maintained column); rendered as a star/indicator (esp. mobile). Orthogonal to stage — a re-solicited LP shows the star **and** a live stage at once. Lifecycle: prospect runs Lead→…→Commitment → $ recorded in the grid cell → they light up as an Existing Investor.
|
||
|
||
**3. Staleness — a derived overlay on the stage + a Matrix nudge, NEVER an auto-demotion.** Governing principle: **derive-and-display freely; mutate state only via a human.**
|
||
- A quiet deal does **not** change stage. Staleness shows on the **last-contact recency value** (the grid row's / mobile card's "2d ago"): light-grey when fresh → **amber → red** by days since `last_activity_at`, appending "stale" once it crosses the threshold (e.g. "35d stale"). The stage chip stays clean; the warning rides the recency line. The **same `last_activity_at` source drives the desktop grid and the mobile card**, so both color-code automatically.
|
||
- **Why not auto-flip off Engaged/Diligence:** it re-couples axes B+C, silently destroys information ("stalled mid-diligence" vs "never engaged"), is a silent un-approved mutation (against the human-in-the-loop guardrail), and creates a perverse "log junk to stay alive" incentive.
|
||
- The "auto" part is the **nudge = W1b nurture-gap** (see the W1/W2/W3 backlog; this refines its target set to **Engaged/Diligence**, not Commitment): daily job flags "in pipeline (Engaged/Diligence) + no activity > threshold + no open reminder" → bot **suggests a reminder**, a human confirms → re-engage (logging a comm resets `last_activity_at`) **or** consciously graveyard. The system nudges; the human acts. Deals never silently fall off.
|
||
- **Stale threshold: ONE global threshold (locked 2026-06-19).** Not stage-aware for v1 (Diligence-trips-faster was considered and deferred). Pick the amber/red day-counts at build.
|
||
- **Stale-as-a-view:** also a saved grid view keyed on `last_activity_at` (e.g. >90d, not graveyarded) — distinct from the per-stage overlay; both reuse `last_activity_at`, no new field.
|
||
|
||
**Accepted tradeoff (per-investor, not per-fund — Grant's call):** re-soliciting an existing LP for a new fund reuses their single opportunity (set fund + reset stage) — you won't see "Funded Fund I / Diligence Fund III" as two simultaneous pipeline entries. The grid's per-fund $ columns remain the record of which funds an investor is in; the pipeline shows only the *current* raise. (Per-fund stage was considered and deferred as a bigger build.)
|
||
|
||
**Concrete change set (cost asymmetry: labels/overlays are cheap; the enum is the one-time expensive bit):**
|
||
1. **Enum:** `PIPELINE_STAGES = ['lead','engaged','diligence','commitment']` (`server.py:1833`) + the ~8 mirror sites: report ordering CASEs (`server.py:3782/3859`), `nl_query/intents.py:34/37`, frontend kanban (`index.html:4168`, mock `:2174`), opp-form `<option>`s (`:7732`), and the `'funded'/'lost'` filters in `total_funded`/`pipeline_value` (`server.py:2721/3766/3877`).
|
||
2. **Data migration** of existing `opportunities.stage`: `outreach,meeting→engaged`; `due_diligence→diligence`; `committed,funded→commitment`. Reconcile the stray `lost` value (not in the settable enum) to graveyard-flag semantics.
|
||
3. **Existing-Investor flag:** derive from `total_invested > 0`, injected read-only (grid column + mobile star).
|
||
4. **Drop Longshot:** remove the derived `longshot_followup` + its deprecated view filter.
|
||
5. **Staleness overlay:** green/amber/red on the injected `pipeline_stage` by `last_activity_at`, + the stale saved view.
|
||
6. **Nudge:** specialize **W1b** to Engaged/Diligence in-pipeline deals.
|
||
|
||
Items 3–6 are cheap (derived/read-time/frontend, reuse `last_activity_at`, no migration); items 1–2 are the deliberate one-time enum + migration.
|
||
|
||
**Card presentation (mobile + grid, locked 2026-06-19):**
|
||
- **Stage chip** = one of the 4 stages, shown **only when the row is in the pipeline** (most grid rows aren't — no chip / a faint "+ Pipeline" affordance otherwise).
|
||
- **Top-right corner** = the **Priority** disposition only (star/pill when flagged, empty otherwise). Graveyard rows live in the Graveyard view / render muted — not a corner badge.
|
||
- **Existing Investor** (auto-derived, `total_invested > 0`) = its own distinct indicator (star by the name or a left accent — **not** a per-card banner; keep it restrained per `design/DESIGN.md`).
|
||
- **Last-contact recency** carries the staleness color (grey→amber→red, "Nd stale").
|
||
- This **replaces the design-mockup's INVESTOR/PROSPECT category chip** — we have no prospect/investor *type*; that two-value badge was the tool deriving committed-$>0, which is exactly our Existing-Investor flag. Feeds `design/BRIEF.md` §3a.
|
||
|
||
### Mobile-first implementation — backlog (design landed 2026-06-19)
|
||
*The `/design` round-trip is complete: the contract now describes the mobile-first system
|
||
(`design/DESIGN.md` §8 + the `mobile` token group), provenance + per-surface interaction model
|
||
are in `design/_imports/2026-06-19/`, and the input brief is `design/BRIEF.md`. This is the gap
|
||
between that contract and the current desktop-only `frontend/index.html` — the implementation
|
||
backlog. **Not yet started; scope/plan to be developed next (the user's stated next step).**
|
||
The comps are signed-off prototypes, **not drop-in** (Claude Design runtime, seed data) — each
|
||
surface is re-authored in the app's React idiom and wired to the **real API**.*
|
||
|
||
**Hard prerequisite — inline-style→CSS migration.** Responsive layout cannot live in the
|
||
~1,300 inline `style={{}}` objects (they can't carry media queries). Mobile-first means
|
||
authoring a 375px base + `min-width` enhancements in the CSS `<style>` block / utility classes.
|
||
This migration is **large and not yet scoped** — it gates everything below and is the first
|
||
thing the implementation plan must size. (Precedent for a mechanical sweep: the design guide's
|
||
inline-hex→`var()` field notes; this is bigger — structural layout, not just values.)
|
||
|
||
**Data-layer dependency — the locked pipeline-stages/flags spec** (see the section above) lands
|
||
**first or together**: the mobile cards render the 4-stage chip, the auto-derived
|
||
Existing-Investor star, and the staleness overlay, all of which need the stage enum + migration +
|
||
`total_invested>0` derivation + the `last_activity_at` ramp. Building the cards before the data
|
||
layer means hardcoding against a model that's about to change.
|
||
|
||
**UI workstreams (rough order; real sequencing comes with the plan):**
|
||
1. **Responsive shell + nav:** viewport-gated mobile shell, the safe-area-aware **4-tab bottom
|
||
bar** (Grid·Pipeline·Reminders·Contacts), the top-bar account/logout control, the bottom-sheet
|
||
primitive, and the type/touch bump — the chrome every surface shares.
|
||
2. **Grid (do first — canonical + the crux):** card list + bottom-sheet **view picker** + search;
|
||
full-screen detail with per-field bottom-sheet edits (name, contact pills, stage, reminder, log
|
||
note) + the `+`-create flow with client-side dedup typeahead. **Writes go through the targeted
|
||
one-row `POST /api/fundraising/log-communication` path + the pipeline link→`PATCH stage` flow —
|
||
never whole-grid `PUT /state`** (the `BRIEF.md` §3a "Backend reality"). Commitments read-only.
|
||
3. **Contacts (lowest-risk validator):** read-only A–Z list + full-screen detail; proves the
|
||
list+detail+sheet pattern before the heavier surfaces.
|
||
4. **Pipeline:** swipe-between-stages (snap-scroll + segmented control + dots), per-card stage move
|
||
sharing the same opportunities endpoints as the Grid detail.
|
||
5. **Reminders:** urgency-grouped list, swipe complete/snooze, add/edit sheets on `/api/reminders`.
|
||
6. **Light theme + toggle (adopted as a planned feature, 2026-06-19).** Ship the light palette
|
||
(`tokens.tokens.json` `color.light`) behind a `[data-theme]` switch + a top-bar toggle; dark
|
||
stays the default. Naturally co-lands with the inline-style→CSS migration (theming wants CSS
|
||
custom properties, not per-element inline values). Per-component light tints (stage/staleness/
|
||
note badges) are in `_imports/2026-06-19/GridApp.dc.html`.
|
||
|
||
**Note on `design-checker`:** not run for this round-trip — it audits *existing* UI conformance,
|
||
and the desktop UI still conforms to §1–7 (unchanged). The mobile gap is greenfield
|
||
implementation (captured here), not conformance drift, so there's nothing for it to flag yet; run
|
||
it after the mobile surfaces exist.
|
||
|
||
## Definition of done for "Airtable substitute" v1
|
||
- Team can manage all investors in one master table
|
||
- Saved views replicate current Airtable workflows
|
||
- CSV import from Airtable is reliable and repeatable
|
||
- Data persists safely and supports multi-user access
|
||
- Auth is invite-only and backups are automated
|