Keysat
6563a7811e
The email-activity panel surfaced every captured message, including cold/ unknown-sender email with no investor association. Gate query_email_activity on EXISTS(email_investor_links) so the panel shows only email tied to a known investor/contact. Capture is unchanged — unmatched email is still stored (metadata-only) and will appear automatically if its sender is later added as an investor; this is a read-side filter only. Graveyard investors are unaffected (their email has a link), so they remain visible/searchable as an audit surface, hidden only from the filter picker.
114 lines
7.4 KiB
Markdown
114 lines
7.4 KiB
Markdown
---
|
|
paths:
|
|
- backend/email_integration/**
|
|
- backend/digest_mailer.py
|
|
- backend/smtp_send.py
|
|
---
|
|
|
|
# Email capture & drafts (Gmail)
|
|
|
|
Read this before editing Gmail capture or draft creation.
|
|
|
|
## What it does
|
|
|
|
- `backend/email_integration/` captures Gmail via **domain-wide delegation** (`credentials.py`, `matcher.py`, `parser.py`, `db.py`, `sync.py`, `scheduler.py`, `routes.py`) and creates Tier-B in-thread drafts (`compose.py`). It has its own `migrations/`.
|
|
- Captured email becomes CRM activity through a **propose → approve** flow — nothing lands on a contact record until a human approves the proposal.
|
|
|
|
## Hard rule
|
|
|
|
- **Agents draft; humans send.** Never let an agent send email, post, or contact an LP autonomously. Tier-B `compose.py` only *creates* a Gmail draft for human review.
|
|
|
|
## Outbound mail — the daily digest (internal; exempt from "agents draft")
|
|
|
|
The CRM sends an internal **daily activity digest** to the fund's own admins. This is the
|
|
ONE automated send path, and it does **not** violate the hard rule above: that rule governs
|
|
outward **LP/prospect** contact. An internal ops email to the team's own inboxes is a
|
|
different category. **Never extend this path to send to LPs/prospects.**
|
|
|
|
- **Transport selector: `backend/digest_mailer.py`** (top-level, not in this package) —
|
|
`send_digest(conn, to_addrs, subject, body)` picks **Gmail-DWD (preferred) → SMTP (fallback)**.
|
|
DWD-impersonation sender = `CRM_DIGEST_SENDER` env, else the first active admin.
|
|
- **Gmail-DWD path: `gmail_send.py`** (this package) — reuses `credentials.py`'s
|
|
`DWDCredentialProvider` with the **`gmail.compose`** scope to call `users.messages.send`
|
|
(REST, mirrors `compose.py`; body is `{raw}` not the draft's `{message:{raw}}`). The
|
|
deployment's DWD grant includes `gmail.compose` (which authorizes send) but **not** the
|
|
narrow `gmail.send` — so request `gmail.compose`. Verified live 2026-06-15 (token mint +
|
|
a real `messages.send`).
|
|
- **SMTP fallback: `backend/smtp_send.py`** (top-level) — stdlib smtplib reading `SMTP_*` env,
|
|
populated on the box by the **Configure Digest SMTP** Start9 action (writes
|
|
`/data/secrets/smtp/*`; entrypoint exports `SMTP_*`). A dedicated per-package account,
|
|
independent of any StartOS system-wide SMTP.
|
|
- The admin **`POST /api/admin/digest/test-email`** restricts recipients to the active-admin
|
|
set (not an open relay), and logs send failures rather than echoing them (an auth error can
|
|
carry a token/credential).
|
|
|
|
### Phase B — the daily digest itself (built)
|
|
|
|
- **Content builder: `backend/digest_builder.py`** (top-level). `build_digest(conn, since_iso,
|
|
until_iso, chat_fn=None)` returns `{subject, body, has_activity, user_count, email_count,
|
|
investor_count}` and composes **two sections**:
|
|
- **By team member** — `collect_user_activity`: per registered user, both directions
|
|
(per-mailbox `eam.is_sent`), with **one Spark narrative paragraph** per user
|
|
(`ingest/llm.py` → Spark Control `/v1/chat/completions`), **never Claude** (the digest is
|
|
deliberately un-anonymized — real LP names + substance stay local). Deterministic
|
|
count-only fallback if Spark is unreachable (always-send must not fail).
|
|
- **By investor** — `collect_investor_activity`: re-pivots the same window across the whole
|
|
team, **deduped per email** (a reply to several teammates counts once), direction decided
|
|
at the **email level** (outbound if `from_email` is one of our mailboxes, else inbound).
|
|
Structured list, no extra Spark calls.
|
|
- Soft-delete filters: `email_account_messages.deleted_at IS NULL` + `users.is_active = 1`,
|
|
and the org/contact name joins drop soft-deleted rows (falling back to the matched address).
|
|
- **Control is DB-backed, set from the admin panel** — `digest_builder.load_digest_policy(conn)`
|
|
reads `app_settings.digest_policy` = `{enabled, send_hour}`. Precedence: **DB row wins**
|
|
(the Settings → Admin toggle + send-time dropdown), else `CRM_DIGEST_ENABLED`/
|
|
`CRM_DIGEST_SEND_HOUR` seed a first-boot default, else `{false, 18}`. `GET`/`PATCH
|
|
/api/admin/digest/policy` (admin-only) read/write it. **Not a StartOS action** — it's an
|
|
operational toggle, so it lives in-app where it's discoverable and takes effect live.
|
|
- **Scheduler: `backend/email_integration/digest_scheduler.py`** (co-located with the sync
|
|
scheduler). One daemon thread, **always started**; each cycle (60s) re-reads the DB policy
|
|
and sends once per local day at/after `send_hour` **only when `enabled`** — so toggling in
|
|
the panel takes effect with no restart. Content window = (last send, now]; cursor
|
|
(`digest_last_sent_at`) + once-per-day guard (`digest_last_sent_date`) live in `app_settings`,
|
|
so a missed day rolls into the next digest. Recipients = all active admins.
|
|
- **On-demand: `POST /api/admin/digest/send-now`** (admin-only) → `maybe_send_digest(force=True)`
|
|
builds the real last-24h digest and sends to the admin set regardless of the policy and
|
|
**without** touching the daily cursor (a preview never suppresses the scheduled send).
|
|
Surfaced as a "Send Digest Now" button in Settings → Admin, beside "Send Test Digest Email".
|
|
- **Decisions (locked):** 6 PM default send · always-send (empty days get a "no activity"
|
|
note) · per-user narrative + by-investor structured section · enable/time controlled in the
|
|
admin panel. Tests: `backend/test_digest_builder.py` (per-user + per-investor queries,
|
|
soft-delete, inbound dedup, two-section compose, fallback, policy resolver, scheduler guards
|
|
— stubbed LLM + transport).
|
|
|
|
## Email-activity panel (Communications tab) — admin-only
|
|
|
|
The **Communications** tab (frontend) is the admin-only search over captured Gmail. The
|
|
classic manual "Log Communication" form was retired (the Fundraising Grid context menu is
|
|
the manual-log path). Backed by **`GET /api/email/activity`** (`routes.py:_h_activity`,
|
|
`require_admin` server-side) → **`db.query_email_activity(conn, ...)`** (the pure, tested
|
|
query). Filters: `investor_id`, `account_id` (mailbox), `direction` (`inbound`/`outbound`),
|
|
`q` (free-text over subject/snippet/from). Non-obvious semantics to preserve:
|
|
|
|
- **Matched-only:** the panel surfaces ONLY email that links to a known
|
|
investor/contact (`query_email_activity` gates on `EXISTS email_investor_links`).
|
|
Capture still stores unmatched cold/unknown-sender email (metadata only, see "match-only
|
|
full storage"), but it is never shown here — the Communications tab is the
|
|
investor-relationship view, not the raw mailbox.
|
|
- **Soft-delete lives on the per-mailbox sighting**, not the email: `emails` has no
|
|
`deleted_at`. An email is "live" iff it has a sighting with `email_account_messages.
|
|
deleted_at IS NULL` — the query gates on `EXISTS(... deleted_at IS NULL)`. (Investor
|
|
links are email-level and carry no `deleted_at`, so they need no separate filter.)
|
|
- **Direction is decided at the email level** — outbound if `from_email` is one of our
|
|
`email_accounts` addresses, else inbound — mirroring `digest_builder._own_addresses`.
|
|
- **Graveyard investors** are hidden from the filter *dropdown* (CRM-wide `graveyard = 0`),
|
|
but their captured email still shows in the list and stays findable by free-text search —
|
|
it's an audit surface, so history is never hidden, only the picker is.
|
|
|
|
Tests: `backend/email_integration/test_email_activity_panel.py`.
|
|
|
|
## Known gap
|
|
|
|
- Tier-B drafts currently reply to the **LP only**; reply-all is the next change (see AGENTS.md → Current state).
|
|
|
|
See also `docs/gmail-enablement-runbook.md`.
|