Keysat
c7b74a2704
Communications tab (search/query roadmap items 1 & 2): - Fix the investor dropdown: the facet only listed grid investors, so it came back empty whenever email matched a classic contact or org domain (no grid id — the common case). It now mirrors the email list, resolving each link to a typed identity (fund:/org:/contact:/addr:) with precedence grid -> org -> contact -> address; investor_id accepts the typed key (bare id = fund: for back-compat) and an unknown prefix matches nothing. - Add a date-range filter and a click-to-expand full-body view (GET /api/email/detail, admin, soft-delete-gated; body_text only, never raw remote HTML). - Add a "Search content" mode: GET /api/email/search wraps the ingest hybrid_search over the Qdrant email index (doc_type=email), hydrated and soft-delete-filtered against SQLite (canonical), 503 if Spark/Qdrant down. Daily digest: - Settings -> Admin builds a digest over a chosen window (last 24h or since a date) as an in-app preview before sending (POST /api/admin/digest/preview), so the local-Spark summarizer can be verified on demand even on a quiet day. Manual send uses the same window; neither advances the daily cursor, so a preview never suppresses the scheduled digest. Code-only, migrations no-op. 22/22 backend tests, render-smoke pass.
157 lines
11 KiB
Markdown
157 lines
11 KiB
Markdown
---
|
|
paths:
|
|
- backend/email_integration/**
|
|
- backend/digest_mailer.py
|
|
- backend/smtp_send.py
|
|
---
|
|
|
|
# Email capture & drafts (Gmail)
|
|
|
|
Read this before editing Gmail capture or draft creation.
|
|
|
|
## What it does
|
|
|
|
- `backend/email_integration/` captures Gmail via **domain-wide delegation** (`credentials.py`, `matcher.py`, `parser.py`, `db.py`, `sync.py`, `scheduler.py`, `routes.py`) and creates Tier-B in-thread drafts (`compose.py`). It has its own `migrations/`.
|
|
- Captured email becomes CRM activity through a **propose → approve** flow — nothing lands on a contact record until a human approves the proposal.
|
|
|
|
## Hard rule
|
|
|
|
- **Agents draft; humans send.** Never let an agent send email, post, or contact an LP autonomously. Tier-B `compose.py` only *creates* a Gmail draft for human review.
|
|
|
|
## Outbound mail — the daily digest (internal; exempt from "agents draft")
|
|
|
|
The CRM sends an internal **daily activity digest** to the fund's own admins. This is the
|
|
ONE automated send path, and it does **not** violate the hard rule above: that rule governs
|
|
outward **LP/prospect** contact. An internal ops email to the team's own inboxes is a
|
|
different category. **Never extend this path to send to LPs/prospects.**
|
|
|
|
- **Transport selector: `backend/digest_mailer.py`** (top-level, not in this package) —
|
|
`send_digest(conn, to_addrs, subject, body)` picks **Gmail-DWD (preferred) → SMTP (fallback)**.
|
|
DWD-impersonation sender = `CRM_DIGEST_SENDER` env, else the first active admin.
|
|
- **Gmail-DWD path: `gmail_send.py`** (this package) — reuses `credentials.py`'s
|
|
`DWDCredentialProvider` with the **`gmail.compose`** scope to call `users.messages.send`
|
|
(REST, mirrors `compose.py`; body is `{raw}` not the draft's `{message:{raw}}`). The
|
|
deployment's DWD grant includes `gmail.compose` (which authorizes send) but **not** the
|
|
narrow `gmail.send` — so request `gmail.compose`. Verified live 2026-06-15 (token mint +
|
|
a real `messages.send`).
|
|
- **SMTP fallback: `backend/smtp_send.py`** (top-level) — stdlib smtplib reading `SMTP_*` env,
|
|
populated on the box by the **Configure Digest SMTP** Start9 action (writes
|
|
`/data/secrets/smtp/*`; entrypoint exports `SMTP_*`). A dedicated per-package account,
|
|
independent of any StartOS system-wide SMTP.
|
|
- The admin **`POST /api/admin/digest/test-email`** restricts recipients to the active-admin
|
|
set (not an open relay), and logs send failures rather than echoing them (an auth error can
|
|
carry a token/credential).
|
|
|
|
### Phase B — the daily digest itself (built)
|
|
|
|
- **Content builder: `backend/digest_builder.py`** (top-level). `build_digest(conn, since_iso,
|
|
until_iso, chat_fn=None)` returns `{subject, body, has_activity, user_count, email_count,
|
|
investor_count}` and composes **two sections**:
|
|
- **By team member** — `collect_user_activity`: per registered user, both directions
|
|
(per-mailbox `eam.is_sent`), with **one Spark narrative paragraph** per user
|
|
(`ingest/llm.py` → Spark Control `/v1/chat/completions`), **never Claude** (the digest is
|
|
deliberately un-anonymized — real LP names + substance stay local). Deterministic
|
|
count-only fallback if Spark is unreachable (always-send must not fail).
|
|
- **By investor** — `collect_investor_activity`: re-pivots the same window across the whole
|
|
team, **deduped per email** (a reply to several teammates counts once), direction decided
|
|
at the **email level** (outbound if `from_email` is one of our mailboxes, else inbound).
|
|
Structured list, no extra Spark calls.
|
|
- Soft-delete filters: `email_account_messages.deleted_at IS NULL` + `users.is_active = 1`,
|
|
and the org/contact name joins drop soft-deleted rows (falling back to the matched address).
|
|
- **Control is DB-backed, set from the admin panel** — `digest_builder.load_digest_policy(conn)`
|
|
reads `app_settings.digest_policy` = `{enabled, send_hour}`. Precedence: **DB row wins**
|
|
(the Settings → Admin toggle + send-time dropdown), else `CRM_DIGEST_ENABLED`/
|
|
`CRM_DIGEST_SEND_HOUR` seed a first-boot default, else `{false, 18}`. `GET`/`PATCH
|
|
/api/admin/digest/policy` (admin-only) read/write it. **Not a StartOS action** — it's an
|
|
operational toggle, so it lives in-app where it's discoverable and takes effect live.
|
|
- **Scheduler: `backend/email_integration/digest_scheduler.py`** (co-located with the sync
|
|
scheduler). One daemon thread, **always started**; each cycle (60s) re-reads the DB policy
|
|
and sends once per local day at/after `send_hour` **only when `enabled`** — so toggling in
|
|
the panel takes effect with no restart. Content window = (last send, now]; cursor
|
|
(`digest_last_sent_at`) + once-per-day guard (`digest_last_sent_date`) live in `app_settings`,
|
|
so a missed day rolls into the next digest. Recipients = all active admins.
|
|
- **Windowed preview + manual send (Settings → Admin "Manual run & preview"):**
|
|
- **`POST /api/admin/digest/preview`** (admin-only) builds the digest over a chosen window
|
|
and returns `{subject, body, …, window}` **without sending** — it runs the **real Spark
|
|
summarization**, so widening the window is how you verify the summarizer on a quiet day
|
|
(a last-24h window with no activity never calls Spark). Rendered in an in-panel preview.
|
|
- **`POST /api/admin/digest/send-now`** (admin-only) sends over the **same** window to the
|
|
admin set now.
|
|
- Both take the window from the body: default last 24h, `{"hours": N}`, or
|
|
`{"since": "YYYY-MM-DD"}` (a **local** date → that day's midnight). Resolved by
|
|
`digest_builder.resolve_digest_window` (capped at `MAX_WINDOW_DAYS`=92, validated → 400 on
|
|
bad input). The send goes through `digest_scheduler.send_digest_window`, which — like the
|
|
old `force=True` path — **does NOT advance the daily cursor**, so a wide manual preview/send
|
|
never suppresses the scheduled daily digest.
|
|
- The **"Send transport test"** button (`POST /api/admin/digest/test-email`) stays as a pure
|
|
pipe check (fixed message, admin-recipient-restricted).
|
|
- **Decisions (locked):** 6 PM default send · always-send (empty days get a "no activity"
|
|
note) · per-user narrative + by-investor structured section · enable/time controlled in the
|
|
admin panel. Tests: `backend/test_digest_builder.py` (per-user + per-investor queries,
|
|
soft-delete, inbound dedup, two-section compose, fallback, policy resolver, scheduler guards
|
|
— stubbed LLM + transport).
|
|
|
|
## Email-activity panel (Communications tab) — admin-only
|
|
|
|
The **Communications** tab (frontend) is the admin-only search over captured Gmail. The
|
|
classic manual "Log Communication" form was retired (the Fundraising Grid context menu is
|
|
the manual-log path). Backed by **`GET /api/email/activity`** (`routes.py:_h_activity`,
|
|
`require_admin` server-side) → **`db.query_email_activity(conn, ...)`** (the pure, tested
|
|
query). Filters: `investor_id`, `account_id` (mailbox), `direction` (`inbound`/`outbound`),
|
|
`q` (free-text over subject/snippet/from). Non-obvious semantics to preserve:
|
|
|
|
- **Matched-only:** the panel surfaces ONLY email that links to a known
|
|
investor/contact (`query_email_activity` gates on `EXISTS email_investor_links`).
|
|
Capture still stores unmatched cold/unknown-sender email (metadata only, see "match-only
|
|
full storage"), but it is never shown here — the Communications tab is the
|
|
investor-relationship view, not the raw mailbox.
|
|
- **Soft-delete lives on the per-mailbox sighting**, not the email: `emails` has no
|
|
`deleted_at`. An email is "live" iff it has a sighting with `email_account_messages.
|
|
deleted_at IS NULL` — the query gates on `EXISTS(... deleted_at IS NULL)`. (Investor
|
|
links are email-level and carry no `deleted_at`, so they need no separate filter.)
|
|
- **Direction is decided at the email level** — outbound if `from_email` is one of our
|
|
`email_accounts` addresses, else inbound — mirroring `digest_builder._own_addresses`.
|
|
- **Graveyard investors** are hidden from the filter *dropdown* (CRM-wide `graveyard = 0`),
|
|
but their captured email still shows in the list and stays findable by free-text search —
|
|
it's an audit surface, so history is never hidden, only the picker is.
|
|
- **Typed investor facet (the dropdown).** The picker mirrors what the list resolves: one
|
|
entry per distinct matched entity, with the digest's precedence (**grid investor → org →
|
|
contact → raw address**) and a **typed key** — `fund:<id>` / `org:<id>` / `contact:<id>`
|
|
(`investor_id=` accepts these; a bare id is treated as `fund:` for back-compat). This fixed
|
|
the "dropdown only shows *All investors*" bug: matches that land on a **classic contact or
|
|
org domain** (no grid id — common, since `fundraising_contacts.email` is sparsely populated)
|
|
now resolve to a real name and appear in the picker, instead of the facet coming back empty.
|
|
Raw-address-only matches stay out of the *picker* (noisy) but still show + search in the list.
|
|
Helpers: `db._resolve_entity` + the shared `_LINK_IDENTITY_COLS`/`_LINK_IDENTITY_JOINS`.
|
|
- **Date range:** `since`/`until` filter `e.sent_at` as a half-open `[since, until)`
|
|
interval; the UI sends `from` as `…T00:00:00` and `to` as the **next day's** midnight,
|
|
so the whole "to" day is included regardless of the stored timestamp's precision/zone.
|
|
- **Detail view:** **`GET /api/email/detail?id=`** (`_h_detail`, `require_admin`) →
|
|
`db.query_email_detail` returns the full body + to/cc recipients + attachments + typed
|
|
identities, **soft-delete-gated on a live sighting** (404 otherwise). The UI renders
|
|
`body_text` (escaped) — **never** raw remote `body_html` (XSS); click a row to expand.
|
|
|
|
## Content search (semantic, over email bodies) — admin-only
|
|
|
|
The Communications tab has a **Filter ⇄ Search content** toggle. "Search content" is semantic
|
|
search over the email *bodies* indexed in Qdrant (distinct from the structured subject/sender
|
|
LIKE filters above). **`GET /api/email/search?q=`** (`routes._h_search`, `require_admin`):
|
|
|
|
- Retrieval = `ingest/search.py:hybrid_search` (dense + BM25, reranked) pre-filtered to
|
|
`doc_type='email'`, imported **lazily** (the ingest stack — Spark Control + Qdrant + the
|
|
sparse encoder — ships in the Docker image, not the bare CRM); any failure → a clean **503**.
|
|
- Only **matched** email bodies are indexed (see `ingest/chunking.py`); the Qdrant payload
|
|
carries `source_id`=email_id, `lp_name`, `date_ts`, so hits link straight back to the row.
|
|
- **Hydrated + soft-delete-filtered against SQLite (canonical):** `db.search_hit_emails`
|
|
drops any hit whose email no longer has a live sighting — the derived index can lag a
|
|
deletion, and we never surface a fact from Qdrant that SQLite has tombstoned.
|
|
|
|
Tests: `backend/email_integration/test_email_activity_panel.py` (panel filters/facets/detail +
|
|
the search route's hydrate/drop/503/admin paths, with retrieval stubbed).
|
|
|
|
## Known gap
|
|
|
|
- Tier-B drafts currently reply to the **LP only**; reply-all is the next change (see AGENTS.md → Current state).
|
|
|
|
See also `docs/gmail-enablement-runbook.md`.
|