Email search/query + windowed digest preview (v0.1.0:83)

Communications tab (search/query roadmap items 1 & 2):
- Fix the investor dropdown: the facet only listed grid investors, so it
  came back empty whenever email matched a classic contact or org domain
  (no grid id — the common case). It now mirrors the email list, resolving
  each link to a typed identity (fund:/org:/contact:/addr:) with precedence
  grid -> org -> contact -> address; investor_id accepts the typed key
  (bare id = fund: for back-compat) and an unknown prefix matches nothing.
- Add a date-range filter and a click-to-expand full-body view
  (GET /api/email/detail, admin, soft-delete-gated; body_text only, never
  raw remote HTML).
- Add a "Search content" mode: GET /api/email/search wraps the ingest
  hybrid_search over the Qdrant email index (doc_type=email), hydrated and
  soft-delete-filtered against SQLite (canonical), 503 if Spark/Qdrant down.

Daily digest:
- Settings -> Admin builds a digest over a chosen window (last 24h or since
  a date) as an in-app preview before sending (POST /api/admin/digest/preview),
  so the local-Spark summarizer can be verified on demand even on a quiet day.
  Manual send uses the same window; neither advances the daily cursor, so a
  preview never suppresses the scheduled digest.

Code-only, migrations no-op. 22/22 backend tests, render-smoke pass.
This commit is contained in:
Keysat
2026-06-16 20:46:15 -05:00
parent c29ac2f2ee
commit c7b74a2704
14 changed files with 989 additions and 138 deletions
+48 -5
View File
@@ -70,10 +70,21 @@ different category. **Never extend this path to send to LPs/prospects.**
the panel takes effect with no restart. Content window = (last send, now]; cursor
(`digest_last_sent_at`) + once-per-day guard (`digest_last_sent_date`) live in `app_settings`,
so a missed day rolls into the next digest. Recipients = all active admins.
- **On-demand: `POST /api/admin/digest/send-now`** (admin-only) → `maybe_send_digest(force=True)`
builds the real last-24h digest and sends to the admin set regardless of the policy and
**without** touching the daily cursor (a preview never suppresses the scheduled send).
Surfaced as a "Send Digest Now" button in Settings → Admin, beside "Send Test Digest Email".
- **Windowed preview + manual send (Settings → Admin "Manual run & preview"):**
- **`POST /api/admin/digest/preview`** (admin-only) builds the digest over a chosen window
and returns `{subject, body, …, window}` **without sending** — it runs the **real Spark
summarization**, so widening the window is how you verify the summarizer on a quiet day
(a last-24h window with no activity never calls Spark). Rendered in an in-panel preview.
- **`POST /api/admin/digest/send-now`** (admin-only) sends over the **same** window to the
admin set now.
- Both take the window from the body: default last 24h, `{"hours": N}`, or
`{"since": "YYYY-MM-DD"}` (a **local** date → that day's midnight). Resolved by
`digest_builder.resolve_digest_window` (capped at `MAX_WINDOW_DAYS`=92, validated → 400 on
bad input). The send goes through `digest_scheduler.send_digest_window`, which — like the
old `force=True` path — **does NOT advance the daily cursor**, so a wide manual preview/send
never suppresses the scheduled daily digest.
- The **"Send transport test"** button (`POST /api/admin/digest/test-email`) stays as a pure
pipe check (fixed message, admin-recipient-restricted).
- **Decisions (locked):** 6 PM default send · always-send (empty days get a "no activity"
note) · per-user narrative + by-investor structured section · enable/time controlled in the
admin panel. Tests: `backend/test_digest_builder.py` (per-user + per-investor queries,
@@ -103,8 +114,40 @@ query). Filters: `investor_id`, `account_id` (mailbox), `direction` (`inbound`/`
- **Graveyard investors** are hidden from the filter *dropdown* (CRM-wide `graveyard = 0`),
but their captured email still shows in the list and stays findable by free-text search —
it's an audit surface, so history is never hidden, only the picker is.
- **Typed investor facet (the dropdown).** The picker mirrors what the list resolves: one
entry per distinct matched entity, with the digest's precedence (**grid investor → org →
contact → raw address**) and a **typed key** — `fund:<id>` / `org:<id>` / `contact:<id>`
(`investor_id=` accepts these; a bare id is treated as `fund:` for back-compat). This fixed
the "dropdown only shows *All investors*" bug: matches that land on a **classic contact or
org domain** (no grid id — common, since `fundraising_contacts.email` is sparsely populated)
now resolve to a real name and appear in the picker, instead of the facet coming back empty.
Raw-address-only matches stay out of the *picker* (noisy) but still show + search in the list.
Helpers: `db._resolve_entity` + the shared `_LINK_IDENTITY_COLS`/`_LINK_IDENTITY_JOINS`.
- **Date range:** `since`/`until` filter `e.sent_at` as a half-open `[since, until)`
interval; the UI sends `from` as `…T00:00:00` and `to` as the **next day's** midnight,
so the whole "to" day is included regardless of the stored timestamp's precision/zone.
- **Detail view:** **`GET /api/email/detail?id=`** (`_h_detail`, `require_admin`) →
`db.query_email_detail` returns the full body + to/cc recipients + attachments + typed
identities, **soft-delete-gated on a live sighting** (404 otherwise). The UI renders
`body_text` (escaped) — **never** raw remote `body_html` (XSS); click a row to expand.
Tests: `backend/email_integration/test_email_activity_panel.py`.
## Content search (semantic, over email bodies) — admin-only
The Communications tab has a **Filter ⇄ Search content** toggle. "Search content" is semantic
search over the email *bodies* indexed in Qdrant (distinct from the structured subject/sender
LIKE filters above). **`GET /api/email/search?q=`** (`routes._h_search`, `require_admin`):
- Retrieval = `ingest/search.py:hybrid_search` (dense + BM25, reranked) pre-filtered to
`doc_type='email'`, imported **lazily** (the ingest stack — Spark Control + Qdrant + the
sparse encoder — ships in the Docker image, not the bare CRM); any failure → a clean **503**.
- Only **matched** email bodies are indexed (see `ingest/chunking.py`); the Qdrant payload
carries `source_id`=email_id, `lp_name`, `date_ts`, so hits link straight back to the row.
- **Hydrated + soft-delete-filtered against SQLite (canonical):** `db.search_hit_emails`
drops any hit whose email no longer has a live sighting — the derived index can lag a
deletion, and we never surface a fact from Qdrant that SQLite has tombstoned.
Tests: `backend/email_integration/test_email_activity_panel.py` (panel filters/facets/detail +
the search route's hydrate/drop/503/admin paths, with retrieval stubbed).
## Known gap