Email search/query + windowed digest preview (v0.1.0:83)
Communications tab (search/query roadmap items 1 & 2): - Fix the investor dropdown: the facet only listed grid investors, so it came back empty whenever email matched a classic contact or org domain (no grid id — the common case). It now mirrors the email list, resolving each link to a typed identity (fund:/org:/contact:/addr:) with precedence grid -> org -> contact -> address; investor_id accepts the typed key (bare id = fund: for back-compat) and an unknown prefix matches nothing. - Add a date-range filter and a click-to-expand full-body view (GET /api/email/detail, admin, soft-delete-gated; body_text only, never raw remote HTML). - Add a "Search content" mode: GET /api/email/search wraps the ingest hybrid_search over the Qdrant email index (doc_type=email), hydrated and soft-delete-filtered against SQLite (canonical), 503 if Spark/Qdrant down. Daily digest: - Settings -> Admin builds a digest over a chosen window (last 24h or since a date) as an in-app preview before sending (POST /api/admin/digest/preview), so the local-Spark summarizer can be verified on demand even on a quiet day. Manual send uses the same window; neither advances the daily cursor, so a preview never suppresses the scheduled digest. Code-only, migrations no-op. 22/22 backend tests, render-smoke pass.
This commit is contained in:
+48
-5
@@ -70,10 +70,21 @@ different category. **Never extend this path to send to LPs/prospects.**
|
||||
the panel takes effect with no restart. Content window = (last send, now]; cursor
|
||||
(`digest_last_sent_at`) + once-per-day guard (`digest_last_sent_date`) live in `app_settings`,
|
||||
so a missed day rolls into the next digest. Recipients = all active admins.
|
||||
- **On-demand: `POST /api/admin/digest/send-now`** (admin-only) → `maybe_send_digest(force=True)`
|
||||
builds the real last-24h digest and sends to the admin set regardless of the policy and
|
||||
**without** touching the daily cursor (a preview never suppresses the scheduled send).
|
||||
Surfaced as a "Send Digest Now" button in Settings → Admin, beside "Send Test Digest Email".
|
||||
- **Windowed preview + manual send (Settings → Admin "Manual run & preview"):**
|
||||
- **`POST /api/admin/digest/preview`** (admin-only) builds the digest over a chosen window
|
||||
and returns `{subject, body, …, window}` **without sending** — it runs the **real Spark
|
||||
summarization**, so widening the window is how you verify the summarizer on a quiet day
|
||||
(a last-24h window with no activity never calls Spark). Rendered in an in-panel preview.
|
||||
- **`POST /api/admin/digest/send-now`** (admin-only) sends over the **same** window to the
|
||||
admin set now.
|
||||
- Both take the window from the body: default last 24h, `{"hours": N}`, or
|
||||
`{"since": "YYYY-MM-DD"}` (a **local** date → that day's midnight). Resolved by
|
||||
`digest_builder.resolve_digest_window` (capped at `MAX_WINDOW_DAYS`=92, validated → 400 on
|
||||
bad input). The send goes through `digest_scheduler.send_digest_window`, which — like the
|
||||
old `force=True` path — **does NOT advance the daily cursor**, so a wide manual preview/send
|
||||
never suppresses the scheduled daily digest.
|
||||
- The **"Send transport test"** button (`POST /api/admin/digest/test-email`) stays as a pure
|
||||
pipe check (fixed message, admin-recipient-restricted).
|
||||
- **Decisions (locked):** 6 PM default send · always-send (empty days get a "no activity"
|
||||
note) · per-user narrative + by-investor structured section · enable/time controlled in the
|
||||
admin panel. Tests: `backend/test_digest_builder.py` (per-user + per-investor queries,
|
||||
@@ -103,8 +114,40 @@ query). Filters: `investor_id`, `account_id` (mailbox), `direction` (`inbound`/`
|
||||
- **Graveyard investors** are hidden from the filter *dropdown* (CRM-wide `graveyard = 0`),
|
||||
but their captured email still shows in the list and stays findable by free-text search —
|
||||
it's an audit surface, so history is never hidden, only the picker is.
|
||||
- **Typed investor facet (the dropdown).** The picker mirrors what the list resolves: one
|
||||
entry per distinct matched entity, with the digest's precedence (**grid investor → org →
|
||||
contact → raw address**) and a **typed key** — `fund:<id>` / `org:<id>` / `contact:<id>`
|
||||
(`investor_id=` accepts these; a bare id is treated as `fund:` for back-compat). This fixed
|
||||
the "dropdown only shows *All investors*" bug: matches that land on a **classic contact or
|
||||
org domain** (no grid id — common, since `fundraising_contacts.email` is sparsely populated)
|
||||
now resolve to a real name and appear in the picker, instead of the facet coming back empty.
|
||||
Raw-address-only matches stay out of the *picker* (noisy) but still show + search in the list.
|
||||
Helpers: `db._resolve_entity` + the shared `_LINK_IDENTITY_COLS`/`_LINK_IDENTITY_JOINS`.
|
||||
- **Date range:** `since`/`until` filter `e.sent_at` as a half-open `[since, until)`
|
||||
interval; the UI sends `from` as `…T00:00:00` and `to` as the **next day's** midnight,
|
||||
so the whole "to" day is included regardless of the stored timestamp's precision/zone.
|
||||
- **Detail view:** **`GET /api/email/detail?id=`** (`_h_detail`, `require_admin`) →
|
||||
`db.query_email_detail` returns the full body + to/cc recipients + attachments + typed
|
||||
identities, **soft-delete-gated on a live sighting** (404 otherwise). The UI renders
|
||||
`body_text` (escaped) — **never** raw remote `body_html` (XSS); click a row to expand.
|
||||
|
||||
Tests: `backend/email_integration/test_email_activity_panel.py`.
|
||||
## Content search (semantic, over email bodies) — admin-only
|
||||
|
||||
The Communications tab has a **Filter ⇄ Search content** toggle. "Search content" is semantic
|
||||
search over the email *bodies* indexed in Qdrant (distinct from the structured subject/sender
|
||||
LIKE filters above). **`GET /api/email/search?q=`** (`routes._h_search`, `require_admin`):
|
||||
|
||||
- Retrieval = `ingest/search.py:hybrid_search` (dense + BM25, reranked) pre-filtered to
|
||||
`doc_type='email'`, imported **lazily** (the ingest stack — Spark Control + Qdrant + the
|
||||
sparse encoder — ships in the Docker image, not the bare CRM); any failure → a clean **503**.
|
||||
- Only **matched** email bodies are indexed (see `ingest/chunking.py`); the Qdrant payload
|
||||
carries `source_id`=email_id, `lp_name`, `date_ts`, so hits link straight back to the row.
|
||||
- **Hydrated + soft-delete-filtered against SQLite (canonical):** `db.search_hit_emails`
|
||||
drops any hit whose email no longer has a live sighting — the derived index can lag a
|
||||
deletion, and we never surface a fact from Qdrant that SQLite has tombstoned.
|
||||
|
||||
Tests: `backend/email_integration/test_email_activity_panel.py` (panel filters/facets/detail +
|
||||
the search route's hydrate/drop/503/admin paths, with retrieval stubbed).
|
||||
|
||||
## Known gap
|
||||
|
||||
|
||||
Reference in New Issue
Block a user