diff --git a/.env.example b/.env.example index 6503dff..b048799 100644 --- a/.env.example +++ b/.env.example @@ -64,6 +64,12 @@ MATRIX_INTAKE_ROOM=!: # whole email-review poll loop. The bot must be a member of this room. Needs the server side in the # s9pk (≥ v0.1.0:89) and the bot's CRM user set to role 'bot' (see docs/guides/matrix-intake.md). MATRIX_EMAIL_REVIEW_ROOM=!: +# Dedicated read-only Q&A room (W2): every top-level message here is answered as a natural-language +# query (translated on the box's LOCAL model — nothing leaves the box), no '?'/'@bot' trigger needed. +# The '?'/'@bot' trigger still also works in the intake room. Leave empty to disable the dedicated +# room (questions then go via the intake-room trigger). The bot must be a member of this room. Needs +# the server side in the s9pk (POST /api/query/nl) and the bot's CRM user set to role 'bot'. +MATRIX_QUERY_ROOM=!: # CRM write-back: the bot logs in as a DEDICATED service user (admin-created CRM user; # the CRM has no service-key path, so it uses normal Bearer-JWT auth). CRM_API_BASE=http://127.0.0.1:8080 diff --git a/AGENTS.md b/AGENTS.md index a5181c5..e77296f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -106,13 +106,13 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude ## Current state -_Phase 0 + Phase 1 built; **box live at v0.1.0:91; repo at v0.1.0:92** (reminders, deploy pending). **The fundraising grid + email capture is the canonical system of record.** Active thread: **W2 natural-language query** (backend done + validated locally; web/Matrix UI next). Deploy/feature history: git log + `start9/0.4/startos/versions/`; longer-term backlog/debt: `ROADMAP.md` / `EVALUATION.md`._ +_Phase 0 + Phase 1 built; **box live at v0.1.0:91; repo at v0.1.0:92** (reminders, deploy pending). **The fundraising grid + email capture is the canonical system of record.** Active thread: **W2 natural-language query** (backend + Matrix `@bot` surface built; web "Ask" box next). Deploy/feature history: git log + `start9/0.4/startos/versions/`; longer-term backlog/debt: `ROADMAP.md` / `EVALUATION.md`._ -- **W2 — natural-language query (read-only): BACKEND built + tested + validated locally 2026-06-18; web/Matrix UI next.** `backend/nl_query/` — 12 curated parameterized queries + a slot validator (the trust boundary; no generic SQL) + a **local-Qwen** translator (question→{intent,slots} via Spark Control; nothing leaves the box, **no Claude, no redaction** — the simplification Grant chose). `POST /api/query/nl` (also accepts direct `{intent,slots}`) + `GET /api/query/catalog`, `require_bot_or_admin`, audited (`entity_type='nl_query'`). **Local Qwen translated 12/12 of Grant's real example questions correctly against the live Spark** — settles local-only (Claude not needed). Soft-delete-correct per table (gotcha: `fundraising_*` has **no `deleted_at`** — `graveyard` is the axis; emails via a live `eam` sighting). Guide: `docs/guides/nl-query.md`. **Next: step 4 web "Ask" box (Communications tab), step 5 Matrix `@bot `** — both thin clients of the endpoint. Not committed at deploy/version level yet; folds into the next s9pk. +- **W2 — natural-language query (read-only): BACKEND + MATRIX `@bot` surface built + tested locally 2026-06-18; web "Ask" box next.** `backend/nl_query/` — 12 curated parameterized queries + a slot validator (the trust boundary; no generic SQL) + a **local-Qwen** translator (question→{intent,slots} via Spark Control; nothing leaves the box, **no Claude, no redaction** — the simplification Grant chose). `POST /api/query/nl` (also accepts direct `{intent,slots}`) + `GET /api/query/catalog`, `require_bot_or_admin`, audited (`entity_type='nl_query'`). **Local Qwen translated 12/12 of Grant's real example questions correctly against the live Spark** — settles local-only (Claude not needed). Soft-delete-correct per table (gotcha: `fundraising_*` has **no `deleted_at`** — `graveyard` is the axis; emails via a live `eam` sighting). Guide: `docs/guides/nl-query.md`. **Step 5 (Matrix Q&A) DONE** — thin client in `backend/matrix_intake/query.py` (trigger grammar + answer rendering) + `crm_client.nl_query` + `bot.py` wiring, read-only (no approval gate), tested in `test_query.py`. **Two entry points (room-per-purpose model):** a **dedicated Q&A room** (`MATRIX_QUERY_ROOM`) where every message is a question, **and** the `?`/`@bot` trigger still working in the intake room as a cross-room convenience. Ships on the **Spark** (git pull + restart, no s9pk for the bot). Q&A room `!RGlJEObVaIUtUVcHtx:matrix.gilliam.ai` created + bot invited (2026-06-18). **BUT the box-side `/api/query/nl` endpoint is NOT live yet** (box v91; verified 404 on 2026-06-18) — it lands with the **v93 s9pk** (reminders + W2). **So DON'T activate the bot deploy (set `MATRIX_QUERY_ROOM` + restart) until v93 is installed**, or every question 404s. Code committed + pushed; bot deploy is staged to follow the v93 install. **Next: step 4 web "Ask" box (Communications tab)** — the last thin client. - **W1 — reminders & follow-ups: BUILT + tested locally (v0.1.0:92), DEPLOY PENDING.** First-class tickler tied to the grid (migration `0006`; CRUD `GET/POST/PATCH/DELETE /api/reminders`; derived `reminder_status` grid column; Reminders page + dashboard card + digest section; the `last_activity_at` recency rollup that W2 reuses). Needs s9pk build + install (authorize first; verify `0006` against a DB copy). Deferred **W1b** = nurture-gap auto-suggested reminders. - **Done & live (detail in git log / ROADMAP):** email-proposal Matrix review + `bot` role (box v91); grid-driven Pipeline (v88); Matrix intake bot (Spark `matrix-intake` container); Gmail capture (DWD) + propose→approve + daily digest; Thesis Workshop + Architect (Claude, dual-approval); outreach drafts + radar. All draft-only. -- **Tests:** **34/34 backend green** (`python3 backend/run_tests.py`; +`nl_query/` suite), `py_compile` clean; render-smoke gates `make`. -- **Next (priority order):** 1) **W2 step 4** web Ask box, then **step 5** Matrix `@bot`; 2) **deploy reminders (v92) + W2 together** — bump to **v0.1.0:93**, build s9pk, install, browser-verify (authorize first; verify `0006` against a DB copy); 3) **W3** bot grid-mutations behind the Matrix approval gate (local-Qwen parse); 4) **W1b** nurture-gap reminders; 5) Grant + Jonathan freeze v2.0 canonical; 6) in-room smoke of the intake disambiguation numbered-pick grammar; then P2 debt (reports comms-aggregate soft-delete sweep, `?limit=abc` crash, auth regression test, oversized StartOS icon). +- **Tests:** **35/35 backend green** (`python3 backend/run_tests.py`; +`nl_query/` + matrix `test_query.py` suites), `py_compile` clean; render-smoke gates `make`. +- **Next (priority order):** 1) **deploy reminders (v92) + W2 together** — bump to **v0.1.0:93**, build s9pk, install, browser-verify (authorize first; verify `0006` against a DB copy) — **this is the gate for the Matrix Q&A: the bot's step-5 surface 404s until `/api/query/nl` is on the box**; THEN activate the bot deploy (set `MATRIX_QUERY_ROOM` on the Spark + git pull + restart) + in-room smoke; 2) **W2 step 4** web Ask box (last NL-query client); 3) **W3** bot grid-mutations behind the Matrix approval gate (local-Qwen parse); 4) **W1b** nurture-gap reminders; 5) Grant + Jonathan freeze v2.0 canonical; 6) in-room smoke of the intake disambiguation numbered-pick grammar; then P2 debt (reports comms-aggregate soft-delete sweep, `?limit=abc` crash, auth regression test, oversized StartOS icon). - **Open / risks:** W2 translation only **happy-path-validated** (typos/ambiguous/no-match phrasings shake out in live use); **Claude/Architect path still unverified live on the box**; v2.0 reserve-asset spine is the *working approved* spine but **not canonical** (needs dual sign-off); doc drift — `crm-overview.md` + `EVALUATION.md` still call `lp_profiles` live. diff --git a/backend/matrix_intake/bot.py b/backend/matrix_intake/bot.py index 08f4c7a..f9366c3 100644 --- a/backend/matrix_intake/bot.py +++ b/backend/matrix_intake/bot.py @@ -18,6 +18,7 @@ import email_proposals import matrix_io import parse import proposals +import query import settings UNCLEAR_HELP = ( @@ -42,6 +43,7 @@ async def main(): if roster: print(f"matrix-intake: team roster loaded ({len(roster)} names)", flush=True) review_room = settings.email_review_room() # CRM-drafted email proposals (empty → feature off) + query_room = settings.query_room() # dedicated read-only Q&A room (empty → use the intake trigger) email_threads = {} # Matrix thread-root event_id -> {id, investor_name, note} for an email proposal async def handle_intake(room_id, root, text): @@ -97,6 +99,19 @@ async def main(): # easy to miss inside a thread (the full card + yes/edit/no stay in the thread). await nudge(room_id, proposals.summary_line(proposal), root) + async def handle_query(room_id, root, question): + """A read-only NL question ('@bot …' / '?…') — translate + run it on the BOX (local Qwen, + nothing leaves the box) and post the answer in a thread. No write path, no approval gate: + it only reads curated, parameterized queries. The endpoint returns its structured result + even on a soft no-match / model-down, so we render that; a transport/auth failure raises + and we show a brief error.""" + try: + result = await asyncio.to_thread(crm_client.nl_query, question) + except Exception as exc: + await say(room_id, f"⚠️ couldn't run that query: {str(exc)[:200]}", root) + return + await say(room_id, query.render_answer(result), root) + async def handle_reply(room_id, root, text): # Claim the proposal synchronously — BEFORE any await — so a second reply that # arrives while a commit is in flight can't double-process it. asyncio is @@ -299,6 +314,12 @@ async def main(): if root and root in email_threads: await handle_email_reply(room.room_id, root, text) return + # Dedicated Q&A room: every top-level message IS a question — no trigger needed. Threaded + # messages (the answers we post, or follow-ups) aren't acted on in v1. + if query_room and room.room_id == query_room: + if not root: + await handle_query(room.room_id, event.event_id, text) + return if room.room_id != intake_room: return if root and store.has(root): @@ -306,7 +327,15 @@ async def main(): elif root: return # threaded message not tied to a live proposal — ignore else: - await handle_intake(room.room_id, event.event_id, text) + # A top-level message is either an NL question (explicitly addressed with '?'/'@bot') + # or an intake note. The trigger is required, so plain notes still flow to intake. + q = query.parse_trigger(text) + if q is None: + await handle_intake(room.room_id, event.event_id, text) + elif not q: + await say(room.room_id, query.HELP, event.event_id) + else: + await handle_query(room.room_id, event.event_id, q) # Prime the sync token past history, THEN register the callback — only react to messages # arriving after startup (no backlog replay). (matrix-bridge pattern.) @@ -325,6 +354,14 @@ async def main(): print(f"matrix-intake: could not join review room {review_room}: {exc}", flush=True) tasks.append(asyncio.create_task(poll_email_proposals())) print(f"matrix-intake: reviewing email proposals in room {review_room} (every {EMAIL_POLL_SEC}s)", flush=True) + if query_room: + # Read-only Q&A room — just join and listen (no poll task; questions are interactive). + # "Invited" isn't "joined": the bot must join before it can post answers (idempotent). + try: + await client.join(query_room) + except Exception as exc: + print(f"matrix-intake: could not join Q&A room {query_room}: {exc}", flush=True) + print(f"matrix-intake: answering questions in room {query_room}", flush=True) try: await asyncio.gather(*tasks) finally: diff --git a/backend/matrix_intake/crm_client.py b/backend/matrix_intake/crm_client.py index e79d70d..2f83d08 100644 --- a/backend/matrix_intake/crm_client.py +++ b/backend/matrix_intake/crm_client.py @@ -139,6 +139,19 @@ def decide_email_proposal(proposal_id, decision, note=None): return data.get("data") or {} +def nl_query(question): + """Ask the read-only NL-query endpoint (POST /api/query/nl). Translation runs on the box's + LOCAL model — the question never leaves the box and no write is possible. Returns the + endpoint's structured result dict ({intent, slots, rows, summary, ...} or {error, detail}); + the server returns that same body on a hit AND on the soft 503 (model down) / 500 (query + fault) status codes, so we hand it straight to the renderer. Any OTHER status — auth (403), + a malformed request (400), an unexpected shape — raises so the caller posts a brief error.""" + status, data = _authed("POST", "/api/query/nl", {"question": question, "source": "matrix"}) + if status not in (200, 500, 503): + raise RuntimeError(f"nl-query failed ({status}): {data.get('error') or data}") + return data.get("data") or {} + + def build_commit_payload(proposal): """Pure: map a proposal to the /api/fundraising/log-communication request body. diff --git a/backend/matrix_intake/query.py b/backend/matrix_intake/query.py new file mode 100644 index 0000000..4b7c299 --- /dev/null +++ b/backend/matrix_intake/query.py @@ -0,0 +1,189 @@ +"""NL-query Matrix surface (W2 step 5) — turn an '@bot ' message into a read-only +answer from the CRM's curated NL-query endpoint, and render that answer for the chat room. + +This module is PURE (no network, no matrix-nio) so it's unit-testable offline; the async wiring +(call the endpoint, post in a thread) lives in bot.py. The endpoint does the real work: +translation runs on the box's LOCAL model (nothing leaves the box) and only the curated, +parameterized queries can run — there is no write path here, so no approval gate applies. + +Trigger: a top-level message starting with '?' / '@bot' / '/ask' (see parse_trigger). We +deliberately do NOT accept a bare leading 'ask', which would collide with intake notes like +"Ask Jane to send the Q3 deck". +""" + +# Markers a human wouldn't start an intake note with. '?' is handled separately (single char). +QUERY_PREFIXES = ("@bot", "/ask", "/query", "/q") + +# Soft cap on rows rendered into a single chat answer. The endpoint already caps the SQL result +# (server MAX_ROWS), but 500 rows is unreadable on mobile — show the first N and say how many +# more there are (never a silent cut). Refine the question or use the web Ask box for the rest. +MAX_DISPLAY_ROWS = 30 + +# Column-name hints used only for nicer formatting (money / dates). Cosmetic — never affects +# what's queried (that's fixed in intents.py). +_MONEY_HINTS = ("amount", "invested", "total", "expected", "committed") +# 0/1 flag columns: suppress when 0 (noise), show a label when 1. +_FLAG_LABELS = {"graveyard": "retired", "overdue": "⚠️ overdue"} + + +def parse_trigger(text): + """If `text` is addressed to the query bot, return the question (the remainder after the + trigger, possibly an empty string when the trigger is bare). Return None if it isn't a query, + so the caller routes it to intake instead.""" + s = (text or "").strip() + if not s: + return None + if s[0] == "?": + return s[1:].strip() + low = s.lower() + for p in QUERY_PREFIXES: + if low.startswith(p): + rest = s[len(p):] + # Require a separator so '/asking …' isn't read as the '/ask' trigger. + if rest == "" or rest[0] in " \t\n:,": + return rest.lstrip(" \t\n:,").strip() + return None + + +def _examples(): + return ("Try things like:\n" + "• `?which investors haven't we contacted in 90 days?`\n" + "• `?top 10 investors by committed capital`\n" + "• `?when did we last reach out to Acme Capital?`\n" + "• `?how many emails has Grant sent this month?`") + + +HELP = ("💬 Ask me about the fundraising database — start your message with `?` (or `@bot`).\n\n" + + _examples()) + + +def _is_money_col(name): + n = name.lower() + return any(h in n for h in _MONEY_HINTS) + + +def _fmt_value(col, val): + """Format one scalar cell for chat: dates -> YYYY-MM-DD, money columns -> $1,234, else str.""" + if val is None: + return "" + name = col.lower() + if name.endswith("_at") or name.endswith("date"): + return str(val)[:10] + if isinstance(val, (int, float)) and _is_money_col(col): + return f"${val:,.0f}" + return str(val) + + +def _render_contacts(contacts): + """investor_lookup's nested contact dicts -> 'Name (title · city, state)' lines.""" + out = [] + for c in contacts: + bits = c.get("full_name") or "?" + if c.get("email"): + bits += f" <{c['email']}>" + loc = ", ".join(x for x in (c.get("city"), c.get("state"), c.get("country")) if x) + extra = " · ".join(x for x in (c.get("title"), loc) if x) + if extra: + bits += f" ({extra})" + out.append(bits) + return out + + +def _render_commitments(commitments): + """investor_lookup's nested per-fund commitments -> 'Fund: $amount' lines.""" + out = [] + for c in commitments: + fund = c.get("fund_name") or "?" + amt = c.get("amount") + out.append(f"{fund}: ${amt:,.0f}" if isinstance(amt, (int, float)) else f"{fund}: {amt}") + return out + + +def _render_row(i, row, columns): + cols = columns or list(row.keys()) + lead = None + scalars = [] + sublines = [] + for col in cols: + val = row.get(col) + if isinstance(val, list): + if not val: + continue + if col == "contacts": + sublines += [f" – {x}" for x in _render_contacts(val)] + elif col == "commitments": + sublines += [f" – {x}" for x in _render_commitments(val)] + else: # generic list-of-dicts fallback (no intent uses this yet) + sublines += [f" – {', '.join(f'{k}={v}' for k, v in d.items())}" + for d in val if isinstance(d, dict)] + continue + if col in _FLAG_LABELS: + if val: + scalars.append(_FLAG_LABELS[col]) + continue + s = _fmt_value(col, val) + if s == "": + continue + if lead is None: # first non-empty column is the bold identifier for the row + lead = s + else: + scalars.append(f"{col}: {s}") + head = f"{i}. **{lead}**" if lead else f"{i}." + if scalars: + head += " — " + " · ".join(scalars) + return "\n".join([head] + sublines) + + +def _render_interpretation(intent, slots): + if not intent: + return "" + if slots: + return f"read as: {intent} ({', '.join(f'{k}={v}' for k, v in slots.items())})" + return f"read as: {intent}" + + +def _render_error(err, result): + detail = (result.get("detail") or "").strip() + if err == "no_match": + return "🤷 I couldn't map that to one of my saved queries.\n\n" + _examples() + if err == "model_unavailable": + return "⚠️ The local query model is unreachable right now — try again in a moment." + if err == "query_failed": + return f"⚠️ That query failed to run{(': ' + detail) if detail else ''}." + # unknown_intent / bad_slot / anything unexpected + return (f"⚠️ I couldn't run that ({err}){(': ' + detail) if detail else ''}.\n\n" + _examples()) + + +def render_answer(result): + """Render the NL-query endpoint's structured result into a Matrix markdown answer. + + `result` is the endpoint body: a hit {intent, slots, columns, rows, summary, truncated} or + an error {error, detail}. Results never go back to any model — this is a deterministic format.""" + result = result or {} + err = result.get("error") + if err: + return _render_error(err, result) + + summary = (result.get("summary") or "").strip() + rows = result.get("rows") or [] + columns = result.get("columns") or [] + header = f"📊 {summary}" if summary else "📊 Done." + interp = _render_interpretation(result.get("intent"), result.get("slots") or {}) + if interp: + header += f"\n_{interp}_" + if not rows: + return header + "\n\n(no matching records)" + + shown = rows[:MAX_DISPLAY_ROWS] + blocks = [_render_row(i + 1, r, columns) for i, r in enumerate(shown)] + out = header + "\n\n" + "\n".join(blocks) + + notes = [] + extra = len(rows) - len(shown) + if extra > 0: + notes.append(f"+{extra} more not shown") + if result.get("truncated"): + notes.append("results hit the server cap") + if notes: + out += "\n\n_" + "; ".join(notes) + " — refine your question or use the web Ask box._" + return out diff --git a/backend/matrix_intake/settings.py b/backend/matrix_intake/settings.py index ed1aff4..c82cee4 100644 --- a/backend/matrix_intake/settings.py +++ b/backend/matrix_intake/settings.py @@ -68,3 +68,13 @@ def team_roster(): # intake flow. Unset/empty disables the whole email-review poll loop (the bot just does intake). def email_review_room(): return os.environ.get("MATRIX_EMAIL_REVIEW_ROOM", "").strip() + + +# Dedicated Q&A room for read-only natural-language queries (W2). In this room EVERY top-level +# message is treated as a question — no '?'/'@bot' trigger needed (the trigger only exists to +# disambiguate question-vs-note when Q&A shares the intake room; here that's unnecessary). The +# '?'/'@bot' trigger still works in the intake room too, as a cross-room convenience. Unset/empty +# just means no dedicated room (questions then go through the intake-room trigger). The bot must be +# a member of this room. Read-only — no approval gate, no redaction, no special power level needed. +def query_room(): + return os.environ.get("MATRIX_QUERY_ROOM", "").strip() diff --git a/backend/matrix_intake/test_crm_client.py b/backend/matrix_intake/test_crm_client.py index 2d4baf1..d9e5101 100644 --- a/backend/matrix_intake/test_crm_client.py +++ b/backend/matrix_intake/test_crm_client.py @@ -113,6 +113,41 @@ def test_match_no_query_skips_network(): assert res == {"match": None, "candidates": []} +def test_nl_query_returns_endpoint_data(): + cap = {} + orig = _with_stub_authed( + (200, {"data": {"intent": "top_investors_committed", "rows": [], "summary": "ok"}}), cap) + try: + res = crm_client.nl_query("top investors") + finally: + crm_client._authed = orig + assert res["intent"] == "top_investors_committed" + assert cap["path"] == "/api/query/nl" + + +def test_nl_query_passes_through_soft_503(): + # Model-down still carries a structured body (the endpoint 503s with the error in `data`) — + # return it for the renderer to surface, don't raise. + orig = _with_stub_authed((503, {"data": {"error": "model_unavailable"}})) + try: + res = crm_client.nl_query("anything") + finally: + crm_client._authed = orig + assert res["error"] == "model_unavailable" + + +def test_nl_query_raises_on_auth_failure(): + orig = _with_stub_authed((403, {"error": "Bot or admin required"})) + raised = False + try: + crm_client.nl_query("x") + except RuntimeError: + raised = True + finally: + crm_client._authed = orig + assert raised + + if __name__ == "__main__": fns = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)] for fn in fns: diff --git a/backend/matrix_intake/test_query.py b/backend/matrix_intake/test_query.py new file mode 100644 index 0000000..a94880f --- /dev/null +++ b/backend/matrix_intake/test_query.py @@ -0,0 +1,112 @@ +"""Tests for the NL-query Matrix surface: trigger detection + answer rendering (pure, no network).""" +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import query # noqa: E402 + + +# ── parse_trigger ─────────────────────────────────────────────────────────────────────── +def test_trigger_question_mark(): + assert query.parse_trigger("?who are our top investors") == "who are our top investors" + assert query.parse_trigger(" ? spaced out ") == "spaced out" + + +def test_trigger_at_bot(): + assert query.parse_trigger("@bot top 10 investors") == "top 10 investors" + assert query.parse_trigger("@bot: top 10 investors") == "top 10 investors" # pill-style colon + assert query.parse_trigger("@BOT spaced") == "spaced" # case-insensitive + + +def test_trigger_slash_forms(): + assert query.parse_trigger("/ask when did we last email Acme?") == "when did we last email Acme?" + assert query.parse_trigger("/query top investors") == "top investors" + assert query.parse_trigger("/q top investors") == "top investors" + + +def test_trigger_bare_returns_empty_string(): + # A bare trigger is matched (so we show help) but carries no question. + assert query.parse_trigger("@bot") == "" + assert query.parse_trigger("?") == "" + + +def test_non_trigger_routes_to_intake(): + assert query.parse_trigger("New investor: Acme — Jane ") is None + # 'ask' as a note verb must NOT trigger (would collide with real intake notes). + assert query.parse_trigger("Ask Jane to send the Q3 deck") is None + assert query.parse_trigger("/asking for a friend") is None # needs a separator after /ask + assert query.parse_trigger("") is None + assert query.parse_trigger(" ") is None + + +# ── render_answer ─────────────────────────────────────────────────────────────────────── +def test_render_scalar_rows(): + out = query.render_answer({ + "intent": "top_investors_committed", "slots": {"limit": 2}, + "summary": "Top 2 investor(s) by committed capital.", + "columns": ["investor_name", "total_invested", "lead"], + "rows": [{"investor_name": "Acme Capital", "total_invested": 5000000, "lead": "Grant"}, + {"investor_name": "Beta Fund", "total_invested": 2500000, "lead": "Jonathan"}], + "truncated": False}) + assert "Top 2 investor(s)" in out + assert "**Acme Capital**" in out + assert "$5,000,000" in out # money formatting + assert "read as: top_investors_committed" in out # interpretation footer + + +def test_render_nested_contacts_and_commitments(): + out = query.render_answer({ + "intent": "investor_lookup", "slots": {"name": "Acme"}, + "summary": '1 investor(s) matching "Acme".', + "columns": ["investor_name", "lead", "total_invested", "graveyard", "contacts", "commitments"], + "rows": [{"investor_name": "Acme Capital", "lead": "Grant", "total_invested": 5000000, + "graveyard": 0, + "contacts": [{"full_name": "Jane Doe", "email": "jane@acme.com", "title": "GP", + "city": "Austin", "state": "TX", "country": ""}], + "commitments": [{"fund_name": "Fund I", "amount": 5000000}]}], + "truncated": False}) + assert "Jane Doe " in out + assert "Fund I: $5,000,000" in out + assert "graveyard" not in out # 0-valued flag column suppressed + + +def test_render_flag_when_set(): + out = query.render_answer({ + "intent": "investors_follow_up", "slots": {}, + "summary": "1 investor(s) with an open follow-up reminder.", + "columns": ["investor_name", "title", "due_date", "status", "overdue"], + "rows": [{"investor_name": "Acme", "title": "Send deck", "due_date": "2026-01-01", + "status": "open", "overdue": 1}]}) + assert "⚠️ overdue" in out + assert "2026-01-01" in out # date truncated to YYYY-MM-DD + + +def test_render_no_rows(): + out = query.render_answer({"intent": "investors_by_city", "slots": {"city": "Nowhere"}, + "summary": '0 investor contact(s) in "Nowhere".', + "columns": [], "rows": []}) + assert "no matching" in out.lower() + + +def test_render_overflow_note(): + rows = [{"investor_name": f"Inv {i}", "total_invested": i} + for i in range(query.MAX_DISPLAY_ROWS + 5)] + out = query.render_answer({"intent": "top_investors_committed", "slots": {}, "summary": "many", + "columns": ["investor_name", "total_invested"], "rows": rows}) + assert "+5 more not shown" in out + + +def test_render_errors(): + assert "couldn't map" in query.render_answer({"error": "no_match", "question": "huh"}).lower() + assert "unreachable" in query.render_answer({"error": "model_unavailable"}).lower() + assert "failed" in query.render_answer({"error": "query_failed", "detail": "boom"}).lower() + assert "bad_slot" in query.render_answer({"error": "bad_slot", "detail": "x"}) + + +if __name__ == "__main__": + fns = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)] + for fn in fns: + fn() + print(f"ok {fn.__name__}") + print(f"\n{len(fns)} passed") diff --git a/docs/guides/matrix-intake.md b/docs/guides/matrix-intake.md index 0211bb9..b1befe2 100644 --- a/docs/guides/matrix-intake.md +++ b/docs/guides/matrix-intake.md @@ -145,6 +145,43 @@ the web panel. The CRM (box, stdlib, no matrix-nio) can't post to Matrix, so the won't route — re-send/decide the recorded one). A mid-revise bot restart loses the in-memory revised note (rebuilt from `open` = the original `proposed_note`; still a valid proposal). +## NL query — read-only Q&A (W2 step 5) + +A read-only "ask the database in plain English" flow, answered in-thread. **No write path, no +approval gate** — it only runs the curated, parameterized queries behind the CRM's NL-query +endpoint, so it's exempt from the draft→approve dance the write flows need. Two entry points, +same `handle_query` → `crm_client.nl_query` underneath: + +- **Dedicated Q&A room** (`MATRIX_QUERY_ROOM`, recommended) — **every** top-level message is a + question; no trigger needed. This is the room-per-purpose model (intake / email-review / Q&A, + with a future reminders-push room): the trigger grammar below exists *only* to disambiguate + question-vs-note when Q&A shares the intake room, which a dedicated room makes unnecessary. The + simplest room of the three — read-only, no approval, no redaction, **no special power level**. +- **`@bot`/`?` trigger in the intake room** (cross-room convenience) — fire a quick question + without switching rooms. `query.parse_trigger` (pure/tested) matches a top-level message starting + with `?`, `@bot`, `/ask`, `/query`, or `/q`. The trigger is **required** there, so plain intake + notes still route to intake. A bare leading `ask` is deliberately **not** a trigger — it would + collide with notes like *"Ask Jane to send the deck"*. A bare trigger (`@bot` alone) posts help. +- **One endpoint call** (`crm_client.nl_query` → `POST /api/query/nl {question, source:"matrix"}`): + translation runs on the box's **local Qwen** (nothing leaves the box; **no Claude, no scrub** — + same basis as intake) and only the fixed `nl_query` catalog can run. The bot is a thin client — + see `docs/guides/nl-query.md` for the trust model. +- **Rendering** (`query.render_answer`, pure/tested): a deterministic Matrix-markdown answer + (summary + interpreted intent + compact rows, money/date formatting, nested contacts/commitments + for `investor_lookup`). **Results never go back to any model.** Mobile soft-cap `MAX_DISPLAY_ROWS` + (30) with an explicit "+N more" note — never a silent cut. +- **Status passthrough:** the endpoint returns its structured body on a hit *and* on the soft + 503 (model down) / 500 (query fault) codes, so `nl_query` hands those to the renderer; only an + auth/shape failure (403/400) raises → a brief ⚠️ in-thread. +- **Ships on the Spark** (bot-side, `query.py` + `crm_client.nl_query` + `bot.py` wiring) via + `git pull` + restart — **no s9pk for the bot**. **But it depends on the box-side `/api/query/nl` + endpoint**, which ships in the s9pk and is **not live until v93** (reminders + W2). Deploying the + bot before that = a Q&A room that 404s every question (same server-side/bot split as the v83→v84 + `/api/intake/match` 404). **Sequence: install v93 first, then** set `MATRIX_QUERY_ROOM` + invite + the bot + restart. Pure logic tested in `test_query.py` (+ `nl_query` cases in + `test_crm_client.py`); the in-room smoke (a bare message in the Q&A room, or `?…` in the intake + room) is live-only. + ## Rules / gotchas - **Module-name collision:** the intake config module is `settings.py`, **not** `config.py`, @@ -236,6 +273,13 @@ the ingest client (`SPARK_CONTROL_URL`, `CRM_CHAT_MODEL`). - **`MATRIX_EMAIL_REVIEW_ROOM`** (optional) — the dedicated room for the email-activity proposal review leg (above). Unset/empty disables that leg entirely (the bot does intake only). The bot must be invited to + joined in this room. Read once at startup, like the room/roster. + +- **`MATRIX_QUERY_ROOM`** (optional) — the dedicated read-only Q&A room (NL query section above). + In it, every top-level message is answered as a query (no `?`/`@bot` trigger). Unset/empty just + means no dedicated room — questions still work via the trigger in the intake room. The bot must be + invited to + joined in this room (`settings.query_room()`, read once at startup). No poll loop and + no power level needed (read-only). Needs the server side in the s9pk (`POST /api/query/nl`, ≥ the + W2 backend) and the bot's CRM user set to role `bot`. - **Bot CRM user needs role `bot`.** The email-proposal endpoints (`/api/intake/email-proposals*`) are gated to `require_bot_or_admin` because they expose LP email content (the proposals are admin-only on the web). The `bot` role is **authenticated-but-not-admin** — it passes these diff --git a/docs/guides/nl-query.md b/docs/guides/nl-query.md index 7b5505b..8459908 100644 --- a/docs/guides/nl-query.md +++ b/docs/guides/nl-query.md @@ -54,6 +54,13 @@ Keep the two in sync; the soft-delete test guards the copy. - `POST /api/query/nl` (`require_bot_or_admin`, read-only) — body `{question}` (local translate) or `{intent, slots}` (direct, e.g. a UI re-run). Returns `{intent, slots, rows, summary, question}`. `GET /api/query/catalog` returns the askable surface for the UI. +- **Clients (thin):** the **Matrix Q&A** surface is built — it lives bot-side in + `backend/matrix_intake/query.py` (trigger grammar + deterministic answer rendering) + + `crm_client.nl_query`, and ships on the Spark (no s9pk for the bot). Two entry points: a + **dedicated Q&A room** (`MATRIX_QUERY_ROOM`, every message is a question) and the `?`/`@bot` + trigger in the intake room. **It depends on this endpoint being live on the box** — which lands + with the v93 s9pk (reminders + W2); deploy the bot only after that, or it 404s. See the + matrix-intake guide. The **web "Ask" box** (Communications tab) is the remaining client. - Status: local-model outage → **503**; unexpected SQL fault → **500**; everything else (a hit, or a soft `no_match`/`unknown_intent`) → **200** with the structured result, because the UI always wants the interpreted query back, not a bare code.