Read-only natural-language query over the curated nl_query endpoint, answered
in-thread. Two entry points (room-per-purpose model): a dedicated Q&A room
(MATRIX_QUERY_ROOM) where every top-level message is a question, plus the
?/@bot trigger in the intake room as a cross-room convenience. Both routes hit
the same handle_query -> crm_client.nl_query -> POST /api/query/nl; translation
runs on the box's local model, nothing leaves the box, and there is no write
path so no approval gate applies.
Pure logic (trigger parsing, answer rendering) in query.py with offline tests;
async room wiring in bot.py (live-smoke only, per the bot's convention).
Bot-side only, ships on the Spark via git pull + restart. Depends on the
box-side /api/query/nl endpoint, which lands with the v93 s9pk (reminders + W2):
until v93 is installed the Q&A surface 404s, so the bot deploy is staged to
follow that install.
Read-only "ask the database in plain English" backend. Translation runs on
the local Qwen via Spark Control (question -> {intent, slots}); nothing leaves
the box, no Claude and no redaction boundary (the simplification chosen after
pressure-testing). The safe surface is a curated catalog of ~12 hand-written
parameterized queries; a slot validator is the trust boundary (no generic SQL,
no dynamic identifiers). POST /api/query/nl + GET /api/query/catalog, gated
require_bot_or_admin, read-only, audited. Soft-delete-correct per table.
Local Qwen translated 12/12 real example questions correctly against the live
Spark. Web "Ask" box and Matrix bot still to come (steps 4-5).
Durable updates after the email-proposal review session:
- AGENTS.md: roles admin/member -> admin/member/bot; add a Conventions entry on
the bot role and the reach(role)-vs-autonomy(approval gate) principle.
- matrix-intake guide: rewrite the bridge section to final behavior (redact_thread
whole-thread redaction, the Element 'show deleted' client-setting dependency for
full clearing, the redact_resolved.py backfill tool, deploy gotchas).
- Current state rewritten lean (14->8 bullets); test count 27->30.
The bot was granted a redact/mod power level in the review room, so it can now
clear a resolved thread entirely, not just the card: redact_thread redacts the
card root then scans recent history for its m.thread replies (the human's yes/no
+ any bot messages) and redacts those too, so decided threads drop out of the
threads view, not only the main timeline. Drops the in-thread confirmation on a
successful decision (the thread clearing is the ack; a confirmation would keep
the thread alive). redact_resolved.py extended to also clear replies of already-
resolved threads for the one-time backfill. Bot-only; no s9pk change.
Cards decided before the auto-redact behavior shipped are already 'closed' in
the CRM, so the bot's to_close sweep never redacts them. redact_resolved.py walks
the review room, keeps cards still pending (CRM 'open' list), and redacts the
rest. Dry-run by default; --apply to act. Run via docker compose on the Spark.
Three post-smoke refinements to the Matrix email-proposal review:
1. Dash separators (bot): every card/reply is framed with a dash rule top and
bottom so threads stop bleeding together vertically on mobile.
2. Remove decided threads (bot): on a conclusive approve/dismiss from either
surface, the bot redacts the card (client.room_redact) so the room clears
down to only undecided items. Redacting the bot's own card needs no power;
the web->Matrix path now redacts instead of posting a closure note.
3. Clearer note wording (server v91 + bot): the proposed grid note now names who
emailed whom -- "{teammate} emailed {investor}" (outbound) / "{sender} emailed
the team" (inbound) -- instead of an ambiguous "Sent"/"Received". Outbound
detection also matches our corporate domain (public providers excluded), so a
teammate's mail from a non-enrolled @ten31.xyz address no longer reads as
"Received". Going-forward only; no schema change. The card drops its bare
direction label since the note now carries the relationship.
Tests updated; 30/30 green, render-smoke green.
room_send to a room the bot is only invited to (not joined) fails M_FORBIDDEN;
join explicitly on startup (idempotent if already a member). Bot-only change —
ships via the Spark git pull, no s9pk bump.
v89 added the 'bot' role for the Matrix email-review bot's endpoints but kept it
out of the UI, leaving no click-path to assign it. Add 'bot' to the Settings ->
Admin edit-user role dropdown (the teammate-invite form stays member/admin only —
provisioning an agent account is an admin re-classification of a dedicated user,
not a teammate invite). The backend update validator already accepts 'bot'.
Frontend-only, no schema change.
The email-capture "proposed grid notes" gain two review surfaces:
1. Inline source email — each proposed-note card on the Email Capture page
gets a "View email" toggle that lazily fetches the existing
GET /api/email/detail and shows from/to/cc/date/subject + scrollable body,
so a reviewer can judge the note against the email it was drafted from.
2. CRM->Matrix review bridge — the CRM (box, stdlib, no matrix-nio) can't post
to Matrix, so the intake bot (Spark) PULLS: GET /api/intake/email-proposals
returns to_post/open/to_close work-lists; the bot posts a review card
(metadata + snippet + draft note) to a dedicated review room
(MATRIX_EMAIL_REVIEW_ROOM) and relays in-thread yes / no / NL-edit
(POST .../{id}/decide, note revised via local Qwen). Decisions sync both
ways: web decide -> bot announces + closes the thread; Matrix decide -> the
web panel's ~25s poll clears the card. State lives CRM-side in the new
email_proposal_matrix side row (email-integration migration 0003, additive
+ idempotent CREATE TABLE IF NOT EXISTS), so it survives a bot restart.
Adds a 'bot' role (authenticated, never admin; require_bot_or_admin) to gate
the email-proposal endpoints rather than handing the bot full admin — the
principled base for the coming agentic capabilities. Role controls reach;
the draft->approve gate still controls autonomy (a human approves every write).
Deploy split: endpoints + migration + role + frontend ship in the s9pk; the
bot poll loop + review-room handling ship on the Spark. The bot's CRM user
must be flipped member->bot and joined to the review room (one-time).
Tests: backend/test_email_proposal_matrix.py + matrix_intake/test_email_proposals.py
(30/30 suite green, render-smoke green, migration verified twice on a DB copy).
Box and repo on v0.1.0:88; the +Pipeline -> board -> advance-stage -> remove
round-trip is verified on the box. Pipeline adoption is closed out; ROADMAP
item marked done and the Next list advances to the spark-control intake card.
Opportunities are now born only from a fundraising-grid investor row
("+ Pipeline"), which matches how the team works — they live in the grid,
not on the board. The old "+ New Opportunity" button created a deal by
picking a contact, a path that contradicts the grid-is-canonical model and
the contact-vs-investor framing.
Remove the button, its create-by-contact modal, the now-dead handler/state,
and the Pipeline page's unused /api/contacts fetch. Replace the button with a
muted "Add deals from the Fundraising Grid" hint. The board is now a view +
stage-management surface. Frontend-only; no backend or schema change.
Render-smoke green.
Box and repo now on v0.1.0:87: StartOS chain ...86->87 clean,
0005_grid_pipeline_link.sql applied on the box, server up on :8080.
Next is an in-room/board live-smoke of the Add-to-Pipeline round-trip.
The fundraising grid (canonical) now drives the classic opportunities
Pipeline board, instead of the board being a disconnected second data-entry
surface. An "Add to Pipeline" row action creates a durably-linked opportunity
via the new opportunities.fundraising_investor_id (migration 0005, additive +
reversible), reusing the grid's already-synced contact — retiring the
POST /api/contacts side-door — and mapping the grid lead to the opp owner.
Ownership is split so the two stay reconciled: the grid owns whether the link
exists and the seed; the board owns stage/probability/owner. The link endpoint
is idempotent (one live opp per investor; a re-link never reseeds funnel
fields). "Is in pipeline?"/"what stage?" are derived from a live opp join and
injected as read-only grid columns on read, stripped on write, so they never
persist or dirty the autosave. Remove-from-pipeline soft-deletes the opp and
leaves the grid row fully intact; deleting an investor from the grid archives
its orphaned opp.
Also fixes the standing soft-delete leak in handle_pipeline_report and the
dashboard pipeline aggregates, which counted tombstoned opportunities.
Tests: backend/test_grid_pipeline_link.py (link/idempotent/round-trip/guards/
unlink-intact/re-link/orphan/aggregates); 28/28 suite green, render-smoke green.
Local-smoke found "jonathan is chatting with wyoming" extracted the teammate, not
the prospect. Feed the parser an optional team roster (INTAKE_TEAM_ROSTER) via a
build_system(roster) outreach frame: roster names/initials are the people doing
outreach and are never extracted; the other party is the investor/prospect. Same
framing on the revise leg. Unset roster = prior behavior.
Bot now runs as a docker-compose service on the Spark (verified live, listening). Docs (matrix-intake guide ops, ROADMAP, AGENTS Current state) updated. Also logs the live-smoke parse bug (teammate read as investor -> team-roster fix), the spark-control dashboard-card handoff, and the long-term dedicated-repo extraction.
Turn the bot from a bare nohup process (silently dies on a Spark reboot) into a docker-compose service. Dockerfile bundles backend/matrix_intake + the stdlib backend/ingest Spark client it reuses; .env is mounted read-only at runtime, never baked. The existing repo-root .dockerignore (shared with the s9pk build) already keeps data/ and .env out of context. Also adds a handoff doc for wiring a spark-control dashboard card in a later session.
Box installed to 0.1.0:86 (migration chain ...85->86 clean, candidates endpoint verified live); bot pulled + restarted on the Spark. Only the Matrix live-smoke remains.
Close the two locked post-deploy enhancements for the Matrix intake bot.
Fuzzy matching (server-side, ships in the s9pk): new find_intake_candidates in
server.py returns ranked deterministic near-matches (difflib name similarity +
token-set Jaccard, legal-suffix-aware, + email Levenshtein <= 2); GET
/api/intake/match now returns {match, candidates}. The bot surfaces a numbered
shortlist so a near-duplicate (Charlie/Charles, Acme Capital vs Acme Capital LLC,
a one-char email typo) is confirmed by a human instead of silently creating a
second investor. Exact match still auto-attaches; fuzzy candidates are never
auto-attached. The optional LLM-judge re-rank is deferred.
Conversational edits (bot-side, ships on the Spark): any in-thread reply that
isn't yes/no/edit field=value is treated as a natural-language revision and
re-run through local Qwen (parse.revise). Email integrity is preserved -- a
changed address must literally appear in the instruction; the model's email
field is structurally unreachable. No-op revisions re-prompt.
Docs/current-state brought current; 27/27 backend tests green.
The grid note line was "YYYY-MM-DD [type] Contact: summary"; for the default
"note" type the tag is noise. Omit it for "note"; keep it for informative
types (call, meeting, …). Shared by the Matrix intake bot and grid-UI logging.
Built + installed to the box (installed-version 0.1.0:85, clean 84->85
migration). No schema change.
Four bot-side UX fixes surfaced by the live smoke:
- Post a brief pointer in the main timeline (a reply to the user's message)
alongside the in-thread proposal card, so proposals aren't missed inside a
thread. Pointer only — approvals still happen in the thread, where the note
is visible (you can't make an informed yes/no without seeing it).
- A bare yes/no typed in the main timeline while a proposal is pending now
gets a "reply in the thread" redirect instead of "couldn't tell what to record."
- Clearer commit confirmations: "Created a new grid entry for X" vs
"Logged a note on X (existing grid entry)."
- Send a blank communication subject when a note is present so the grid's
one-line note summary shows the note text, not the "(Matrix)" label
(provenance stays in source="matrix_intake").
The intake bot's server-side dependencies — GET /api/intake/match (new-vs-
existing lookup) and `source` provenance on log-communication — shipped in
source with 7ad0ee7 but were never packaged. The box ran v83 (pre-7ad0ee7),
so the bot's match calls 404'd: a note on an existing investor would have
created a duplicate, and writes weren't tagged matrix_intake. Bump + build +
install verified live (installed-version 0.1.0:84, clean 83->84 migration,
match endpoint now resolves by name and email). No schema change.
Also log the conversational (LLM-mediated) edit enhancement in ROADMAP.
normalize()'s email regex matched non-@/non-space runs, so "Name <addr>"
(the most common contact format) yielded "<addr"; only trailing punctuation
was stripped, never leading. Tighten the regex to standard local@domain.tld
so the bare address is extracted from <…>, (…), and trailing-period forms.
Found via the live-deploy pre-flight. Add a regression test.
Also log two intake backlog items in ROADMAP: the scoped service-credential
auth path (deferred; bot uses a member login for now) and fuzzy match +
in-thread confirm (post-deploy).
New backend/matrix_intake/ runs as its own process (matrix-nio isolated from the
stdlib CRM): local-Qwen parse via Spark Control → in-thread human approval
(yes/edit/no) → write through the CRM's own log-communication endpoint, tagged
source=matrix_intake. Adds read-only GET /api/intake/match (returns grid row id,
no-duplicate contract); threads provenance through handle_log_fundraising_communication.
Reviewer-passed: pop-before-commit closes a double-approve race; edit-grammar fix.
Text-only v1; business-card photo (M3) deferred (no Spark vision model).
26/26 tests green; live Matrix smoke pending deploy.
Fix AGENTS.md access wording (ClearNet/StartTunnel + app-level auth, not
LAN/Tailscale). Add the StartOS icon bug to Known debt and the email-sync-status
error as Next #8. Add ROADMAP sections for Matrix-bridge grid intake (next,
high priority) and an admin-vs-all-users UI audit.
Communications tab (search/query roadmap items 1 & 2):
- Fix the investor dropdown: the facet only listed grid investors, so it
came back empty whenever email matched a classic contact or org domain
(no grid id — the common case). It now mirrors the email list, resolving
each link to a typed identity (fund:/org:/contact:/addr:) with precedence
grid -> org -> contact -> address; investor_id accepts the typed key
(bare id = fund: for back-compat) and an unknown prefix matches nothing.
- Add a date-range filter and a click-to-expand full-body view
(GET /api/email/detail, admin, soft-delete-gated; body_text only, never
raw remote HTML).
- Add a "Search content" mode: GET /api/email/search wraps the ingest
hybrid_search over the Qdrant email index (doc_type=email), hydrated and
soft-delete-filtered against SQLite (canonical), 503 if Spark/Qdrant down.
Daily digest:
- Settings -> Admin builds a digest over a chosen window (last 24h or since
a date) as an in-app preview before sending (POST /api/admin/digest/preview),
so the local-Spark summarizer can be verified on demand even on a quiet day.
Manual send uses the same window; neither advances the daily cursor, so a
preview never suppresses the scheduled digest.
Code-only, migrations no-op. 22/22 backend tests, render-smoke pass.
Record the v82 vendor+SRI + render-smoke work in durable docs: packaging guide
gains the verified-build gate + re-vendor instructions; Current state rewritten
and compressed for v82; ROADMAP logs the deferred pre-compile-JSX alternative.
React/ReactDOM/Babel were loaded from the unpkg CDN at runtime — react@18
and react-dom@18 weren't even exact-pinned, and none had SRI. A CDN swap (or
react auto-resolving a new 18.x) could blank the whole app with no change on
our side: exactly the v78/v79 blank-screen class. It also made the self-hosted
box depend on outbound internet to render.
Vendor the three libs into frontend/assets/vendor/ (React 18.3.1, ReactDOM
18.3.1, @babel/standalone 7.29.7) and load them same-origin with sha384
integrity attributes. They now ship inside the s9pk (Dockerfile already COPYs
frontend/; server.py serves /assets/* with the path-containment check), so a
CDN can never swap prod deps again and no outbound fetch is needed at runtime.
Add start9/0.4/render-smoke.mjs: a jsdom render smoke check that (1) runs the
shipped Babel over the app's inline JSX and asserts a classic, non-module,
parseable script (the v79 ESM-import regression), and (2) mounts the app in
jsdom and asserts the login UI renders (the v78 blank-screen class). Wired into
the default `make` goal so every package build is gated on the frontend
actually rendering — closing the "verified live via curl only" gap. jsdom is a
build-time devDependency, not shipped in the image.
The email-activity panel surfaced every captured message, including cold/
unknown-sender email with no investor association. Gate query_email_activity
on EXISTS(email_investor_links) so the panel shows only email tied to a known
investor/contact. Capture is unchanged — unmatched email is still stored
(metadata-only) and will appear automatically if its sender is later added as
an investor; this is a read-side filter only.
Graveyard investors are unaffected (their email has a link), so they remain
visible/searchable as an audit surface, hidden only from the filter picker.
The Communications tab is now an admin-only search over captured Gmail
(email_* tables), part of consolidating on the fundraising grid + email
capture as the canonical system of record.
- New GET /api/email/activity (admin-enforced server-side): filter by
investor / mailbox / direction with free-text search over subject,
snippet, and sender. Query logic in db.query_email_activity.
- Soft-delete honored on the per-mailbox sighting (emails carry no
deleted_at; deletion lives on email_account_messages).
- Direction decided at the email level (outbound if the sender is one of
our mailboxes), mirroring digest_builder.
- Graveyard investors are hidden from the filter dropdown (CRM-wide
graveyard=0 convention) but their email stays visible in the list and
findable by free-text search — this is an audit surface.
- Communications page rewritten to render the panel; the classic manual
"Log Communication" form is retired (the grid context menu remains the
manual-log path). Nav item + page are admin-only.
- Tests: email_integration/test_email_activity_panel.py (filters,
per-sighting soft-delete, roll-ups, graveyard handling, route 401/403);
full suite 22/22. Frontend render verified via a jsdom mount smoke test
plus the pinned classic-runtime Babel transform.
Code-only, no schema migration (version migrations are no-ops).
Record the Babel-pin fix + root cause, the 3 newly admin-gated GET endpoints, the corrected deploy-verification convention (browser render, not curl/health), and the re-ordered Next list (vendor+SRI, auth regression test, email-activity panel in the admin-only Communications tab).
The web UI rendered a blank screen for every user. Root cause: the page
loaded @babel/standalone from unpkg with no version pin, so the CDN silently
served Babel 8.0.0. Babel 8 defaults @babel/preset-react to the automatic JSX
runtime, which prepends `import {jsx} from "react/jsx-runtime"` to the compiled
output. An ESM import is illegal in this classic (non-module) inline <script>,
so the browser rejected the whole bundle and React never mounted — hence the
blank screen. The prior "verified live" checks were server-up/curl, which can't
catch a browser-render failure.
- Pin @babel/standalone@7.29.7 (its preset-react defaults to the classic
React.createElement runtime). Verified via headless render: app mounts, login
screen renders, no console error. Follow-up: vendor + SRI-pin the CDN libs so
a third party can't swap our front-end deps in production again.
- Close three server-side admin gaps surfaced by a permissions audit — endpoints
that were UI-hidden from members but not API-enforced: GET /api/users,
/api/email/status, /api/email/accounts now require_admin. Removed the now-dead
non-admin mailbox-row filter. 21/21 backend tests green; py_compile clean.
The fundraising grid + email capture is the canonical system of record. lp_profiles
was a superseded single-fund model with no reachable create/edit path, and the LP
Tracker page was already orphaned (no nav entry + a redirect bouncing it to the grid).
- Remove /api/lp-profiles* endpoints + handlers, the unused lp-breakdown report,
the contact-dossier LP section, the demo-seed LP block, and (frontend) the
LPTrackerPage component + its lp-tracker->fundraising-grid redirect.
- Dashboard "Total Committed" now sums fundraising_investors.total_invested
(graveyarded investors excluded) instead of the orphaned lp_profiles table, which
read ~$0. "Total Funded" dropped: the grid tracks commitments, not a funded amount,
and the frontend never rendered it.
- Leave the empty lp_profiles table/index, the contact-delete soft-delete cascade,
and the --reset-all-data clear in place (never-hard-delete).
- Tests: add test_dashboard_report.py; update test_soft_delete_reads.py. 21/21 green.
Docs-only: packaging guide notes start-cli install is silent on success (verify
with installed-version/logs); AGENTS.md adds the operational-toggles-in-the-admin-
panel convention and tightens the digest Current state.
installed-version on the box -> 0.1.0:77; migration chain ran through 76->77;
server up on :8080 with the digest scheduler running (policy-controlled, auto-send
off by default). Docs-only.
Sends a once-a-day internal email to all active admins summarizing each team
member's email activity per investor, plus a team-wide by-investor view
(inbound + outbound, deduped). Narratives are generated on the LOCAL Spark
model, never Claude — the digest is intentionally un-anonymized, so substance
stays on Ten31 infra. This is an internal ops email, exempt from the
'agents draft, humans send' rule (which governs outward LP contact).
- backend/digest_builder.py: per-user + per-investor activity queries
(soft-delete filtered), per-user Spark narrative with a deterministic
fallback, two-section plain-text body, and the DB-backed policy resolver.
- backend/email_integration/digest_scheduler.py: always-on daily thread that
re-reads the policy each cycle and sends once/day; window cursor in
app_settings so a missed day rolls forward.
- server.py: POST /api/admin/digest/send-now and GET/PATCH
/api/admin/digest/policy; scheduler wired into main().
- Control lives in Settings -> Admin (enable toggle + send-time dropdown),
not StartOS actions; env vars only seed the first-boot default.
- Tests: backend/test_digest_builder.py.
Extend docs/guides/email.md paths: frontmatter (and its AGENTS.md index entry) to
include backend/digest_mailer.py and backend/smtp_send.py, so the guide auto-loads
when editing the outbound-digest send path — not just backend/email_integration/**.
Portability-checker: compliant.
- docs/guides/email.md: new "Outbound mail — the daily digest" section (Gmail-DWD
primary → SMTP fallback; gmail.compose send capability; the internal-digest
exemption from the agents-draft rule).
- AGENTS.md: add digest env names (CRM_DIGEST_SENDER, SMTP_*); consolidate the
v75/v76 deploy bullets into one current bullet; drop finished v74 narrative.
Non-blocking items from the v76 reviewer pass. No redeploy needed — the box runs
v76 and its happy path is unaffected; these ride the next build:
- digest_mailer.send_digest: when Gmail is enabled but no sender resolves
(CRM_DIGEST_SENDER unset and no admin email), raise NoTransport so the caller
returns a clear 400 instead of a generic 502.
- gmail_send.send_via_gmail: wrap OSError/URLError (timeout/DNS) as a RuntimeError
("Gmail API unreachable: ...") to match the HTTPError handling; include the
sender in the HTTPError message for debuggability.
- credentials.py: correct the now-stale GMAIL_COMPOSE_SCOPE comment (the digest
mailer sends with this scope; only outreach drafts never send).
- test_gmail_send.py: add the HTTPError->RuntimeError branch, default_sender DB
fallback (+None case + env override), and the send_digest SMTP-tag path.
19/19 backend tests green.
Keysat