diff --git a/.env.example b/.env.example index 84ba2fe..0c19d73 100644 --- a/.env.example +++ b/.env.example @@ -19,3 +19,14 @@ X_API_KEY= # ── CRM (ingest opens the SQLite file directly, read-only) ── CRM_DB_PATH=./data/crm.db CRM_DEV_DB_PATH=./data/crm_dev.db + +# ── Daily-digest outbound SMTP (dev override of the per-package mailbox) ── +# On the Start9 box these are set by the "Configure Digest SMTP" action (written +# to /data/secrets/smtp/* and exported by docker_entrypoint.sh). For dev, set them +# here. SMTP_SECURITY is one of: starttls (587) | tls (465) | none. +SMTP_HOST= +SMTP_PORT=587 +SMTP_SECURITY=starttls +SMTP_FROM= +SMTP_USERNAME= +SMTP_PASSWORD= diff --git a/AGENTS.md b/AGENTS.md index 4a7753e..fd39a5f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -100,14 +100,14 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude ## Current state -_Phase 0 substrate + Phase 1 thesis/outreach are built; current package is **v0.1.0:74**. Longer-term backlog: `ROADMAP.md`._ +_Phase 0 substrate + Phase 1 thesis/outreach are built; **deployed box is v0.1.0:74**, **repo is at v0.1.0:75** (committed, not yet built/deployed). Longer-term backlog: `ROADMAP.md`._ - **Working (all draft-only):** CRM + ingest (chunk→embed→Qdrant + retrieval) + redaction boundary; Gmail capture (DWD) + email-activity propose→approve; Thesis Workshop + Architect (Claude) with dual-approval gate; Outreach Draft Assistant + follow-up radar + per-user voice + Tier-B in-thread Gmail draft creation. - **Deployed & verified live (2026-06-13):** v0.1.0:74 is **installed and healthy on the box** (`$START9_BOX_HOST` / immense-voyage.local). Grant confirms login works; `/assets/` traversal 404s live (plain + URL-encoded), root health 200. On boot, `ensure_thesis_v2_promoted` makes the v2.0 reserve-asset spine the working *approved* spine (node-level, reversible). -- **Repo ahead of the box (committed, NOT yet built/deployed):** since v74, `main` adds the **list-view soft-delete aggregate fix** (`server.py`: org `contact_count`/`total_funded`, contacts `comm_count`/`last_contact_date` now filter `deleted_at`), three **regression tests** (traversal/soft-delete/NER), and an **aggregate test runner**. The deployed box is still pristine v74 — **bump the version before the next s9pk build** to ship these. +- **Repo ahead of the box (committed, NOT yet built/deployed):** the box is pristine v74; `main` is at **v0.1.0:75** and carries two unshipped batches. (a) Post-v74: the **list-view soft-delete aggregate fix** (`server.py`: org `contact_count`/`total_funded`, contacts `comm_count`/`last_contact_date` now filter `deleted_at`), three **regression tests**, and an **aggregate test runner**. (b) **v0.1.0:75 — daily-digest Phase A** (outbound SMTP send): the **`configureDigestSmtp`** Start9 action writes a per-package SMTP account to `/data/secrets/smtp/*` (password over stdin; independent of any StartOS system-wide SMTP), `docker_entrypoint.sh` exports `SMTP_*`, `backend/smtp_send.py` (stdlib smtplib) sends, and admin **`POST /api/admin/digest/test-email`** proves the pipe (recipients restricted to the active-admin set — not an open relay). One `make` ships both batches. - **Shipped in v0.1.0:74** (security/privacy hardening from the 2026-06-12 full-eval; report in `EVALUATION.md`): closed a pre-auth `/assets/` path traversal (could read crm.db / JWT secret / Gmail key); wired the local-Qwen NER backstop into the outreach redaction boundary (free-prose email bodies were reaching Claude with unknown names in the clear); added `deleted_at IS NULL` to every get-by-id + nested sub-select read path. Verified locally (py_compile, query exec, redaction/outreach tests, containment logic) + two reviewer passes. -- **Tests (2026-06-13):** **16/16 backend tests green** via `python3 backend/run_tests.py` (the new aggregate runner; +3 regression tests this session). `py_compile` clean; `./start.sh`/`./start_beta.sh` boot (health 200, auth 401); `make` builds the x86 s9pk. The 2 stale thesis tests stay fixed (seed structure in `docs/guides/thesis.md`). +- **Tests (2026-06-15):** **18/18 backend tests green** via `python3 backend/run_tests.py` (+`test_smtp_send.py`/`test_smtp_endpoint.py` this session). `py_compile` clean; the s9pk TypeScript typechecks (`cd start9/0.4 && npm run check`, deps installed); `docker_entrypoint.sh` passes `sh -n`. The 2 stale thesis tests stay fixed (seed structure in `docs/guides/thesis.md`). - **Decided, not yet built:** CRM as canonical thesis backbone with the signal-engine reading from it (reconciliation unwired); reply-all for Tier-B drafts (drafts currently reply to the LP only). - **Known debt (P2, not deploy-blocking):** the **reports subsystem** (`handle_dashboard_report`/`handle_pipeline_report`/`handle_lp_breakdown_report`, ~16 aggregate queries over contacts/opportunities/communications/lp_profiles) still counts soft-deleted rows — the list/detail aggregates were fixed (v74 + the org/contacts list-view follow-up) but the reports were not; needs its own pass + report-endpoint tests; `?limit=abc` crashes the request thread (authenticated list path); scrub-gateway TLS verify off; `cryptography==42.0.5`; unpkg/no-SRI frontend; stale user-visible `start9/0.4/assets/ABOUT.md`; hardcoded Spark/Qdrant IPs in the s9pk; the 5.4k-line `server.py` monolith. P3 batch + full list in `EVALUATION.md`. - **Other gaps:** the v2.0 spine is the *working* spine but **not a canonical `thesis_version`** (needs Grant + Jonathan dual sign-off); Appendix-A conviction/exposure (incl. ~40% Strike) stay Grant's working read, not canonical, not fed to the engine; live features (Claude/Qdrant/Gmail) unverified on the box. -- **Next:** 1) **reports-subsystem soft-delete sweep** — ~16 dashboard/pipeline/LP aggregate queries still count soft-deleted rows; fix + add report-endpoint tests; 2) **bump version + rebuild/redeploy** to ship the list-view fix + tests now sitting ahead of the box; 3) `?limit=abc` crash (P2); 4) Grant + Jonathan freeze v2.0 canonical; 5) build reply-all; 6) confirm Appendix-A + Maple/OpenSecret/Primal, then promote. +- **Next:** 1) **build/deploy v0.1.0:75** (one `make` ships the list-view fix + digest Phase-A SMTP); then on the box: run **Configure Digest SMTP**, restart, and hit **Send Test Digest Email** to verify the pipe; 2) **digest Phase B** — daily scheduler + per-user→per-investor activity query (`deleted_at IS NULL`) + **Spark-narrative** summary (never Claude) → email all admins (decisions locked in `ROADMAP.md`); 3) **reports-subsystem soft-delete sweep** (~16 aggregates still leak; fix + tests); 4) `?limit=abc` crash (P2); 5) Grant + Jonathan freeze v2.0 canonical; 6) build reply-all; 7) confirm Appendix-A + Maple/OpenSecret/Primal, then promote. diff --git a/ROADMAP.md b/ROADMAP.md index 226ded4..69ab9da 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -87,7 +87,13 @@ ## Backlog (post-Phase-1 agentic) ### Daily activity digest (email to the team) -*Requested 2026-06-15. Not yet built.* +*Requested 2026-06-15. **Phase A built in v0.1.0:75** (outbound SMTP send capability + admin test-email endpoint; not yet deployed). Phase B (digest content + Spark summarization + daily scheduler) remains.* + +**Decisions (locked 2026-06-15):** recipients = **all active admins**; summarization = **Spark-LLM narrative** (never Claude — un-anonymized substance stays local); granularity = **grouped by user** (→ per investor). + +**Phase A — DONE (v0.1.0:75):** `configureDigestSmtp` Start9 action writes a per-package SMTP account to `/data/secrets/smtp/*`; `docker_entrypoint.sh` exports `SMTP_*`; `backend/smtp_send.py` (stdlib smtplib) + admin `POST /api/admin/digest/test-email` (recipient-restricted to the admin set — not an open relay). Tests: `test_smtp_send.py`, `test_smtp_endpoint.py`. + +**Phase B — TODO:** daily scheduler (co-locate with `email_integration/scheduler.py`); per-user→per-investor activity query (`deleted_at IS NULL` throughout); Spark-narrative summary of captured email substance; compose + send to all admins. Have the CRM send a **daily digest email** summarizing each registered user's activity — primarily **who emailed which investors and the substance of those emails** — to the fund principal (and eventually other admins). Scales with the synced-user count: 2 users synced today, ~5 eventually. @@ -98,11 +104,7 @@ Have the CRM send a **daily digest email** summarizing each registered user's ac - **Scheduling:** a daily cron, naturally co-located with the existing `backend/email_integration/scheduler.py` sync cadence. - **Soft-delete:** every aggregate/read in the digest must filter `deleted_at IS NULL` (see the standing soft-delete rule). -Open design questions to resolve before building: -1. Recipients — just the principal, or all admins / each user gets their own? -2. Summarization — rule-based rollup vs. Spark-LLM narrative summary of email substance? -3. Digest granularity — per-user → per-investor threads, or a fund-wide activity roll-up? -4. Cadence/controls — fixed daily time, opt-out, "nothing happened today" suppression. +Open design questions (Phase B detail, still to settle): fixed daily send time; "nothing happened today" suppression; whether the Spark summary is per-investor-thread or a single per-user narrative. ## Definition of done for "Airtable substitute" v1 - Team can manage all investors in one master table diff --git a/backend/server.py b/backend/server.py index b0011c9..40aa3ce 100644 --- a/backend/server.py +++ b/backend/server.py @@ -1919,6 +1919,8 @@ class CRMHandler(BaseHTTPRequestHandler): return self.handle_admin_create_user(user, body) if path == '/api/admin/reset-all-data': return self.handle_admin_reset_all_data(user, body) + if path == '/api/admin/digest/test-email': + return self.handle_admin_send_test_email(user, body) if path == '/api/fundraising/backup': return self.handle_backup_fundraising_state(user) if path == '/api/fundraising/restore-preview': @@ -4234,6 +4236,67 @@ class CRMHandler(BaseHTTPRequestHandler): } }) + def handle_admin_send_test_email(self, user, body): + """Send a test email through the configured digest SMTP account, to prove + the outbound pipe before the daily digest itself is built (Phase A).""" + if not require_admin(user): + return self.send_error_json("Admin only", 403) + + from smtp_send import send_email, smtp_configured, SMTPNotConfigured + if not smtp_configured(): + return self.send_error_json( + "SMTP is not configured. Set it via the 'Configure Digest SMTP' " + "Start9 action (or SMTP_* in .env for dev), then restart.", 400) + + # Recipients are restricted to the active-admin set (the real digest + # audience). An explicit `to` may NARROW to specific admins but can never + # introduce an outside address — this endpoint is not an open mail relay. + conn = get_db() + try: + rows = conn.execute( + "SELECT email FROM users WHERE role = 'admin' AND is_active = 1 " + "AND email IS NOT NULL AND TRIM(email) != ''" + ).fetchall() + finally: + conn.close() + admin_emails = [str(r['email']).strip() for r in rows if str(r['email']).strip()] + admin_lower = {e.lower() for e in admin_emails} + + to = body.get('to') + if to: + requested = [to] if isinstance(to, str) else list(to) + requested = [str(e).strip() for e in requested if str(e).strip()] + outside = [e for e in requested if e.lower() not in admin_lower] + if outside: + return self.send_error_json( + "Test email may only go to an active admin address; " + f"not allowed: {', '.join(outside)}", 400) + recipients = requested + else: + recipients = admin_emails + if not recipients: + return self.send_error_json( + "No recipient: give an active admin an email address first.", 400) + + subject = "Ten31 CRM — test digest email" + email_body = ( + "This is a test message from the Ten31 CRM daily-digest mailer.\n\n" + f"Triggered by {user.get('full_name') or user.get('user_id')} at {now()}.\n\n" + "If you received this, outbound SMTP works and the daily digest can be " + "delivered to this address." + ) + try: + result = send_email(recipients, subject, email_body) + except SMTPNotConfigured as exc: + return self.send_error_json(f"SMTP not configured: {exc}", 400) + except Exception as exc: + # Never echo the exception to the client — an SMTP auth error can carry + # the server's reply (and potentially the credential). Log it instead. + print(f"[smtp] test send failed: {type(exc).__name__}: {exc}", file=sys.stderr) + return self.send_error_json("Send failed — see server logs for details.", 502) + + return self.send_json({"data": {"status": "sent", **result}}) + def handle_list_audit_log(self, user, params): if not require_admin(user): return self.send_error_json("Admin access required", 403) diff --git a/backend/smtp_send.py b/backend/smtp_send.py new file mode 100644 index 0000000..b09bf01 --- /dev/null +++ b/backend/smtp_send.py @@ -0,0 +1,99 @@ +"""Minimal outbound SMTP sender for the CRM (daily digest, test sends). + +Config comes ONLY from SMTP_* environment variables. Two ways those get set: + * dev / bare run: a local .env (see .env.example). + * Start9 box: docker_entrypoint.sh reads the files the "Configure Digest SMTP" + StartOS action writes under /data/secrets/smtp/ and exports them as env. +The backend never reads those files directly, so dev and prod share one path. + +This is the package's OWN dedicated mailbox (per-package custom SMTP) — it is +independent of any StartOS system-wide SMTP account; nothing here calls into the +platform. Stdlib only (smtplib/ssl/email), consistent with the rest of runtime. +""" +import os +import smtplib +import ssl +from email.message import EmailMessage + + +class SMTPNotConfigured(Exception): + """Raised when SMTP_* env is absent — callers turn this into a clear 'not + configured' response rather than a 500.""" + + +def smtp_configured(): + return bool(os.environ.get("SMTP_HOST", "").strip()) + + +def load_smtp_config(): + host = os.environ.get("SMTP_HOST", "").strip() + if not host: + raise SMTPNotConfigured("SMTP_HOST is not set") + # Port/security come from a free-text action field; normalize defensively. + try: + port = int(str(os.environ.get("SMTP_PORT", "") or "587").strip()) + except ValueError: + port = 587 + security = (os.environ.get("SMTP_SECURITY", "") or "starttls").strip().lower() + if security not in ("starttls", "tls", "none"): + security = "starttls" + return { + "host": host, + "port": port, + "from_addr": os.environ.get("SMTP_FROM", "").strip(), + "username": os.environ.get("SMTP_USERNAME", "").strip(), + "password": os.environ.get("SMTP_PASSWORD", ""), + "security": security, + } + + +def _connect(cfg, timeout): + """Open an authenticated SMTP connection per the configured security mode. + 'tls' = implicit TLS (SMTPS, usually 465); 'starttls' = upgrade on 587; + 'none' = plaintext (for a LAN relay that does its own transport security).""" + if cfg["security"] == "tls": + ctx = ssl.create_default_context() + server = smtplib.SMTP_SSL(cfg["host"], cfg["port"], timeout=timeout, context=ctx) + else: + server = smtplib.SMTP(cfg["host"], cfg["port"], timeout=timeout) + server.ehlo() + if cfg["security"] == "starttls": + server.starttls(context=ssl.create_default_context()) + server.ehlo() + if cfg["username"]: + server.login(cfg["username"], cfg["password"]) + return server + + +def send_email(to_addrs, subject, body, *, html=None, cfg=None, timeout=30): + """Send one message. `to_addrs` is a str or list; `body` is plain text and + `html` an optional HTML alternative. Returns {'sent_to', 'from'} on success; + raises SMTPNotConfigured / ValueError / smtplib.SMTPException otherwise.""" + cfg = cfg or load_smtp_config() + if isinstance(to_addrs, str): + to_addrs = [to_addrs] + to_addrs = [a for a in (str(x).strip() for x in to_addrs) if a] + if not to_addrs: + raise ValueError("no recipients") + + from_addr = cfg["from_addr"] or cfg["username"] + if not from_addr: + raise SMTPNotConfigured("no SMTP_FROM or SMTP_USERNAME to use as sender") + + msg = EmailMessage() + msg["From"] = from_addr + msg["To"] = ", ".join(to_addrs) + msg["Subject"] = subject + msg.set_content(body) + if html: + msg.add_alternative(html, subtype="html") + + server = _connect(cfg, timeout) + try: + server.send_message(msg, from_addr=from_addr, to_addrs=to_addrs) + finally: + try: + server.quit() + except Exception: + pass + return {"sent_to": to_addrs, "from": from_addr} diff --git a/backend/test_smtp_endpoint.py b/backend/test_smtp_endpoint.py new file mode 100644 index 0000000..32bdaee --- /dev/null +++ b/backend/test_smtp_endpoint.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 +"""Endpoint tests for POST /api/admin/digest/test-email (handle_admin_send_test_email). + +Boots the REAL server in-process against a throwaway DB, monkeypatches the SMTP +sender (no network), and proves the security-relevant contract: + * admin-gated: 401 without a token, 403 for a non-admin. + * recipients are restricted to the active-admin set — an arbitrary `to` is + rejected (the endpoint is NOT an open relay), an admin `to` is allowed, + and the default audience is every active admin. + * a send failure does not echo the exception (which could carry the SMTP + credential) back to the caller. + +Run: cd backend && python3 test_smtp_endpoint.py +""" +import http.client +import json +import os +import sys +import tempfile +import threading +from http.server import ThreadingHTTPServer + +_BASE = tempfile.mkdtemp() +_FRONTEND = os.path.join(_BASE, "frontend") +os.makedirs(os.path.join(_FRONTEND, "assets")) +_DATA = os.path.join(_BASE, "data") +os.makedirs(_DATA) +with open(os.path.join(_FRONTEND, "index.html"), "w") as f: + f.write("crm") +os.environ["CRM_FRONTEND_DIR"] = _FRONTEND +os.environ["CRM_DATA_DIR"] = _DATA +os.environ["CRM_DB_PATH"] = os.path.join(_DATA, "crm.db") + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import server # noqa: E402 +import smtp_send # noqa: E402 + +FAILS = [] + +# ── fake sender: record calls, optionally raise (no network) ── +SENT = [] +RAISE = {"exc": None} + + +def fake_send(to_addrs, subject, body, **kw): + if RAISE["exc"] is not None: + raise RAISE["exc"] + SENT.append({"to": list(to_addrs), "subject": subject}) + return {"sent_to": list(to_addrs), "from": "digest@ten31.test"} + + +smtp_send.send_email = fake_send +smtp_send.smtp_configured = lambda: True + + +def check(cond, msg): + print((" PASS " if cond else " FAIL ") + msg) + if not cond: + FAILS.append(msg) + + +class _Quiet(server.CRMHandler): + def log_message(self, *a): + pass + + +def _post(port, path, token=None, body=None): + conn = http.client.HTTPConnection("127.0.0.1", port, timeout=10) + headers = {"Content-Type": "application/json"} + if token: + headers["Authorization"] = "Bearer " + token + conn.request("POST", path, body=json.dumps(body or {}), headers=headers) + resp = conn.getresponse() + raw = resp.read().decode("utf-8", "replace") + conn.close() + try: + parsed = json.loads(raw) + except Exception: + parsed = None + return resp.status, raw, parsed + + +def main(): + server.init_db() + + # Admin = first registered user. A member is inserted directly (self-register + # is disabled after the first user) so get_user resolves a real active row. + httpd = ThreadingHTTPServer(("127.0.0.1", 0), _Quiet) + port = httpd.server_address[1] + threading.Thread(target=httpd.serve_forever, daemon=True).start() + try: + st, raw, j = _post(port, "/api/auth/register", body={ + "username": "admin", "email": "admin@ten31.test", + "password": "password123", "full_name": "Admin User"}) + check(st == 201 and j and j.get("token"), f"register first user as admin (got {st})") + admin_token = j["token"] + admin_email = j["user"]["email"] + + conn = server.get_db() + conn.execute( + "INSERT INTO users (id, username, email, password_hash, full_name, role, is_active) " + "VALUES (?,?,?,?,?,?,1)", + ("member-1", "member1", "member1@ten31.test", + server.hash_password("password123"), "Member One", "member")) + conn.commit() + conn.close() + member_token = server.create_token("member-1", "member1", "member") + + path = "/api/admin/digest/test-email" + + # 1. unauthenticated -> 401, sender untouched + SENT.clear() + st, raw, j = _post(port, path) + check(st == 401, f"no token -> 401 (got {st})") + check(not SENT, "no token: sender not called") + + # 2. non-admin -> 403 + SENT.clear() + st, raw, j = _post(port, path, token=member_token) + check(st == 403, f"member -> 403 (got {st})") + check(not SENT, "member: sender not called") + + # 3. admin, no `to` -> 200, default audience = the admin set + SENT.clear() + st, raw, j = _post(port, path, token=admin_token) + check(st == 200, f"admin default -> 200 (got {st})") + check(len(SENT) == 1 and SENT[0]["to"] == [admin_email], + f"default recipients = active admins ({admin_email}); got {SENT}") + + # 4. admin, arbitrary outside `to` -> 400, NOT an open relay + SENT.clear() + st, raw, j = _post(port, path, token=admin_token, body={"to": "attacker@evil.com"}) + check(st == 400, f"outside `to` -> 400 (got {st})") + check(not SENT, "outside `to`: sender not called (no relay)") + + # 5. admin, `to` an admin address (case-insensitive) -> 200 + SENT.clear() + st, raw, j = _post(port, path, token=admin_token, body={"to": admin_email.upper()}) + check(st == 200, f"admin `to` -> 200 (got {st})") + check(len(SENT) == 1 and SENT[0]["to"] == [admin_email.upper()], + "admin `to` is delivered as given") + + # 6. mixed list with one outside address -> 400, all rejected + SENT.clear() + st, raw, j = _post(port, path, token=admin_token, + body={"to": [admin_email, "outsider@evil.com"]}) + check(st == 400, f"mixed list with outsider -> 400 (got {st})") + check(not SENT, "mixed list: sender not called") + + # 7. send failure must NOT leak the exception text (could carry the credential) + SENT.clear() + secret = "PWLEAK_SsHh99" + RAISE["exc"] = Exception(f"535 auth failed for user with password {secret}") + st, raw, j = _post(port, path, token=admin_token) + RAISE["exc"] = None + check(st == 502, f"send failure -> 502 (got {st})") + check(secret not in raw, "send-failure response does not leak the exception/credential text") + finally: + httpd.shutdown() + + print() + if FAILS: + print(f"FAILED ({len(FAILS)}):") + for f in FAILS: + print(f" - {f}") + sys.exit(1) + print("ALL PASS (digest test-email endpoint)") + + +if __name__ == "__main__": + main() diff --git a/backend/test_smtp_send.py b/backend/test_smtp_send.py new file mode 100644 index 0000000..5f6a803 --- /dev/null +++ b/backend/test_smtp_send.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +"""Standalone tests for backend/smtp_send.py — config parsing + send paths. + +No network: smtplib.SMTP / SMTP_SSL are monkeypatched with a fake that records +the calls. Run directly or via backend/run_tests.py. +""" +import os +import smtplib + +import smtp_send + +FAILS = [] + + +def check(cond, label): + print(f" {'ok' if cond else 'XX'} {label}") + if not cond: + FAILS.append(label) + + +class FakeServer: + def __init__(self, kind, log): + self.kind, self.log = kind, log + self.logged_in = None + self.sent = [] + + def ehlo(self, *a): + self.log.append(("ehlo", self.kind)) + + def starttls(self, *a, **k): + self.log.append(("starttls", self.kind)) + + def login(self, user, pw): + self.logged_in = (user, pw) + self.log.append(("login", user)) + + def send_message(self, msg, from_addr=None, to_addrs=None): + self.sent.append({"from_addr": from_addr, "to_addrs": to_addrs}) + self.log.append(("send", from_addr)) + + def quit(self): + self.log.append(("quit", self.kind)) + + +def install_fakes(): + """Patch smtplib; return (log, holder) where holder['last'] is the server.""" + log, holder = [], {} + + def mk(kind): + def factory(host, port, timeout=None, context=None): + log.append((kind, host, port)) + srv = FakeServer(kind, log) + holder["last"] = srv + return srv + return factory + + smtplib.SMTP = mk("SMTP") + smtplib.SMTP_SSL = mk("SMTP_SSL") + return log, holder + + +def set_env(**kw): + for k in ("SMTP_HOST", "SMTP_PORT", "SMTP_FROM", "SMTP_USERNAME", + "SMTP_PASSWORD", "SMTP_SECURITY"): + os.environ.pop(k, None) + for k, v in kw.items(): + os.environ["SMTP_" + k.upper()] = str(v) + + +def main(): + # 1. unconfigured + set_env() + check(not smtp_send.smtp_configured(), "smtp_configured() false when unset") + try: + smtp_send.load_smtp_config(); ok = False + except smtp_send.SMTPNotConfigured: + ok = True + check(ok, "load_smtp_config raises SMTPNotConfigured without host") + + # 2. parse + normalize garbage + set_env(host="smtp.x.com", port="nope", security="WEIRD", username="u@x", password="p") + os.environ["SMTP_FROM"] = "f@x" + cfg = smtp_send.load_smtp_config() + check(cfg["host"] == "smtp.x.com", "host parsed") + check(cfg["port"] == 587, "non-numeric port falls back to 587") + check(cfg["security"] == "starttls", "unknown security falls back to starttls") + + # 3. STARTTLS path: plain SMTP, upgrade, auth, multi-recipient + log, holder = install_fakes() + set_env(host="h", port="587", security="starttls", username="u@x", password="pw") + os.environ["SMTP_FROM"] = "f@x" + res = smtp_send.send_email(["a@x.com", "b@x.com"], "s", "body") + srv = holder["last"] + check(log[0] == ("SMTP", "h", 587), "starttls connects via plain SMTP") + check(("starttls", "SMTP") in log, "starttls negotiated") + check(srv.logged_in == ("u@x", "pw"), "login with username/password") + check(srv.sent[0]["from_addr"] == "f@x", "from = SMTP_FROM") + check(srv.sent[0]["to_addrs"] == ["a@x.com", "b@x.com"], "recipients passed through") + check(res["sent_to"] == ["a@x.com", "b@x.com"] and res["from"] == "f@x", "result reports send") + + # 4. implicit TLS path: SMTP_SSL, no STARTTLS + log, holder = install_fakes() + set_env(host="h", port="465", security="tls", username="u@x", password="pw") + os.environ["SMTP_FROM"] = "f@x" + smtp_send.send_email("a@x.com", "s", "b") + check(log[0][0] == "SMTP_SSL", "tls uses SMTP_SSL") + check(all(e[0] != "starttls" for e in log), "no STARTTLS on implicit TLS") + + # 5. security=none, from falls back to username + log, holder = install_fakes() + set_env(host="h", port="25", security="none", username="relay@x", password="") + smtp_send.send_email("a@x.com", "s", "b") + srv = holder["last"] + check(all(e[0] != "starttls" for e in log), "no STARTTLS when security=none") + check(srv.sent[0]["from_addr"] == "relay@x", "from falls back to username") + + # 6. no username -> no login attempted + log, holder = install_fakes() + set_env(host="h", port="25", security="none") + os.environ["SMTP_FROM"] = "f@x" + smtp_send.send_email("a@x.com", "s", "b") + check(holder["last"].logged_in is None, "no login when username blank") + + # 7. no sender at all -> SMTPNotConfigured + set_env(host="h", port="25", security="none") + try: + smtp_send.send_email("a@x.com", "s", "b"); ok = False + except smtp_send.SMTPNotConfigured: + ok = True + check(ok, "no from/username raises SMTPNotConfigured") + + # 8. empty recipients -> ValueError + set_env(host="h", port="587", security="starttls", username="u", password="p") + os.environ["SMTP_FROM"] = "f@x" + try: + smtp_send.send_email([], "s", "b"); ok = False + except ValueError: + ok = True + check(ok, "empty recipients raises ValueError") + + print() + if FAILS: + print(f"FAILED ({len(FAILS)}):") + for f in FAILS: + print(f" - {f}") + raise SystemExit(1) + print("ALL PASS (smtp_send)") + + +if __name__ == "__main__": + main() diff --git a/start9/0.4/docker_entrypoint.sh b/start9/0.4/docker_entrypoint.sh index 3e49f5c..7e94a3b 100755 --- a/start9/0.4/docker_entrypoint.sh +++ b/start9/0.4/docker_entrypoint.sh @@ -69,6 +69,28 @@ elif [ -z "${ANTHROPIC_API_KEY:-}" ]; then echo "[entrypoint] Architect: no API key yet (drop it at $ANTHROPIC_KEY_FILE to enable thesis generation)" fi +# ── Daily-digest SMTP (per-package custom mailbox) ────────────── +# The CRM emails a daily activity digest. Credentials come from the "Configure +# Digest SMTP" StartOS action, which writes one file per field under +# $SECRETS_DIR/smtp. We read them back here (plain cat — never eval) and export +# SMTP_* for the server process. Each value is read only if not already set in +# the service environment, so an operator override still wins. Self-disabling +# until host is present (the digest mailer reports "not configured"). +SMTP_DIR="$SECRETS_DIR/smtp" +if [ -z "${SMTP_HOST:-}" ] && [ -f "$SMTP_DIR/host" ]; then + export SMTP_HOST="$(cat "$SMTP_DIR/host")" + export SMTP_PORT="${SMTP_PORT:-$(cat "$SMTP_DIR/port" 2>/dev/null || echo 587)}" + export SMTP_FROM="${SMTP_FROM:-$(cat "$SMTP_DIR/from" 2>/dev/null || true)}" + export SMTP_USERNAME="${SMTP_USERNAME:-$(cat "$SMTP_DIR/username" 2>/dev/null || true)}" + export SMTP_PASSWORD="${SMTP_PASSWORD:-$(cat "$SMTP_DIR/password" 2>/dev/null || true)}" + export SMTP_SECURITY="${SMTP_SECURITY:-$(cat "$SMTP_DIR/security" 2>/dev/null || echo starttls)}" + echo "[entrypoint] Digest SMTP: configured (host $SMTP_HOST)" +elif [ -n "${SMTP_HOST:-}" ]; then + echo "[entrypoint] Digest SMTP: using SMTP_HOST from the service environment" +else + echo "[entrypoint] Digest SMTP: not configured (use the Configure Digest SMTP action)" +fi + # ── Phase-0 ingest / retrieval env ────────────────────────────── # These are consumed by the ingest pipeline (backend/ingest/) and the MCP # server (backend/mcp/) — NOT by the CRM web server, which ignores them. diff --git a/start9/0.4/startos/actions/configureDigestSmtp.ts b/start9/0.4/startos/actions/configureDigestSmtp.ts new file mode 100644 index 0000000..ccac18e --- /dev/null +++ b/start9/0.4/startos/actions/configureDigestSmtp.ts @@ -0,0 +1,202 @@ +import { i18n } from '../i18n' +import { sdk } from '../sdk' +import { DATA_MOUNT_PATH, IMAGE_ID } from '../utils' + +/** + * "Configure Digest SMTP" action. + * + * The CRM sends a daily activity digest by email. This is the package's OWN + * dedicated SMTP account (per-package custom SMTP) — it is independent of any + * StartOS system-wide SMTP account, so the digest can go through a mailbox of + * its own without one being configured server-wide. + * + * Mirrors setAnthropicApiKey: the credentials are written to files under + * /data/secrets/smtp/{host,port,from,username,password,security} + * and docker_entrypoint.sh reads them ONCE at boot and exports SMTP_* into the + * server process. The backend only ever reads SMTP_* env, so dev (.env) and the + * box share one code path. A service RESTART is required to pick up changes. + * + * Security: the password is piped over STDIN (never argv/env) so it can't appear + * in a process listing — exactly the property setAnthropicApiKey relies on. The + * other fields (host/port/from/username/security) are not secret and go via env. + * Each field lands in its own file so the entrypoint reads it with a plain `cat` + * (no shell-eval of operator-supplied values). + */ + +const { InputSpec, Value } = sdk + +const SMTP_DIR = `${DATA_MOUNT_PATH}/secrets/smtp` + +// Always (re)writes host/port/from/username/security from env. The password is +// rewritten only when one was entered (piped on stdin), so re-saving to change +// the host doesn't wipe a previously stored password. +function writeScript(includePassword: boolean): string { + // umask 077 makes every file owner-only at creation; the explicit chmods are a + // belt-and-suspenders that name each file (matches setAnthropicApiKey, and + // avoids a glob that could mis-expand on an empty dir). + const steps = [ + 'set -eu', + 'umask 077', + 'mkdir -p "$DATA_DIR/secrets/smtp"', + 'chmod 700 "$DATA_DIR/secrets" "$DATA_DIR/secrets/smtp" 2>/dev/null || true', + 'printf %s "$SMTP_HOST" > "$DATA_DIR/secrets/smtp/host"', + 'printf %s "$SMTP_PORT" > "$DATA_DIR/secrets/smtp/port"', + 'printf %s "$SMTP_FROM" > "$DATA_DIR/secrets/smtp/from"', + 'printf %s "$SMTP_USERNAME" > "$DATA_DIR/secrets/smtp/username"', + 'printf %s "$SMTP_SECURITY" > "$DATA_DIR/secrets/smtp/security"', + 'chmod 600 "$DATA_DIR/secrets/smtp/host" "$DATA_DIR/secrets/smtp/port" "$DATA_DIR/secrets/smtp/from" "$DATA_DIR/secrets/smtp/username" "$DATA_DIR/secrets/smtp/security"', + ] + if (includePassword) { + // Password from stdin, stripped of stray CR/LF a paste may carry. + steps.push('tr -d \'\\n\\r\' > "$DATA_DIR/secrets/smtp/password"') + steps.push('chmod 600 "$DATA_DIR/secrets/smtp/password"') + } + return steps.join('; ') +} + +export const inputSpec = InputSpec.of({ + host: Value.text({ + name: i18n('SMTP Host'), + description: i18n('SMTP server hostname, e.g. smtp.gmail.com or email-smtp.us-east-1.amazonaws.com.'), + warning: null, + required: true, + default: null, + masked: false, + placeholder: 'smtp.example.com', + }), + port: Value.number({ + name: i18n('SMTP Port'), + description: i18n('Usually 587 for STARTTLS, 465 for TLS/SSL, or 25 for an unauthenticated relay.'), + warning: null, + required: true, + default: 587, + integer: true, + min: 1, + max: 65535, + placeholder: '587', + }), + security: Value.select({ + name: i18n('Connection Security'), + description: i18n('STARTTLS (port 587) or TLS/SSL (port 465) for hosted providers; None only for a trusted LAN relay.'), + default: 'starttls', + values: { + starttls: i18n('STARTTLS'), + tls: i18n('TLS / SSL'), + none: i18n('None (plaintext)'), + }, + }), + fromAddress: Value.text({ + name: i18n('From Address'), + description: i18n('The address the digest is sent from. Defaults to the username if left blank.'), + warning: null, + required: false, + default: null, + masked: false, + placeholder: 'crm-digest@example.com', + }), + username: Value.text({ + name: i18n('SMTP Username'), + description: i18n('Login for the SMTP account. Leave blank only for an unauthenticated relay.'), + warning: null, + required: false, + default: null, + masked: false, + placeholder: 'crm-digest@example.com', + }), + password: Value.text({ + name: i18n('SMTP Password'), + description: i18n( + 'Password or app-password for the SMTP account. Stored only on this server ' + + 'at /data/secrets/smtp/password (owner-only, 0600). Leave blank to keep the ' + + 'currently saved password unchanged.', + ), + warning: null, + required: false, + default: null, + masked: true, + placeholder: '••••••••', + }), +}) + +export const configureDigestSmtp = sdk.Action.withInput( + // id + 'configure-digest-smtp', + + // metadata + async ({ effects }) => ({ + name: i18n('Configure Digest SMTP'), + description: i18n( + 'Set the SMTP account the CRM uses to email the daily activity digest. ' + + 'This is a dedicated mailbox for this service — it does not require a ' + + 'StartOS system-wide SMTP account. Credentials are stored only on this ' + + 'server under /data/secrets/smtp/ (owner-only). Restart the service after ' + + 'saving for the change to take effect.', + ), + warning: i18n( + 'After saving, restart Ten31 Database (Stop, then Start) so the new SMTP ' + + 'settings are loaded. The running service does not pick them up until it restarts.', + ), + allowedStatuses: 'any', + group: null, + visibility: 'enabled', + }), + + // form input specification + inputSpec, + + // pre-fill: never read the secret back out — always present a blank form. + async ({ effects, prefill }) => null, + + // execution + async ({ effects, input }) => { + const includePassword = !!(input.password && input.password.length > 0) + + const subcontainer = await sdk.SubContainer.of( + effects, + { imageId: IMAGE_ID }, + sdk.Mounts.of().mountVolume({ + volumeId: 'main', + subpath: null, + mountpoint: DATA_MOUNT_PATH, + readonly: false, + }), + 'ten31-database-configure-digest-smtp', + ) + + try { + const options: { env: Record; input?: string } = { + env: { + DATA_DIR: DATA_MOUNT_PATH, + SMTP_HOST: input.host, + SMTP_PORT: String(input.port), + SMTP_FROM: input.fromAddress ?? '', + SMTP_USERNAME: input.username ?? '', + SMTP_SECURITY: input.security, + }, + } + // The password is piped in over stdin — never argv/env — so it cannot + // appear in a process listing. + if (includePassword) options.input = input.password ?? '' + + await subcontainer.execFail( + ['/bin/sh', '-c', writeScript(includePassword)], + options, + 60 * 1000, + ) + } finally { + await subcontainer.destroy() + } + + return { + version: '1', + title: i18n('Digest SMTP settings saved'), + message: i18n( + `SMTP settings were written to ${SMTP_DIR} (owner-only). ` + + (includePassword ? '' : 'The existing password was kept unchanged. ') + + 'IMPORTANT: restart Ten31 Database now (Stop, then Start) so the digest ' + + 'mailer picks up the new settings, then use "Send Test Digest Email" to verify.', + ), + result: null, + } + }, +) diff --git a/start9/0.4/startos/actions/index.ts b/start9/0.4/startos/actions/index.ts index ebd33ef..543bf4d 100644 --- a/start9/0.4/startos/actions/index.ts +++ b/start9/0.4/startos/actions/index.ts @@ -1,5 +1,6 @@ import { sdk } from '../sdk' import { buildSearchIndex } from './buildSearchIndex' +import { configureDigestSmtp } from './configureDigestSmtp' import { refreshSearchIndex } from './refreshSearchIndex' import { resolveDuplicates } from './resolveDuplicates' import { setAnthropicApiKey } from './setAnthropicApiKey' @@ -9,3 +10,4 @@ export const actions = sdk.Actions.of() .addAction(refreshSearchIndex) .addAction(resolveDuplicates) .addAction(setAnthropicApiKey) + .addAction(configureDigestSmtp) diff --git a/start9/0.4/startos/utils.ts b/start9/0.4/startos/utils.ts index 3a4ffa7..7f784d8 100644 --- a/start9/0.4/startos/utils.ts +++ b/start9/0.4/startos/utils.ts @@ -39,8 +39,9 @@ export const PACKAGE_TITLE = 'Ten31 Database' // * 0.1.0:71 (voice by-purpose larger sample + Tier-B: create Gmail draft w/ in-thread reply) // * 0.1.0:72 (stage v2.0 reserve-asset thesis spine as Workshop candidates) // * 0.1.0:73 (replace old settlement spine with v2.0 reserve-asset spine across Architect + outreach prompts, seed constants, and docs; promote v2.0 to the working approved spine + soft-retire old settlement nodes, reversibly, node-level only) -// * Current: 0.1.0:74 (security/privacy hardening — full-eval P0+2×P1: close /assets/ path traversal, add NER backstop to the outreach redaction boundary, filter deleted_at on get-by-id) -export const PACKAGE_VERSION = '0.1.0:74' +// * 0.1.0:74 (security/privacy hardening — full-eval P0+2×P1: close /assets/ path traversal, add NER backstop to the outreach redaction boundary, filter deleted_at on get-by-id) +// * Current: 0.1.0:75 (Phase-A digest SMTP: per-package "Configure Digest SMTP" action writes /data/secrets/smtp/*; entrypoint exports SMTP_*; backend smtp_send.py + admin "send test email" endpoint) +export const PACKAGE_VERSION = '0.1.0:75' export const DATA_MOUNT_PATH = '/data' export const WEB_PORT = 8080 diff --git a/start9/0.4/startos/versions/index.ts b/start9/0.4/startos/versions/index.ts index 6731288..b7f697f 100644 --- a/start9/0.4/startos/versions/index.ts +++ b/start9/0.4/startos/versions/index.ts @@ -35,8 +35,9 @@ import { v_0_1_0_71 } from './v0.1.0.71' import { v_0_1_0_72 } from './v0.1.0.72' import { v_0_1_0_73 } from './v0.1.0.73' import { v_0_1_0_74 } from './v0.1.0.74' +import { v_0_1_0_75 } from './v0.1.0.75' export const versionGraph = VersionGraph.of({ - current: v_0_1_0_74, - other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73], + current: v_0_1_0_75, + other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73, v_0_1_0_74], }) diff --git a/start9/0.4/startos/versions/v0.1.0.75.ts b/start9/0.4/startos/versions/v0.1.0.75.ts new file mode 100644 index 0000000..8fb6cc7 --- /dev/null +++ b/start9/0.4/startos/versions/v0.1.0.75.ts @@ -0,0 +1,26 @@ +import { VersionInfo } from '@start9labs/start-sdk' + +// Phase-A of the daily activity digest: outbound SMTP send capability. Code-only, +// no schema change (migrations are no-ops): +// * New "Configure Digest SMTP" StartOS action (actions/configureDigestSmtp.ts): +// writes a per-package, custom SMTP account to /data/secrets/smtp/{host,port, +// from,username,password,security} — independent of any StartOS system-wide +// SMTP account. Password is piped over stdin (never argv/env), like the +// Anthropic-key action. +// * docker_entrypoint.sh reads those files at boot and exports SMTP_* into the +// server process (env still wins for an operator override). +// * backend/smtp_send.py: stdlib smtplib wrapper reading SMTP_* (one code path +// for dev .env and the box). New admin endpoint POST /api/admin/digest/test-email +// sends a test message to the requesting `to` or to all active admins, to prove +// the pipe before the digest itself (Phase B) is built. +export const v_0_1_0_75 = VersionInfo.of({ + version: '0.1.0:75', + releaseNotes: { + en_US: [ + 'Add outbound email: a "Configure Digest SMTP" action sets a dedicated mailbox for this', + 'service (no server-wide SMTP account required), and a new admin "send test email" action', + 'verifies it works — groundwork for the daily activity digest.', + ].join(' '), + }, + migrations: { up: async () => {}, down: async () => {} }, +})