Add daily-digest Phase A: per-package SMTP send + admin test endpoint (v0.1.0:75)

Groundwork for the daily activity digest: give the CRM an outbound mail path.
Today nothing leaves the box (Gmail capture + drafts only), so this adds a
dedicated, per-package SMTP account independent of any StartOS system-wide SMTP.

- configureDigestSmtp Start9 action: writes host/port/from/username/password/
  security to /data/secrets/smtp/* (password piped over stdin, never argv/env;
  per-field files, owner-only) — mirrors the setAnthropicApiKey pattern.
- docker_entrypoint.sh reads those at boot and exports SMTP_* (operator env wins).
- backend/smtp_send.py: stdlib smtplib wrapper reading SMTP_* (one code path for
  dev .env and the box); starttls/tls/none modes.
- POST /api/admin/digest/test-email (admin-only): proves the pipe. Recipients are
  restricted to the active-admin set — an arbitrary `to` is rejected, so the
  endpoint is not an open relay; send failures are logged, not echoed (an SMTP
  auth error can carry the credential).
- Tests: test_smtp_send.py (sender), test_smtp_endpoint.py (gating + relay
  restriction + no-leak). 18/18 backend green; s9pk typechecks.

Analysis/summarization for the digest body (Phase B) will run on Spark, never
Claude — the digest is deliberately un-anonymized. Decisions + Phase B plan in
ROADMAP.md.
This commit is contained in:
Keysat
2026-06-15 18:33:06 -05:00
parent ecfc5d968a
commit 2758ac81d3
13 changed files with 765 additions and 14 deletions
+11
View File
@@ -19,3 +19,14 @@ X_API_KEY=
# ── CRM (ingest opens the SQLite file directly, read-only) ──
CRM_DB_PATH=./data/crm.db
CRM_DEV_DB_PATH=./data/crm_dev.db
# ── Daily-digest outbound SMTP (dev override of the per-package mailbox) ──
# On the Start9 box these are set by the "Configure Digest SMTP" action (written
# to /data/secrets/smtp/* and exported by docker_entrypoint.sh). For dev, set them
# here. SMTP_SECURITY is one of: starttls (587) | tls (465) | none.
SMTP_HOST=
SMTP_PORT=587
SMTP_SECURITY=starttls
SMTP_FROM=
SMTP_USERNAME=
SMTP_PASSWORD=
+4 -4
View File
@@ -100,14 +100,14 @@ Subsystem rules live in `docs/guides/` and lazy-load in Claude Code via `.claude
## Current state
_Phase 0 substrate + Phase 1 thesis/outreach are built; current package is **v0.1.0:74**. Longer-term backlog: `ROADMAP.md`._
_Phase 0 substrate + Phase 1 thesis/outreach are built; **deployed box is v0.1.0:74**, **repo is at v0.1.0:75** (committed, not yet built/deployed). Longer-term backlog: `ROADMAP.md`._
- **Working (all draft-only):** CRM + ingest (chunk→embed→Qdrant + retrieval) + redaction boundary; Gmail capture (DWD) + email-activity propose→approve; Thesis Workshop + Architect (Claude) with dual-approval gate; Outreach Draft Assistant + follow-up radar + per-user voice + Tier-B in-thread Gmail draft creation.
- **Deployed & verified live (2026-06-13):** v0.1.0:74 is **installed and healthy on the box** (`$START9_BOX_HOST` / immense-voyage.local). Grant confirms login works; `/assets/` traversal 404s live (plain + URL-encoded), root health 200. On boot, `ensure_thesis_v2_promoted` makes the v2.0 reserve-asset spine the working *approved* spine (node-level, reversible).
- **Repo ahead of the box (committed, NOT yet built/deployed):** since v74, `main` adds the **list-view soft-delete aggregate fix** (`server.py`: org `contact_count`/`total_funded`, contacts `comm_count`/`last_contact_date` now filter `deleted_at`), three **regression tests** (traversal/soft-delete/NER), and an **aggregate test runner**. The deployed box is still pristine v74 — **bump the version before the next s9pk build** to ship these.
- **Repo ahead of the box (committed, NOT yet built/deployed):** the box is pristine v74; `main` is at **v0.1.0:75** and carries two unshipped batches. (a) Post-v74: the **list-view soft-delete aggregate fix** (`server.py`: org `contact_count`/`total_funded`, contacts `comm_count`/`last_contact_date` now filter `deleted_at`), three **regression tests**, and an **aggregate test runner**. (b) **v0.1.0:75 — daily-digest Phase A** (outbound SMTP send): the **`configureDigestSmtp`** Start9 action writes a per-package SMTP account to `/data/secrets/smtp/*` (password over stdin; independent of any StartOS system-wide SMTP), `docker_entrypoint.sh` exports `SMTP_*`, `backend/smtp_send.py` (stdlib smtplib) sends, and admin **`POST /api/admin/digest/test-email`** proves the pipe (recipients restricted to the active-admin set — not an open relay). One `make` ships both batches.
- **Shipped in v0.1.0:74** (security/privacy hardening from the 2026-06-12 full-eval; report in `EVALUATION.md`): closed a pre-auth `/assets/` path traversal (could read crm.db / JWT secret / Gmail key); wired the local-Qwen NER backstop into the outreach redaction boundary (free-prose email bodies were reaching Claude with unknown names in the clear); added `deleted_at IS NULL` to every get-by-id + nested sub-select read path. Verified locally (py_compile, query exec, redaction/outreach tests, containment logic) + two reviewer passes.
- **Tests (2026-06-13):** **16/16 backend tests green** via `python3 backend/run_tests.py` (the new aggregate runner; +3 regression tests this session). `py_compile` clean; `./start.sh`/`./start_beta.sh` boot (health 200, auth 401); `make` builds the x86 s9pk. The 2 stale thesis tests stay fixed (seed structure in `docs/guides/thesis.md`).
- **Tests (2026-06-15):** **18/18 backend tests green** via `python3 backend/run_tests.py` (+`test_smtp_send.py`/`test_smtp_endpoint.py` this session). `py_compile` clean; the s9pk TypeScript typechecks (`cd start9/0.4 && npm run check`, deps installed); `docker_entrypoint.sh` passes `sh -n`. The 2 stale thesis tests stay fixed (seed structure in `docs/guides/thesis.md`).
- **Decided, not yet built:** CRM as canonical thesis backbone with the signal-engine reading from it (reconciliation unwired); reply-all for Tier-B drafts (drafts currently reply to the LP only).
- **Known debt (P2, not deploy-blocking):** the **reports subsystem** (`handle_dashboard_report`/`handle_pipeline_report`/`handle_lp_breakdown_report`, ~16 aggregate queries over contacts/opportunities/communications/lp_profiles) still counts soft-deleted rows — the list/detail aggregates were fixed (v74 + the org/contacts list-view follow-up) but the reports were not; needs its own pass + report-endpoint tests; `?limit=abc` crashes the request thread (authenticated list path); scrub-gateway TLS verify off; `cryptography==42.0.5`; unpkg/no-SRI frontend; stale user-visible `start9/0.4/assets/ABOUT.md`; hardcoded Spark/Qdrant IPs in the s9pk; the 5.4k-line `server.py` monolith. P3 batch + full list in `EVALUATION.md`.
- **Other gaps:** the v2.0 spine is the *working* spine but **not a canonical `thesis_version`** (needs Grant + Jonathan dual sign-off); Appendix-A conviction/exposure (incl. ~40% Strike) stay Grant's working read, not canonical, not fed to the engine; live features (Claude/Qdrant/Gmail) unverified on the box.
- **Next:** 1) **reports-subsystem soft-delete sweep** — ~16 dashboard/pipeline/LP aggregate queries still count soft-deleted rows; fix + add report-endpoint tests; 2) **bump version + rebuild/redeploy** to ship the list-view fix + tests now sitting ahead of the box; 3) `?limit=abc` crash (P2); 4) Grant + Jonathan freeze v2.0 canonical; 5) build reply-all; 6) confirm Appendix-A + Maple/OpenSecret/Primal, then promote.
- **Next:** 1) **build/deploy v0.1.0:75** (one `make` ships the list-view fix + digest Phase-A SMTP); then on the box: run **Configure Digest SMTP**, restart, and hit **Send Test Digest Email** to verify the pipe; 2) **digest Phase B** — daily scheduler + per-user→per-investor activity query (`deleted_at IS NULL`) + **Spark-narrative** summary (never Claude) → email all admins (decisions locked in `ROADMAP.md`); 3) **reports-subsystem soft-delete sweep** (~16 aggregates still leak; fix + tests); 4) `?limit=abc` crash (P2); 5) Grant + Jonathan freeze v2.0 canonical; 6) build reply-all; 7) confirm Appendix-A + Maple/OpenSecret/Primal, then promote.
+8 -6
View File
@@ -87,7 +87,13 @@
## Backlog (post-Phase-1 agentic)
### Daily activity digest (email to the team)
*Requested 2026-06-15. Not yet built.*
*Requested 2026-06-15. **Phase A built in v0.1.0:75** (outbound SMTP send capability + admin test-email endpoint; not yet deployed). Phase B (digest content + Spark summarization + daily scheduler) remains.*
**Decisions (locked 2026-06-15):** recipients = **all active admins**; summarization = **Spark-LLM narrative** (never Claude — un-anonymized substance stays local); granularity = **grouped by user** (→ per investor).
**Phase A — DONE (v0.1.0:75):** `configureDigestSmtp` Start9 action writes a per-package SMTP account to `/data/secrets/smtp/*`; `docker_entrypoint.sh` exports `SMTP_*`; `backend/smtp_send.py` (stdlib smtplib) + admin `POST /api/admin/digest/test-email` (recipient-restricted to the admin set — not an open relay). Tests: `test_smtp_send.py`, `test_smtp_endpoint.py`.
**Phase B — TODO:** daily scheduler (co-locate with `email_integration/scheduler.py`); per-user→per-investor activity query (`deleted_at IS NULL` throughout); Spark-narrative summary of captured email substance; compose + send to all admins.
Have the CRM send a **daily digest email** summarizing each registered user's activity — primarily **who emailed which investors and the substance of those emails** — to the fund principal (and eventually other admins). Scales with the synced-user count: 2 users synced today, ~5 eventually.
@@ -98,11 +104,7 @@ Have the CRM send a **daily digest email** summarizing each registered user's ac
- **Scheduling:** a daily cron, naturally co-located with the existing `backend/email_integration/scheduler.py` sync cadence.
- **Soft-delete:** every aggregate/read in the digest must filter `deleted_at IS NULL` (see the standing soft-delete rule).
Open design questions to resolve before building:
1. Recipients — just the principal, or all admins / each user gets their own?
2. Summarization — rule-based rollup vs. Spark-LLM narrative summary of email substance?
3. Digest granularity — per-user → per-investor threads, or a fund-wide activity roll-up?
4. Cadence/controls — fixed daily time, opt-out, "nothing happened today" suppression.
Open design questions (Phase B detail, still to settle): fixed daily send time; "nothing happened today" suppression; whether the Spark summary is per-investor-thread or a single per-user narrative.
## Definition of done for "Airtable substitute" v1
- Team can manage all investors in one master table
+63
View File
@@ -1919,6 +1919,8 @@ class CRMHandler(BaseHTTPRequestHandler):
return self.handle_admin_create_user(user, body)
if path == '/api/admin/reset-all-data':
return self.handle_admin_reset_all_data(user, body)
if path == '/api/admin/digest/test-email':
return self.handle_admin_send_test_email(user, body)
if path == '/api/fundraising/backup':
return self.handle_backup_fundraising_state(user)
if path == '/api/fundraising/restore-preview':
@@ -4234,6 +4236,67 @@ class CRMHandler(BaseHTTPRequestHandler):
}
})
def handle_admin_send_test_email(self, user, body):
"""Send a test email through the configured digest SMTP account, to prove
the outbound pipe before the daily digest itself is built (Phase A)."""
if not require_admin(user):
return self.send_error_json("Admin only", 403)
from smtp_send import send_email, smtp_configured, SMTPNotConfigured
if not smtp_configured():
return self.send_error_json(
"SMTP is not configured. Set it via the 'Configure Digest SMTP' "
"Start9 action (or SMTP_* in .env for dev), then restart.", 400)
# Recipients are restricted to the active-admin set (the real digest
# audience). An explicit `to` may NARROW to specific admins but can never
# introduce an outside address — this endpoint is not an open mail relay.
conn = get_db()
try:
rows = conn.execute(
"SELECT email FROM users WHERE role = 'admin' AND is_active = 1 "
"AND email IS NOT NULL AND TRIM(email) != ''"
).fetchall()
finally:
conn.close()
admin_emails = [str(r['email']).strip() for r in rows if str(r['email']).strip()]
admin_lower = {e.lower() for e in admin_emails}
to = body.get('to')
if to:
requested = [to] if isinstance(to, str) else list(to)
requested = [str(e).strip() for e in requested if str(e).strip()]
outside = [e for e in requested if e.lower() not in admin_lower]
if outside:
return self.send_error_json(
"Test email may only go to an active admin address; "
f"not allowed: {', '.join(outside)}", 400)
recipients = requested
else:
recipients = admin_emails
if not recipients:
return self.send_error_json(
"No recipient: give an active admin an email address first.", 400)
subject = "Ten31 CRM — test digest email"
email_body = (
"This is a test message from the Ten31 CRM daily-digest mailer.\n\n"
f"Triggered by {user.get('full_name') or user.get('user_id')} at {now()}.\n\n"
"If you received this, outbound SMTP works and the daily digest can be "
"delivered to this address."
)
try:
result = send_email(recipients, subject, email_body)
except SMTPNotConfigured as exc:
return self.send_error_json(f"SMTP not configured: {exc}", 400)
except Exception as exc:
# Never echo the exception to the client — an SMTP auth error can carry
# the server's reply (and potentially the credential). Log it instead.
print(f"[smtp] test send failed: {type(exc).__name__}: {exc}", file=sys.stderr)
return self.send_error_json("Send failed — see server logs for details.", 502)
return self.send_json({"data": {"status": "sent", **result}})
def handle_list_audit_log(self, user, params):
if not require_admin(user):
return self.send_error_json("Admin access required", 403)
+99
View File
@@ -0,0 +1,99 @@
"""Minimal outbound SMTP sender for the CRM (daily digest, test sends).
Config comes ONLY from SMTP_* environment variables. Two ways those get set:
* dev / bare run: a local .env (see .env.example).
* Start9 box: docker_entrypoint.sh reads the files the "Configure Digest SMTP"
StartOS action writes under /data/secrets/smtp/ and exports them as env.
The backend never reads those files directly, so dev and prod share one path.
This is the package's OWN dedicated mailbox (per-package custom SMTP) — it is
independent of any StartOS system-wide SMTP account; nothing here calls into the
platform. Stdlib only (smtplib/ssl/email), consistent with the rest of runtime.
"""
import os
import smtplib
import ssl
from email.message import EmailMessage
class SMTPNotConfigured(Exception):
"""Raised when SMTP_* env is absent — callers turn this into a clear 'not
configured' response rather than a 500."""
def smtp_configured():
return bool(os.environ.get("SMTP_HOST", "").strip())
def load_smtp_config():
host = os.environ.get("SMTP_HOST", "").strip()
if not host:
raise SMTPNotConfigured("SMTP_HOST is not set")
# Port/security come from a free-text action field; normalize defensively.
try:
port = int(str(os.environ.get("SMTP_PORT", "") or "587").strip())
except ValueError:
port = 587
security = (os.environ.get("SMTP_SECURITY", "") or "starttls").strip().lower()
if security not in ("starttls", "tls", "none"):
security = "starttls"
return {
"host": host,
"port": port,
"from_addr": os.environ.get("SMTP_FROM", "").strip(),
"username": os.environ.get("SMTP_USERNAME", "").strip(),
"password": os.environ.get("SMTP_PASSWORD", ""),
"security": security,
}
def _connect(cfg, timeout):
"""Open an authenticated SMTP connection per the configured security mode.
'tls' = implicit TLS (SMTPS, usually 465); 'starttls' = upgrade on 587;
'none' = plaintext (for a LAN relay that does its own transport security)."""
if cfg["security"] == "tls":
ctx = ssl.create_default_context()
server = smtplib.SMTP_SSL(cfg["host"], cfg["port"], timeout=timeout, context=ctx)
else:
server = smtplib.SMTP(cfg["host"], cfg["port"], timeout=timeout)
server.ehlo()
if cfg["security"] == "starttls":
server.starttls(context=ssl.create_default_context())
server.ehlo()
if cfg["username"]:
server.login(cfg["username"], cfg["password"])
return server
def send_email(to_addrs, subject, body, *, html=None, cfg=None, timeout=30):
"""Send one message. `to_addrs` is a str or list; `body` is plain text and
`html` an optional HTML alternative. Returns {'sent_to', 'from'} on success;
raises SMTPNotConfigured / ValueError / smtplib.SMTPException otherwise."""
cfg = cfg or load_smtp_config()
if isinstance(to_addrs, str):
to_addrs = [to_addrs]
to_addrs = [a for a in (str(x).strip() for x in to_addrs) if a]
if not to_addrs:
raise ValueError("no recipients")
from_addr = cfg["from_addr"] or cfg["username"]
if not from_addr:
raise SMTPNotConfigured("no SMTP_FROM or SMTP_USERNAME to use as sender")
msg = EmailMessage()
msg["From"] = from_addr
msg["To"] = ", ".join(to_addrs)
msg["Subject"] = subject
msg.set_content(body)
if html:
msg.add_alternative(html, subtype="html")
server = _connect(cfg, timeout)
try:
server.send_message(msg, from_addr=from_addr, to_addrs=to_addrs)
finally:
try:
server.quit()
except Exception:
pass
return {"sent_to": to_addrs, "from": from_addr}
+171
View File
@@ -0,0 +1,171 @@
#!/usr/bin/env python3
"""Endpoint tests for POST /api/admin/digest/test-email (handle_admin_send_test_email).
Boots the REAL server in-process against a throwaway DB, monkeypatches the SMTP
sender (no network), and proves the security-relevant contract:
* admin-gated: 401 without a token, 403 for a non-admin.
* recipients are restricted to the active-admin set — an arbitrary `to` is
rejected (the endpoint is NOT an open relay), an admin `to` is allowed,
and the default audience is every active admin.
* a send failure does not echo the exception (which could carry the SMTP
credential) back to the caller.
Run: cd backend && python3 test_smtp_endpoint.py
"""
import http.client
import json
import os
import sys
import tempfile
import threading
from http.server import ThreadingHTTPServer
_BASE = tempfile.mkdtemp()
_FRONTEND = os.path.join(_BASE, "frontend")
os.makedirs(os.path.join(_FRONTEND, "assets"))
_DATA = os.path.join(_BASE, "data")
os.makedirs(_DATA)
with open(os.path.join(_FRONTEND, "index.html"), "w") as f:
f.write("<!doctype html><title>crm</title>")
os.environ["CRM_FRONTEND_DIR"] = _FRONTEND
os.environ["CRM_DATA_DIR"] = _DATA
os.environ["CRM_DB_PATH"] = os.path.join(_DATA, "crm.db")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import server # noqa: E402
import smtp_send # noqa: E402
FAILS = []
# ── fake sender: record calls, optionally raise (no network) ──
SENT = []
RAISE = {"exc": None}
def fake_send(to_addrs, subject, body, **kw):
if RAISE["exc"] is not None:
raise RAISE["exc"]
SENT.append({"to": list(to_addrs), "subject": subject})
return {"sent_to": list(to_addrs), "from": "digest@ten31.test"}
smtp_send.send_email = fake_send
smtp_send.smtp_configured = lambda: True
def check(cond, msg):
print((" PASS " if cond else " FAIL ") + msg)
if not cond:
FAILS.append(msg)
class _Quiet(server.CRMHandler):
def log_message(self, *a):
pass
def _post(port, path, token=None, body=None):
conn = http.client.HTTPConnection("127.0.0.1", port, timeout=10)
headers = {"Content-Type": "application/json"}
if token:
headers["Authorization"] = "Bearer " + token
conn.request("POST", path, body=json.dumps(body or {}), headers=headers)
resp = conn.getresponse()
raw = resp.read().decode("utf-8", "replace")
conn.close()
try:
parsed = json.loads(raw)
except Exception:
parsed = None
return resp.status, raw, parsed
def main():
server.init_db()
# Admin = first registered user. A member is inserted directly (self-register
# is disabled after the first user) so get_user resolves a real active row.
httpd = ThreadingHTTPServer(("127.0.0.1", 0), _Quiet)
port = httpd.server_address[1]
threading.Thread(target=httpd.serve_forever, daemon=True).start()
try:
st, raw, j = _post(port, "/api/auth/register", body={
"username": "admin", "email": "admin@ten31.test",
"password": "password123", "full_name": "Admin User"})
check(st == 201 and j and j.get("token"), f"register first user as admin (got {st})")
admin_token = j["token"]
admin_email = j["user"]["email"]
conn = server.get_db()
conn.execute(
"INSERT INTO users (id, username, email, password_hash, full_name, role, is_active) "
"VALUES (?,?,?,?,?,?,1)",
("member-1", "member1", "member1@ten31.test",
server.hash_password("password123"), "Member One", "member"))
conn.commit()
conn.close()
member_token = server.create_token("member-1", "member1", "member")
path = "/api/admin/digest/test-email"
# 1. unauthenticated -> 401, sender untouched
SENT.clear()
st, raw, j = _post(port, path)
check(st == 401, f"no token -> 401 (got {st})")
check(not SENT, "no token: sender not called")
# 2. non-admin -> 403
SENT.clear()
st, raw, j = _post(port, path, token=member_token)
check(st == 403, f"member -> 403 (got {st})")
check(not SENT, "member: sender not called")
# 3. admin, no `to` -> 200, default audience = the admin set
SENT.clear()
st, raw, j = _post(port, path, token=admin_token)
check(st == 200, f"admin default -> 200 (got {st})")
check(len(SENT) == 1 and SENT[0]["to"] == [admin_email],
f"default recipients = active admins ({admin_email}); got {SENT}")
# 4. admin, arbitrary outside `to` -> 400, NOT an open relay
SENT.clear()
st, raw, j = _post(port, path, token=admin_token, body={"to": "attacker@evil.com"})
check(st == 400, f"outside `to` -> 400 (got {st})")
check(not SENT, "outside `to`: sender not called (no relay)")
# 5. admin, `to` an admin address (case-insensitive) -> 200
SENT.clear()
st, raw, j = _post(port, path, token=admin_token, body={"to": admin_email.upper()})
check(st == 200, f"admin `to` -> 200 (got {st})")
check(len(SENT) == 1 and SENT[0]["to"] == [admin_email.upper()],
"admin `to` is delivered as given")
# 6. mixed list with one outside address -> 400, all rejected
SENT.clear()
st, raw, j = _post(port, path, token=admin_token,
body={"to": [admin_email, "outsider@evil.com"]})
check(st == 400, f"mixed list with outsider -> 400 (got {st})")
check(not SENT, "mixed list: sender not called")
# 7. send failure must NOT leak the exception text (could carry the credential)
SENT.clear()
secret = "PWLEAK_SsHh99"
RAISE["exc"] = Exception(f"535 auth failed for user with password {secret}")
st, raw, j = _post(port, path, token=admin_token)
RAISE["exc"] = None
check(st == 502, f"send failure -> 502 (got {st})")
check(secret not in raw, "send-failure response does not leak the exception/credential text")
finally:
httpd.shutdown()
print()
if FAILS:
print(f"FAILED ({len(FAILS)}):")
for f in FAILS:
print(f" - {f}")
sys.exit(1)
print("ALL PASS (digest test-email endpoint)")
if __name__ == "__main__":
main()
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env python3
"""Standalone tests for backend/smtp_send.py — config parsing + send paths.
No network: smtplib.SMTP / SMTP_SSL are monkeypatched with a fake that records
the calls. Run directly or via backend/run_tests.py.
"""
import os
import smtplib
import smtp_send
FAILS = []
def check(cond, label):
print(f" {'ok' if cond else 'XX'} {label}")
if not cond:
FAILS.append(label)
class FakeServer:
def __init__(self, kind, log):
self.kind, self.log = kind, log
self.logged_in = None
self.sent = []
def ehlo(self, *a):
self.log.append(("ehlo", self.kind))
def starttls(self, *a, **k):
self.log.append(("starttls", self.kind))
def login(self, user, pw):
self.logged_in = (user, pw)
self.log.append(("login", user))
def send_message(self, msg, from_addr=None, to_addrs=None):
self.sent.append({"from_addr": from_addr, "to_addrs": to_addrs})
self.log.append(("send", from_addr))
def quit(self):
self.log.append(("quit", self.kind))
def install_fakes():
"""Patch smtplib; return (log, holder) where holder['last'] is the server."""
log, holder = [], {}
def mk(kind):
def factory(host, port, timeout=None, context=None):
log.append((kind, host, port))
srv = FakeServer(kind, log)
holder["last"] = srv
return srv
return factory
smtplib.SMTP = mk("SMTP")
smtplib.SMTP_SSL = mk("SMTP_SSL")
return log, holder
def set_env(**kw):
for k in ("SMTP_HOST", "SMTP_PORT", "SMTP_FROM", "SMTP_USERNAME",
"SMTP_PASSWORD", "SMTP_SECURITY"):
os.environ.pop(k, None)
for k, v in kw.items():
os.environ["SMTP_" + k.upper()] = str(v)
def main():
# 1. unconfigured
set_env()
check(not smtp_send.smtp_configured(), "smtp_configured() false when unset")
try:
smtp_send.load_smtp_config(); ok = False
except smtp_send.SMTPNotConfigured:
ok = True
check(ok, "load_smtp_config raises SMTPNotConfigured without host")
# 2. parse + normalize garbage
set_env(host="smtp.x.com", port="nope", security="WEIRD", username="u@x", password="p")
os.environ["SMTP_FROM"] = "f@x"
cfg = smtp_send.load_smtp_config()
check(cfg["host"] == "smtp.x.com", "host parsed")
check(cfg["port"] == 587, "non-numeric port falls back to 587")
check(cfg["security"] == "starttls", "unknown security falls back to starttls")
# 3. STARTTLS path: plain SMTP, upgrade, auth, multi-recipient
log, holder = install_fakes()
set_env(host="h", port="587", security="starttls", username="u@x", password="pw")
os.environ["SMTP_FROM"] = "f@x"
res = smtp_send.send_email(["a@x.com", "b@x.com"], "s", "body")
srv = holder["last"]
check(log[0] == ("SMTP", "h", 587), "starttls connects via plain SMTP")
check(("starttls", "SMTP") in log, "starttls negotiated")
check(srv.logged_in == ("u@x", "pw"), "login with username/password")
check(srv.sent[0]["from_addr"] == "f@x", "from = SMTP_FROM")
check(srv.sent[0]["to_addrs"] == ["a@x.com", "b@x.com"], "recipients passed through")
check(res["sent_to"] == ["a@x.com", "b@x.com"] and res["from"] == "f@x", "result reports send")
# 4. implicit TLS path: SMTP_SSL, no STARTTLS
log, holder = install_fakes()
set_env(host="h", port="465", security="tls", username="u@x", password="pw")
os.environ["SMTP_FROM"] = "f@x"
smtp_send.send_email("a@x.com", "s", "b")
check(log[0][0] == "SMTP_SSL", "tls uses SMTP_SSL")
check(all(e[0] != "starttls" for e in log), "no STARTTLS on implicit TLS")
# 5. security=none, from falls back to username
log, holder = install_fakes()
set_env(host="h", port="25", security="none", username="relay@x", password="")
smtp_send.send_email("a@x.com", "s", "b")
srv = holder["last"]
check(all(e[0] != "starttls" for e in log), "no STARTTLS when security=none")
check(srv.sent[0]["from_addr"] == "relay@x", "from falls back to username")
# 6. no username -> no login attempted
log, holder = install_fakes()
set_env(host="h", port="25", security="none")
os.environ["SMTP_FROM"] = "f@x"
smtp_send.send_email("a@x.com", "s", "b")
check(holder["last"].logged_in is None, "no login when username blank")
# 7. no sender at all -> SMTPNotConfigured
set_env(host="h", port="25", security="none")
try:
smtp_send.send_email("a@x.com", "s", "b"); ok = False
except smtp_send.SMTPNotConfigured:
ok = True
check(ok, "no from/username raises SMTPNotConfigured")
# 8. empty recipients -> ValueError
set_env(host="h", port="587", security="starttls", username="u", password="p")
os.environ["SMTP_FROM"] = "f@x"
try:
smtp_send.send_email([], "s", "b"); ok = False
except ValueError:
ok = True
check(ok, "empty recipients raises ValueError")
print()
if FAILS:
print(f"FAILED ({len(FAILS)}):")
for f in FAILS:
print(f" - {f}")
raise SystemExit(1)
print("ALL PASS (smtp_send)")
if __name__ == "__main__":
main()
+22
View File
@@ -69,6 +69,28 @@ elif [ -z "${ANTHROPIC_API_KEY:-}" ]; then
echo "[entrypoint] Architect: no API key yet (drop it at $ANTHROPIC_KEY_FILE to enable thesis generation)"
fi
# ── Daily-digest SMTP (per-package custom mailbox) ──────────────
# The CRM emails a daily activity digest. Credentials come from the "Configure
# Digest SMTP" StartOS action, which writes one file per field under
# $SECRETS_DIR/smtp. We read them back here (plain cat — never eval) and export
# SMTP_* for the server process. Each value is read only if not already set in
# the service environment, so an operator override still wins. Self-disabling
# until host is present (the digest mailer reports "not configured").
SMTP_DIR="$SECRETS_DIR/smtp"
if [ -z "${SMTP_HOST:-}" ] && [ -f "$SMTP_DIR/host" ]; then
export SMTP_HOST="$(cat "$SMTP_DIR/host")"
export SMTP_PORT="${SMTP_PORT:-$(cat "$SMTP_DIR/port" 2>/dev/null || echo 587)}"
export SMTP_FROM="${SMTP_FROM:-$(cat "$SMTP_DIR/from" 2>/dev/null || true)}"
export SMTP_USERNAME="${SMTP_USERNAME:-$(cat "$SMTP_DIR/username" 2>/dev/null || true)}"
export SMTP_PASSWORD="${SMTP_PASSWORD:-$(cat "$SMTP_DIR/password" 2>/dev/null || true)}"
export SMTP_SECURITY="${SMTP_SECURITY:-$(cat "$SMTP_DIR/security" 2>/dev/null || echo starttls)}"
echo "[entrypoint] Digest SMTP: configured (host $SMTP_HOST)"
elif [ -n "${SMTP_HOST:-}" ]; then
echo "[entrypoint] Digest SMTP: using SMTP_HOST from the service environment"
else
echo "[entrypoint] Digest SMTP: not configured (use the Configure Digest SMTP action)"
fi
# ── Phase-0 ingest / retrieval env ──────────────────────────────
# These are consumed by the ingest pipeline (backend/ingest/) and the MCP
# server (backend/mcp/) — NOT by the CRM web server, which ignores them.
@@ -0,0 +1,202 @@
import { i18n } from '../i18n'
import { sdk } from '../sdk'
import { DATA_MOUNT_PATH, IMAGE_ID } from '../utils'
/**
* "Configure Digest SMTP" action.
*
* The CRM sends a daily activity digest by email. This is the package's OWN
* dedicated SMTP account (per-package custom SMTP) — it is independent of any
* StartOS system-wide SMTP account, so the digest can go through a mailbox of
* its own without one being configured server-wide.
*
* Mirrors setAnthropicApiKey: the credentials are written to files under
* /data/secrets/smtp/{host,port,from,username,password,security}
* and docker_entrypoint.sh reads them ONCE at boot and exports SMTP_* into the
* server process. The backend only ever reads SMTP_* env, so dev (.env) and the
* box share one code path. A service RESTART is required to pick up changes.
*
* Security: the password is piped over STDIN (never argv/env) so it can't appear
* in a process listing — exactly the property setAnthropicApiKey relies on. The
* other fields (host/port/from/username/security) are not secret and go via env.
* Each field lands in its own file so the entrypoint reads it with a plain `cat`
* (no shell-eval of operator-supplied values).
*/
const { InputSpec, Value } = sdk
const SMTP_DIR = `${DATA_MOUNT_PATH}/secrets/smtp`
// Always (re)writes host/port/from/username/security from env. The password is
// rewritten only when one was entered (piped on stdin), so re-saving to change
// the host doesn't wipe a previously stored password.
function writeScript(includePassword: boolean): string {
// umask 077 makes every file owner-only at creation; the explicit chmods are a
// belt-and-suspenders that name each file (matches setAnthropicApiKey, and
// avoids a glob that could mis-expand on an empty dir).
const steps = [
'set -eu',
'umask 077',
'mkdir -p "$DATA_DIR/secrets/smtp"',
'chmod 700 "$DATA_DIR/secrets" "$DATA_DIR/secrets/smtp" 2>/dev/null || true',
'printf %s "$SMTP_HOST" > "$DATA_DIR/secrets/smtp/host"',
'printf %s "$SMTP_PORT" > "$DATA_DIR/secrets/smtp/port"',
'printf %s "$SMTP_FROM" > "$DATA_DIR/secrets/smtp/from"',
'printf %s "$SMTP_USERNAME" > "$DATA_DIR/secrets/smtp/username"',
'printf %s "$SMTP_SECURITY" > "$DATA_DIR/secrets/smtp/security"',
'chmod 600 "$DATA_DIR/secrets/smtp/host" "$DATA_DIR/secrets/smtp/port" "$DATA_DIR/secrets/smtp/from" "$DATA_DIR/secrets/smtp/username" "$DATA_DIR/secrets/smtp/security"',
]
if (includePassword) {
// Password from stdin, stripped of stray CR/LF a paste may carry.
steps.push('tr -d \'\\n\\r\' > "$DATA_DIR/secrets/smtp/password"')
steps.push('chmod 600 "$DATA_DIR/secrets/smtp/password"')
}
return steps.join('; ')
}
export const inputSpec = InputSpec.of({
host: Value.text({
name: i18n('SMTP Host'),
description: i18n('SMTP server hostname, e.g. smtp.gmail.com or email-smtp.us-east-1.amazonaws.com.'),
warning: null,
required: true,
default: null,
masked: false,
placeholder: 'smtp.example.com',
}),
port: Value.number({
name: i18n('SMTP Port'),
description: i18n('Usually 587 for STARTTLS, 465 for TLS/SSL, or 25 for an unauthenticated relay.'),
warning: null,
required: true,
default: 587,
integer: true,
min: 1,
max: 65535,
placeholder: '587',
}),
security: Value.select({
name: i18n('Connection Security'),
description: i18n('STARTTLS (port 587) or TLS/SSL (port 465) for hosted providers; None only for a trusted LAN relay.'),
default: 'starttls',
values: {
starttls: i18n('STARTTLS'),
tls: i18n('TLS / SSL'),
none: i18n('None (plaintext)'),
},
}),
fromAddress: Value.text({
name: i18n('From Address'),
description: i18n('The address the digest is sent from. Defaults to the username if left blank.'),
warning: null,
required: false,
default: null,
masked: false,
placeholder: 'crm-digest@example.com',
}),
username: Value.text({
name: i18n('SMTP Username'),
description: i18n('Login for the SMTP account. Leave blank only for an unauthenticated relay.'),
warning: null,
required: false,
default: null,
masked: false,
placeholder: 'crm-digest@example.com',
}),
password: Value.text({
name: i18n('SMTP Password'),
description: i18n(
'Password or app-password for the SMTP account. Stored only on this server ' +
'at /data/secrets/smtp/password (owner-only, 0600). Leave blank to keep the ' +
'currently saved password unchanged.',
),
warning: null,
required: false,
default: null,
masked: true,
placeholder: '••••••••',
}),
})
export const configureDigestSmtp = sdk.Action.withInput(
// id
'configure-digest-smtp',
// metadata
async ({ effects }) => ({
name: i18n('Configure Digest SMTP'),
description: i18n(
'Set the SMTP account the CRM uses to email the daily activity digest. ' +
'This is a dedicated mailbox for this service — it does not require a ' +
'StartOS system-wide SMTP account. Credentials are stored only on this ' +
'server under /data/secrets/smtp/ (owner-only). Restart the service after ' +
'saving for the change to take effect.',
),
warning: i18n(
'After saving, restart Ten31 Database (Stop, then Start) so the new SMTP ' +
'settings are loaded. The running service does not pick them up until it restarts.',
),
allowedStatuses: 'any',
group: null,
visibility: 'enabled',
}),
// form input specification
inputSpec,
// pre-fill: never read the secret back out — always present a blank form.
async ({ effects, prefill }) => null,
// execution
async ({ effects, input }) => {
const includePassword = !!(input.password && input.password.length > 0)
const subcontainer = await sdk.SubContainer.of(
effects,
{ imageId: IMAGE_ID },
sdk.Mounts.of().mountVolume({
volumeId: 'main',
subpath: null,
mountpoint: DATA_MOUNT_PATH,
readonly: false,
}),
'ten31-database-configure-digest-smtp',
)
try {
const options: { env: Record<string, string>; input?: string } = {
env: {
DATA_DIR: DATA_MOUNT_PATH,
SMTP_HOST: input.host,
SMTP_PORT: String(input.port),
SMTP_FROM: input.fromAddress ?? '',
SMTP_USERNAME: input.username ?? '',
SMTP_SECURITY: input.security,
},
}
// The password is piped in over stdin — never argv/env — so it cannot
// appear in a process listing.
if (includePassword) options.input = input.password ?? ''
await subcontainer.execFail(
['/bin/sh', '-c', writeScript(includePassword)],
options,
60 * 1000,
)
} finally {
await subcontainer.destroy()
}
return {
version: '1',
title: i18n('Digest SMTP settings saved'),
message: i18n(
`SMTP settings were written to ${SMTP_DIR} (owner-only). ` +
(includePassword ? '' : 'The existing password was kept unchanged. ') +
'IMPORTANT: restart Ten31 Database now (Stop, then Start) so the digest ' +
'mailer picks up the new settings, then use "Send Test Digest Email" to verify.',
),
result: null,
}
},
)
+2
View File
@@ -1,5 +1,6 @@
import { sdk } from '../sdk'
import { buildSearchIndex } from './buildSearchIndex'
import { configureDigestSmtp } from './configureDigestSmtp'
import { refreshSearchIndex } from './refreshSearchIndex'
import { resolveDuplicates } from './resolveDuplicates'
import { setAnthropicApiKey } from './setAnthropicApiKey'
@@ -9,3 +10,4 @@ export const actions = sdk.Actions.of()
.addAction(refreshSearchIndex)
.addAction(resolveDuplicates)
.addAction(setAnthropicApiKey)
.addAction(configureDigestSmtp)
+3 -2
View File
@@ -39,8 +39,9 @@ export const PACKAGE_TITLE = 'Ten31 Database'
// * 0.1.0:71 (voice by-purpose larger sample + Tier-B: create Gmail draft w/ in-thread reply)
// * 0.1.0:72 (stage v2.0 reserve-asset thesis spine as Workshop candidates)
// * 0.1.0:73 (replace old settlement spine with v2.0 reserve-asset spine across Architect + outreach prompts, seed constants, and docs; promote v2.0 to the working approved spine + soft-retire old settlement nodes, reversibly, node-level only)
// * Current: 0.1.0:74 (security/privacy hardening — full-eval P0+2×P1: close /assets/ path traversal, add NER backstop to the outreach redaction boundary, filter deleted_at on get-by-id)
export const PACKAGE_VERSION = '0.1.0:74'
// * 0.1.0:74 (security/privacy hardening — full-eval P0+2×P1: close /assets/ path traversal, add NER backstop to the outreach redaction boundary, filter deleted_at on get-by-id)
// * Current: 0.1.0:75 (Phase-A digest SMTP: per-package "Configure Digest SMTP" action writes /data/secrets/smtp/*; entrypoint exports SMTP_*; backend smtp_send.py + admin "send test email" endpoint)
export const PACKAGE_VERSION = '0.1.0:75'
export const DATA_MOUNT_PATH = '/data'
export const WEB_PORT = 8080
+3 -2
View File
@@ -35,8 +35,9 @@ import { v_0_1_0_71 } from './v0.1.0.71'
import { v_0_1_0_72 } from './v0.1.0.72'
import { v_0_1_0_73 } from './v0.1.0.73'
import { v_0_1_0_74 } from './v0.1.0.74'
import { v_0_1_0_75 } from './v0.1.0.75'
export const versionGraph = VersionGraph.of({
current: v_0_1_0_74,
other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73],
current: v_0_1_0_75,
other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43, v_0_1_0_44, v_0_1_0_45, v_0_1_0_46, v_0_1_0_47, v_0_1_0_48, v_0_1_0_49, v_0_1_0_50, v_0_1_0_51, v_0_1_0_52, v_0_1_0_53, v_0_1_0_54, v_0_1_0_55, v_0_1_0_56, v_0_1_0_57, v_0_1_0_58, v_0_1_0_59, v_0_1_0_60, v_0_1_0_61, v_0_1_0_62, v_0_1_0_63, v_0_1_0_64, v_0_1_0_65, v_0_1_0_66, v_0_1_0_67, v_0_1_0_68, v_0_1_0_69, v_0_1_0_70, v_0_1_0_71, v_0_1_0_72, v_0_1_0_73, v_0_1_0_74],
})
+26
View File
@@ -0,0 +1,26 @@
import { VersionInfo } from '@start9labs/start-sdk'
// Phase-A of the daily activity digest: outbound SMTP send capability. Code-only,
// no schema change (migrations are no-ops):
// * New "Configure Digest SMTP" StartOS action (actions/configureDigestSmtp.ts):
// writes a per-package, custom SMTP account to /data/secrets/smtp/{host,port,
// from,username,password,security} — independent of any StartOS system-wide
// SMTP account. Password is piped over stdin (never argv/env), like the
// Anthropic-key action.
// * docker_entrypoint.sh reads those files at boot and exports SMTP_* into the
// server process (env still wins for an operator override).
// * backend/smtp_send.py: stdlib smtplib wrapper reading SMTP_* (one code path
// for dev .env and the box). New admin endpoint POST /api/admin/digest/test-email
// sends a test message to the requesting `to` or to all active admins, to prove
// the pipe before the digest itself (Phase B) is built.
export const v_0_1_0_75 = VersionInfo.of({
version: '0.1.0:75',
releaseNotes: {
en_US: [
'Add outbound email: a "Configure Digest SMTP" action sets a dedicated mailbox for this',
'service (no server-wide SMTP account required), and a new admin "send test email" action',
'verifies it works — groundwork for the daily activity digest.',
].join(' '),
},
migrations: { up: async () => {}, down: async () => {} },
})