diff --git a/.gitignore b/.gitignore index 3428f15..e5390b4 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ javascript/ .DS_Store **/__pycache__/ *.pyc +.claude/ diff --git a/README.md b/README.md index 2772c1a..afdf3cd 100644 --- a/README.md +++ b/README.md @@ -1,44 +1,75 @@ -# Boardroom Map — a private document-review panel for your DGX Sparks +# Boardroom Map — private board-deck grading on your DGX Sparks -Boardroom Map is a StartOS service (`.s9pk`) that lets you **drop confidential -documents in and have a panel of local LLMs review them** on your NVIDIA DGX -Sparks. You pick the models and the personas (lenses), and how many reviews to -run; each reviewer writes a report, and an optional **local lead reviewer** -synthesizes them into one consolidated report. There is **no frontier model and -no cloud API key** — in the default air-gapped mode the documents and reviews +Boardroom Map is a StartOS service (`.s9pk`) that **grades portfolio-company +board decks with a panel of local LLMs** on your NVIDIA DGX Sparks. Drop each +company's deck into `inbox//`; the panel grades it against the +**BDEF v1.1 framework** (Girdley + Munger/Buffett), an optional local +**adjudicator** reconciles the panel, and a deterministic Python scorer computes +a 0–100 composite that lands on the company's **running scorecard ledger**. A +web dashboard shows per-company trends. There is **no frontier model and no +cloud API key** — in the default air-gapped mode the decks and their grades never leave your hardware. -It is a sibling of [Nightshift](../nightshift) and reuses the same control-plane +It is a sibling of [Chambers](../chambers) and reuses the same control-plane pattern (a GPU-free orchestrator on StartOS driving the Sparks over SSH), but -with the swarm, the git blackboard, and the Claude overseer removed and replaced -by an on-demand **document-review pipeline**. +swaps the free-form document-review panel for a **deterministic deck-grading +pipeline** with pinned KPI targets and per-company ledgers. + +## The scoring model + +`composite (0–100) = quant 60 + qual 40 − red flags (capped at 15)` + +- **Quantitative 60:** profitability KPI attainment **30** (heaviest slice), + other measurable KPIs **20**, **forecast integrity 10** — deck N's actuals are + chained against deck N−1's stated targets, so moved goalposts cost points. + KPI credit is linear above a floor ratio (default 0.5 → zero credit below). +- **Qualitative 40:** eight BDEF categories (A–H) × 5 points, scored by the + panel with evidence quotes; thin evidence scales down. +- **Red flags:** up to **−15**; silently dropped KPIs are auto-flagged (capped), + and flags raised by a single grader are damped by 0.5. + +Every knob lives in config (`weights`, per-company `pinnedTargets` and +`kpiAliases`) so the model can be retuned without a rebuild. ## Architecture ``` StartOS box (control plane, no GPU) DGX Spark(s) ┌────────────────────────────────────┐ ┌───────────────────────────────┐ -│ FastAPI web UI + job runner │ SSH │ per-job Docker network │ -│ • inbox (drop documents) │ ───────▶│ (──internal in airgapped) │ +│ FastAPI dashboard + job runner │ SSH │ per-job Docker network │ +│ • inbox// (decks) │ ───────▶│ (──internal in airgapped) │ │ • extract text (PDF/DOCX/TXT/MD) │ rsync │ ┌─────────┐ ┌────────────┐ │ │ • plan model "waves" │ ───────▶│ │ vLLM(s) │◀─│ LiteLLM │ │ -│ • launch reviewer containers │ │ └─────────┘ │ router │ │ -│ • pull reports, synthesize, wipe │◀─────── │ ┌──────────────┐ ▲ │ │ -│ • reports saved here only │ rsync │ │ reviewer ×N │──┘ │ │ +│ • extractor → graders → adjudicator│ │ └─────────┘ │ router │ │ +│ • deterministic composite scorer │◀─────── │ ┌──────────────┐ ▲ │ │ +│ • per-company ledgers + scorecards │ rsync │ │ grader ×N │──┘ │ │ └────────────────────────────────────┘ │ │ (read-only, │ │ │ │ │ sandboxed) │ │ │ │ └──────────────┘ │ │ └───────────────────────────────┘ ``` -- **Reviewers** are one-shot, read-only, hardened containers (non-root, +- **Graders** are one-shot, read-only, hardened containers (non-root, `--cap-drop ALL`, read-only rootfs, no docker socket). In air-gapped mode they sit on an `--internal` network and can reach only the model proxy. - **Waves:** the job runner serves models in waves bounded by `maxConcurrentModels`, so a panel can span more models than fit in GPU memory at once. -- **Confidentiality:** documents are extracted to text on the StartOS box; only +- **Air-gap modes:** `airgapped` (default — graders reach only the on-Spark + model proxy, zero egress, models pre-pulled) or `local_services` (graders may + reach LAN services like SearXNG and the second Spark — has egress unless + firewalled). +- **Confidentiality:** decks are extracted to text on the StartOS box; only text crosses to the Sparks, and it is wiped from the Sparks after the job. + Scorecards and ledgers live only on the StartOS box. + +## Setup order + +Configure Sparks → Test Spark Connection → Configure Models → Configure Graders +→ Configure Grading (rubric, air-gap, weights) → Configure Companies (slugs, +KPI aliases, pinned targets — especially profitability thresholds) → drop decks +into `inbox//2026-Q2-deck.pdf` → **Grade Decks** → watch the +dashboard. ## Repo layout @@ -46,25 +77,25 @@ StartOS box (control plane, no GPU) DGX Spark(s) startos/ StartOS package definition (TypeScript / start-sdk) manifest/ main.ts interfaces.ts versions/ file-models/ actions/ orchestrator/ The control-plane app (Python) - app.py FastAPI UI + JSON API - jobs.py the job runner (extract → serve waves → review → synthesize) + app.py FastAPI dashboard + JSON API + jobs.py the job runner (extract → serve waves → grade → adjudicate → score) serving.py vLLM + LiteLLM router on the Sparks, in waves - reviewers.py launch the reviewer panel - synthesis.py the local lead reviewer + graders.py launch the grading panel + adjudicator.py the local lead grader extraction.py PDF/DOCX/TXT/MD → text (on the StartOS box) - preflight.py probe models before launching reviewers + preflight.py probe models before launching graders spark_client.py SSH/rsync helpers + bdef.md the baked-in BDEF v1.1 rubric bm_config.py config defaults (mirrors startos/file-models/config.ts) -sandbox/ reviewer image (built ON the Spark, not packed in the s9pk) - grader_agent.py reviewer.Dockerfile build.sh -openclaw/ what each Spark needs provisioned (OpenClaw's job) +sandbox/ grader image (built ON the Spark, not packed in the s9pk) + grader_agent.py grader.Dockerfile build.sh ``` ## Build -Same path as Nightshift — GitHub CI (`.github/workflows/build.yml`) or a local -build with `start-cli` (see the s9pk-build-on-mac recipe). The vLLM and reviewer -images are built **on the Sparks**, not packed into the `.s9pk`. +GitHub CI (`.github/workflows/build.yml`) or a local build with `start-cli` +(see the s9pk-build-on-mac recipe). The vLLM and grader images are built **on +the Sparks**, not packed into the `.s9pk`. ``` npm ci && npm run check && npm run build # type-check + bundle @@ -74,5 +105,5 @@ make # pack the .s9pk (needs start-cli) ## Status v0.1 — source complete, `tsc`-clean and Python-syntax-clean. Not yet validated -against live Sparks. See `openclaw/OPENCLAW_SPEC.md` for the Spark-side -provisioning (HF model pre-pull is required for air-gapped runs). +against live Sparks. HF model pre-pull on the head Spark is required for +air-gapped runs. diff --git a/assets/instructions.md b/assets/instructions.md index 2250354..aae1c3f 100644 --- a/assets/instructions.md +++ b/assets/instructions.md @@ -1,13 +1,14 @@ # Boardroom Map -Drop confidential documents in and convene a **panel of local LLMs** running on -your DGX Sparks to review them. You choose the models, the personas (lenses), and -how many reviews. An optional **local lead reviewer** synthesizes the panel into -one consolidated report. There is **no frontier model and no cloud key** — in the -default air-gapped mode the documents and their reviews never leave your hardware. +Drop portfolio-company board decks in and have a **panel of local LLMs** running +on your DGX Sparks grade them against the **BDEF v1.1 framework** (Girdley + +Munger/Buffett). A deterministic scorer turns the panel's grades into a 0–100 +composite and appends it to each company's **running scorecard ledger**; the web +dashboard shows the trends. There is **no frontier model and no cloud key** — in +the default air-gapped mode your confidential decks never leave your hardware. -Boardroom Map is a *control plane*: nothing serves or runs on your StartOS box (it only -SSHes to the Sparks and extracts document text on CPU). +Boardroom Map is a *control plane*: nothing serves or runs on your StartOS box +(it only SSHes to the Sparks and extracts deck text on CPU). ## Setup (run the Actions in order) @@ -16,32 +17,54 @@ SSHes to the Sparks and extracts document text on CPU). 2. **Configure Models** — the catalog of local models to serve (alias → HF id → which Spark → port), and serving knobs. For air-gapped runs every model must be on the **head Spark** and present in its HF cache. -3. **Configure Reviewers** — the panel: one entry per review, each a model + a - persona (the lens it reads through) + an optional temperature. -4. **Configure Review** — the rubric, the **Network Mode** (air-gapped vs - local-services), synthesis on/off + lead model, and whether to wipe documents - from the Sparks afterward. +3. **Configure Graders** — the panel: one entry per grader, each a model + a + persona (the lens it grades through — e.g. a Munger inversion skeptic, a + Girdley operator, a skeptical CFO) + an optional temperature. +4. **Configure Grading** — the BDEF rubric override (empty = the built-in + BDEF v1.1), the **Network Mode** (air-gapped vs local-services), the + extractor + adjudicator models, deck retention, and the scoring weights. +5. **Configure Companies** — one entry per portfolio company: its inbox **slug**, + display name, KPI aliases, and **pinned KPI targets**. Pin the profitability + thresholds especially — profitability carries the heaviest weight. -## Running a review +## Grading decks -1. Open the **Web UI** and drag your documents (PDF / DOCX / TXT / MD) onto the - inbox (or drop them in the service's `inbox` folder). -2. Click **Run Review** (or enable *auto-run on drop*). +1. Drop each company's deck into its inbox folder, e.g. + `inbox/acme-widgets/2026-Q2-deck.pdf` (PDF / DOCX / TXT / MD), via the + **Web UI** or the service's `inbox` directory. +2. Run **Grade Decks** (or enable *auto-grade on drop*). 3. Watch the activity log. The service extracts text locally, serves the needed models on the Sparks **in waves** (so a panel can span more models than fit in - GPU memory at once), runs each reviewer, then the lead reviewer, and saves the - reports. Read them in the Web UI or via **View Latest Report**. + GPU memory at once), runs the structured KPI extractor, then each grader, then + the adjudicator, and finally computes the composite and updates the company's + ledger. Read the results on the dashboard or via **View Latest Scorecard**. + +## How the score works + +The composite is **0–100 = quantitative 60 + qualitative 40 − red flags (max 15)**: + +- **Quant 60** — profitability KPI attainment **30** (the heaviest single slice), + other measurable KPIs **20**, and **forecast integrity 10**: deck N's actuals + are chained against what deck N−1 promised, so sandbagging and quietly moved + goalposts cost points. +- **Qual 40** — eight BDEF categories (A–H), up to 5 points each, scored by the + panel with evidence quotes (thin evidence scales the score down). +- **Red flags** — up to **−15**; silently dropped KPIs are flagged automatically, + and flags raised by only one grader are damped. + +Pinned targets from **Configure Companies** are graded every quarter whether or +not the deck mentions them — a deck cannot improve its score by going quiet. ## Network modes -- **Air-gapped (default):** reviewer containers join an `--internal` Docker +- **Air-gapped (default):** grader containers join an `--internal` Docker network — they can reach only the on-Spark model proxy, with zero internet egress. Models are served from a pre-pulled HF cache. All models must be on the head Spark. Strongest confidentiality. -- **Local services:** reviewers may also reach LAN services (e.g. SearXNG) and the +- **Local services:** graders may also reach LAN services (e.g. SearXNG) and the second Spark. This network has egress unless you firewall it — use only when you - accept that reviewers can reach the network. + accept that graders can reach the network. -The original documents are extracted to plain text on the StartOS box; only that +The original decks are extracted to plain text on the StartOS box; only that text is shipped to the Sparks, and it is wiped from the Sparks after the job (the -reports are kept on your StartOS box). +scorecards and ledgers are kept on your StartOS box). diff --git a/icon.svg b/icon.svg index bb7eb35..9e9e1c1 100644 --- a/icon.svg +++ b/icon.svg @@ -6,13 +6,12 @@ - - - - - - - - - + + + + + + + + diff --git a/instructions.md b/instructions.md index 2250354..aae1c3f 100644 --- a/instructions.md +++ b/instructions.md @@ -1,13 +1,14 @@ # Boardroom Map -Drop confidential documents in and convene a **panel of local LLMs** running on -your DGX Sparks to review them. You choose the models, the personas (lenses), and -how many reviews. An optional **local lead reviewer** synthesizes the panel into -one consolidated report. There is **no frontier model and no cloud key** — in the -default air-gapped mode the documents and their reviews never leave your hardware. +Drop portfolio-company board decks in and have a **panel of local LLMs** running +on your DGX Sparks grade them against the **BDEF v1.1 framework** (Girdley + +Munger/Buffett). A deterministic scorer turns the panel's grades into a 0–100 +composite and appends it to each company's **running scorecard ledger**; the web +dashboard shows the trends. There is **no frontier model and no cloud key** — in +the default air-gapped mode your confidential decks never leave your hardware. -Boardroom Map is a *control plane*: nothing serves or runs on your StartOS box (it only -SSHes to the Sparks and extracts document text on CPU). +Boardroom Map is a *control plane*: nothing serves or runs on your StartOS box +(it only SSHes to the Sparks and extracts deck text on CPU). ## Setup (run the Actions in order) @@ -16,32 +17,54 @@ SSHes to the Sparks and extracts document text on CPU). 2. **Configure Models** — the catalog of local models to serve (alias → HF id → which Spark → port), and serving knobs. For air-gapped runs every model must be on the **head Spark** and present in its HF cache. -3. **Configure Reviewers** — the panel: one entry per review, each a model + a - persona (the lens it reads through) + an optional temperature. -4. **Configure Review** — the rubric, the **Network Mode** (air-gapped vs - local-services), synthesis on/off + lead model, and whether to wipe documents - from the Sparks afterward. +3. **Configure Graders** — the panel: one entry per grader, each a model + a + persona (the lens it grades through — e.g. a Munger inversion skeptic, a + Girdley operator, a skeptical CFO) + an optional temperature. +4. **Configure Grading** — the BDEF rubric override (empty = the built-in + BDEF v1.1), the **Network Mode** (air-gapped vs local-services), the + extractor + adjudicator models, deck retention, and the scoring weights. +5. **Configure Companies** — one entry per portfolio company: its inbox **slug**, + display name, KPI aliases, and **pinned KPI targets**. Pin the profitability + thresholds especially — profitability carries the heaviest weight. -## Running a review +## Grading decks -1. Open the **Web UI** and drag your documents (PDF / DOCX / TXT / MD) onto the - inbox (or drop them in the service's `inbox` folder). -2. Click **Run Review** (or enable *auto-run on drop*). +1. Drop each company's deck into its inbox folder, e.g. + `inbox/acme-widgets/2026-Q2-deck.pdf` (PDF / DOCX / TXT / MD), via the + **Web UI** or the service's `inbox` directory. +2. Run **Grade Decks** (or enable *auto-grade on drop*). 3. Watch the activity log. The service extracts text locally, serves the needed models on the Sparks **in waves** (so a panel can span more models than fit in - GPU memory at once), runs each reviewer, then the lead reviewer, and saves the - reports. Read them in the Web UI or via **View Latest Report**. + GPU memory at once), runs the structured KPI extractor, then each grader, then + the adjudicator, and finally computes the composite and updates the company's + ledger. Read the results on the dashboard or via **View Latest Scorecard**. + +## How the score works + +The composite is **0–100 = quantitative 60 + qualitative 40 − red flags (max 15)**: + +- **Quant 60** — profitability KPI attainment **30** (the heaviest single slice), + other measurable KPIs **20**, and **forecast integrity 10**: deck N's actuals + are chained against what deck N−1 promised, so sandbagging and quietly moved + goalposts cost points. +- **Qual 40** — eight BDEF categories (A–H), up to 5 points each, scored by the + panel with evidence quotes (thin evidence scales the score down). +- **Red flags** — up to **−15**; silently dropped KPIs are flagged automatically, + and flags raised by only one grader are damped. + +Pinned targets from **Configure Companies** are graded every quarter whether or +not the deck mentions them — a deck cannot improve its score by going quiet. ## Network modes -- **Air-gapped (default):** reviewer containers join an `--internal` Docker +- **Air-gapped (default):** grader containers join an `--internal` Docker network — they can reach only the on-Spark model proxy, with zero internet egress. Models are served from a pre-pulled HF cache. All models must be on the head Spark. Strongest confidentiality. -- **Local services:** reviewers may also reach LAN services (e.g. SearXNG) and the +- **Local services:** graders may also reach LAN services (e.g. SearXNG) and the second Spark. This network has egress unless you firewall it — use only when you - accept that reviewers can reach the network. + accept that graders can reach the network. -The original documents are extracted to plain text on the StartOS box; only that +The original decks are extracted to plain text on the StartOS box; only that text is shipped to the Sparks, and it is wiped from the Sparks after the job (the -reports are kept on your StartOS box). +scorecards and ledgers are kept on your StartOS box). diff --git a/orchestrator/adjudicator.py b/orchestrator/adjudicator.py index 2f03f1c..b70da4f 100644 --- a/orchestrator/adjudicator.py +++ b/orchestrator/adjudicator.py @@ -1,88 +1,73 @@ -"""Local lead-reviewer synthesis — no frontier model. +"""Panel adjudication — a local model weighs the graders' evidence. No scores. -After the panel finishes, one more hardened container (the "lead reviewer") reads -all the individual reports (mounted read-only at /reports) plus the documents -(/docs), runs a configured local model, and writes a single consolidated report -to /out/CONSOLIDATED_REPORT.md: shared themes, where reviewers disagree, the -consensus, and an overall recommendation. +After the panel finishes grading one deck (and the outputs have been validated), +one more hardened one-shot container reads the structured extraction +(/extraction.json, ro) plus every panel grade report (/grades, ro) and writes a +MARKDOWN adjudication to /out/ADJUDICATION.md: consensus per BDEF category, +material disagreements and whose evidence is stronger, red flags confirmed or +dismissed, and three questions for next quarter. It never computes numbers — +scoring is deterministic Python (scoring.py). -It reuses the same one-shot reviewer image, switched to BM_ROLE=synthesizer. +It reuses the grader image, switched to BM_ROLE=adjudicator. Mount contract +(remote paths under the per-deck dir): + out/ -> /grades:ro (panel *.json; the agent skips + extraction.json and *.invalid) + out/extraction.json -> /extraction.json:ro + personas/adjudicator.md -> /persona/PERSONA.md:ro + adjudicator-out/ -> /out:rw +No /docs — the adjudicator judges the panel's evidence, not the deck first-hand. """ from __future__ import annotations import shlex -import spark_client as sc +import graders as gr_mod import serving - -DEFAULT_LEAD_PERSONA = ( - "You are the lead reviewer chairing the panel. You did not read the documents " - "first-hand for a fresh opinion — your job is to CONSOLIDATE the panel's " - "individual reports into one authoritative report. Identify the findings the " - "reviewers agree on, surface and adjudicate where they conflict, note anything " - "only one reviewer caught, and end with a prioritized recommendation. Attribute " - "points to the reviewer(s) who raised them. Do not invent findings." -) +import spark_client as sc def pick_model(cfg: dict) -> str: - alias = (cfg.get("synthesisModel") or "").strip() + alias = (cfg.get("adjudicatorModel") or "").strip() if alias: return alias models = cfg.get("models") or [] return models[0]["alias"] if models else "" -def run_synthesis(cfg: dict, jobdir: str, rubric: str, log, wait_timeout: int = 1800) -> dict: - """Launch the lead-reviewer container and wait for the consolidated report.""" +def run_adjudication(cfg: dict, remote_deck_dir: str, log, wait_timeout: int = 1800) -> dict: + """Launch the adjudicator container for one deck and wait for ADJUDICATION.md.""" head = sc.head(cfg) q = shlex.quote model = pick_model(cfg) if not model: - raise RuntimeError("no model available for synthesis (configure a model catalog)") + raise RuntimeError("no model available for adjudication (configure a model catalog)") - persona = (cfg.get("synthesisPersona") or "").strip() or DEFAULT_LEAD_PERSONA + rid = "adjudicator" net = serving.net_name(cfg) - base = serving.reviewer_proxy_base(cfg) - rid = "lead-reviewer" - - sc.run(head, - f"mkdir -p {q(jobdir)}/personas {q(jobdir)}/synth-out && " - f"printf '%s' {q(persona)} > {q(jobdir)}/personas/{rid}.md && " - f"printf '%s' {q(rubric)} > {q(jobdir)}/RUBRIC.md", - timeout=30) - - env = ( - f"-e BM_REVIEWER_ID={q(rid)} -e BM_REVIEWER_NAME=lead-reviewer -e BM_ROLE=synthesizer " - f"-e BM_MODEL={q(model)} -e BM_LLM_BASE={q(base)} -e BM_LLM_KEY=sk-local " - f"-e BM_MAX_MODEL_LEN={q(str(int(cfg.get('maxModelLen', 32768))))} -e HOME=/home/rev " - ) - harden = ( - "--user 1000:1000 --security-opt no-new-privileges --cap-drop ALL " - "--read-only --tmpfs /tmp:size=256m --tmpfs /home/rev:size=128m " - "--pids-limit 256 --memory 6g --cpus 4" - ) + env = gr_mod.base_env(cfg, rid, "adjudicator", "adjudicator", model, None) mounts = ( - f"-v {q(jobdir)}/docs:/docs:ro " - f"-v {q(jobdir)}/out:/reports:ro " - f"-v {q(jobdir)}/synth-out:/out " - f"-v {q(jobdir)}/personas/{rid}.md:/persona/PERSONA.md:ro " - f"-v {q(jobdir)}/RUBRIC.md:/RUBRIC.md:ro " + f"-v {q(remote_deck_dir)}/out:/grades:ro " + f"-v {q(remote_deck_dir)}/out/extraction.json:/extraction.json:ro " + f"-v {q(remote_deck_dir)}/personas/{rid}.md:/persona/PERSONA.md:ro " + f"-v {q(remote_deck_dir)}/adjudicator-out:/out " ) - cname = f"bm-grader-{rid}" + cname = f"bm-grader-{rid}-{gr_mod.container_suffix(remote_deck_dir)}" cmd = ( f"docker rm -f {cname} >/dev/null 2>&1; " - f"docker run -d --name {cname} --network {q(net)} {harden} {env} {mounts} {q(cfg['graderImage'])}" + f"docker run -d --name {cname} --network {q(net)} {gr_mod.HARDEN} {env} {mounts} " + f"{q(cfg['graderImage'])}" ) - log(f"[synthesis] lead reviewer up -> {model}") + log(f"[adjudicator] up -> {model}") r = sc.run(head, cmd, timeout=120) if r.returncode != 0: - raise RuntimeError(f"synthesis launch failed: {r.stderr or r.stdout}") + raise RuntimeError(f"adjudicator launch failed: {r.stderr or r.stdout}") w = sc.run(head, f"docker wait {cname}", timeout=wait_timeout) code = (w.stdout or "").strip() - chk = sc.run(head, f"test -s {q(jobdir)}/synth-out/CONSOLIDATED_REPORT.md && echo OK || echo MISSING", timeout=30) + chk = sc.run(head, f"test -s {q(remote_deck_dir)}/adjudicator-out/ADJUDICATION.md " + "&& echo OK || echo MISSING", timeout=30) wrote = "OK" in (chk.stdout or "") sc.run(head, f"docker rm -f {cname} 2>/dev/null; true", timeout=30) - log(f"[synthesis] lead reviewer exited (code={code or '?'}), report={'written' if wrote else 'MISSING'}") + log(f"[adjudicator] exited (code={code or '?'}), " + f"adjudication={'written' if wrote else 'MISSING'}") return {"model": model, "exit": code, "report": wrote} diff --git a/orchestrator/app.py b/orchestrator/app.py index 65f26a4..d010f32 100644 --- a/orchestrator/app.py +++ b/orchestrator/app.py @@ -1,28 +1,34 @@ """Boardroom Map orchestrator web app. -Serves the control-panel UI and a small JSON API, and starts the background job -runner (see jobs.py) that actually convenes the panel. Most configuration happens -through the StartOS *actions* (Configure Sparks / Models / Reviewers / Review); -this UI is for dropping documents, triggering a review, watching it run, and -reading reports. +Serves the portfolio dashboard and a small JSON API, and starts the background +job runner (see jobs.py) that grades the dropped board decks. Most configuration +happens through the StartOS *actions* (Configure Sparks / Models / Graders / +Grading); this UI is for dropping decks per company, triggering a grading run, +watching it, and reading the per-company scorecard ledger. """ from __future__ import annotations import os import threading -from fastapi import FastAPI, HTTPException, UploadFile, File -from fastapi.responses import HTMLResponse, PlainTextResponse +from fastapi import FastAPI, File, Form, HTTPException, UploadFile +from fastapi.responses import HTMLResponse, JSONResponse, PlainTextResponse from fastapi.templating import Jinja2Templates from starlette.requests import Request import bm_config -import extraction -import reviewers as rev_mod +import decks +import graders as grader_mod +import jobs +import ledger as ledger_mod import serving -from jobs import runner, INBOX, REPORTS_DIR DATA_DIR = os.environ.get("BM_DATA_DIR", "/data") +LEDGER_DIR = os.path.join(DATA_DIR, "ledger") + +runner = jobs.runner +INBOX = getattr(jobs, "INBOX", os.path.join(DATA_DIR, "inbox")) +REPORTS_DIR = getattr(jobs, "REPORTS_DIR", os.path.join(DATA_DIR, "reports")) app = FastAPI(title="Boardroom Map Orchestrator") templates = Jinja2Templates(directory=os.path.join(os.path.dirname(__file__), "templates")) @@ -53,19 +59,19 @@ def status(): "configured": { "sparks": bool(cfg.get("primarySparkHost")), "models": len(cfg.get("models") or []), - "reviewers": len(cfg.get("reviewers") or []), + "graders": len(cfg.get("graders") or []), }, "networkMode": cfg.get("networkMode"), - "synthesis": bool(cfg.get("synthesisEnabled")), + "adjudicator": bool(cfg.get("adjudicatorEnabled")), "wipeRemoteDocs": bool(cfg.get("wipeRemoteDocs")), "autoRunOnDrop": bool(cfg.get("autoRunOnDrop")), "models": [{"alias": m["alias"], "hfModel": m["hfModel"], "spark": m.get("spark", "primary")} for m in (cfg.get("models") or [])], - "panel": [{"name": r.get("name"), "model": r.get("model"), - "persona": bool((r.get("persona") or "").strip()), - "known": (r.get("model") in catalog)} - for r in (cfg.get("reviewers") or [])], - "inbox": _inbox_list(), + "panel": [{"name": g.get("name"), "model": g.get("model"), + "persona": bool((g.get("persona") or "").strip()), + "known": (g.get("model") in catalog)} + for g in (cfg.get("graders") or [])], + "inbox": _inbox_grouped(), "runtime": runner.snapshot(), } @@ -75,46 +81,89 @@ def events(): return {"events": runner.events()} -def _inbox_list() -> list[dict]: - if not os.path.isdir(INBOX): - return [] - out = [] - for fn in sorted(os.listdir(INBOX)): - p = os.path.join(INBOX, fn) - if os.path.isfile(p): - ext = os.path.splitext(fn)[1].lower() - out.append({"name": fn, "bytes": os.path.getsize(p), - "supported": ext in extraction.SUPPORTED}) - return out +# ----------------------------------------------------------------------- inbox +def _inbox_grouped() -> dict: + """Company-grouped inbox view: {companies: {slug: [file dicts]}, skipped: [...]}.""" + try: + d = decks.discover(INBOX) + except Exception: + return {"companies": {}, "skipped": []} + companies: dict[str, list] = {} + for u in d.get("units") or []: + lst = companies.setdefault(u["company_slug"], []) + for f in u.get("files") or []: + try: + size = os.path.getsize(f) + except OSError: + size = 0 + lst.append({"name": os.path.basename(f), "bytes": size, + "period": u.get("period"), "supported": True}) + for fn in u.get("ignored") or []: + lst.append({"name": fn, "bytes": 0, "period": None, "supported": False}) + # discover() drops units with no supported files, so sweep the company dirs + # for anything it didn't list (unsupported strays) and flag them. + try: + for entry in sorted(os.listdir(INBOX)): + cdir = os.path.join(INBOX, entry) + if entry.startswith(".") or not os.path.isdir(cdir): + continue + slug = decks.slugify(entry) + seen = {f["name"] for f in companies.get(slug, [])} + for fn in sorted(os.listdir(cdir)): + if fn.startswith(".") or fn in seen or not os.path.isfile(os.path.join(cdir, fn)): + continue + try: + size = os.path.getsize(os.path.join(cdir, fn)) + except OSError: + size = 0 + companies.setdefault(slug, []).append( + {"name": fn, "bytes": size, "period": decks.parse_period_from_name(fn), + "supported": os.path.splitext(fn)[1].lower() in decks.SUPPORTED_EXTS}) + except OSError: + pass + return {"companies": companies, "skipped": d.get("skipped") or []} @app.get("/api/inbox") def inbox(): - return {"inbox": _inbox_list()} + return _inbox_grouped() # ----------------------------------------------------------------------- documents @app.post("/api/upload") -async def upload(files: list[UploadFile] = File(...)): - os.makedirs(INBOX, exist_ok=True) +async def upload(request: Request, + files: list[UploadFile] = File(...), + company: str | None = Form(None)): + name = (company or request.query_params.get("company") or "").strip() + if not name: + raise HTTPException(400, "company is required — root-level files are not graded") + slug = decks.slugify(name) + dest_dir = os.path.join(INBOX, slug) + os.makedirs(dest_dir, exist_ok=True) saved = [] for f in files: - name = os.path.basename(f.filename or "document") - dest = os.path.join(INBOX, name) + fn = os.path.basename(f.filename or "deck") + dest = os.path.join(dest_dir, fn) with open(dest, "wb") as out: while chunk := await f.read(1 << 20): out.write(chunk) - saved.append(name) - return {"ok": True, "saved": saved} + saved.append(fn) + return {"ok": True, "company": slug, "saved": saved} @app.post("/api/inbox/clear") def inbox_clear(): + import shutil if os.path.isdir(INBOX): for fn in os.listdir(INBOX): p = os.path.join(INBOX, fn) - if os.path.isfile(p): - os.remove(p) + try: + if os.path.isfile(p): + os.remove(p) + elif os.path.isdir(p): + shutil.rmtree(p) + except OSError: + pass return {"ok": True} @@ -124,12 +173,16 @@ def run_now(): cfg = bm_config.load() if not cfg.get("primarySparkHost"): raise HTTPException(400, "No Spark configured (Configure Sparks).") - if not (cfg.get("models") and cfg.get("reviewers")): - raise HTTPException(400, "Configure at least one model and one reviewer first.") - if not _inbox_list(): - raise HTTPException(400, "Inbox is empty — upload documents first.") + if not (cfg.get("models") and cfg.get("graders")): + raise HTTPException(400, "Configure at least one model and one grader first.") + try: + units = decks.discover(INBOX).get("units") or [] + except Exception: + units = [] + if not units: + raise HTTPException(400, "Inbox is empty — upload decks into a company folder first.") runner.request_run() - return {"ok": True, "message": "Review requested — watch the activity log."} + return {"ok": True, "message": "Grading requested — watch the activity log."} @app.get("/api/serving") @@ -140,20 +193,21 @@ def serving_status(): return {"serving": serving.health(cfg)} -@app.post("/api/reviewer/build-image") -def build_reviewer_image(): +@app.post("/api/grader/build-image") +def build_grader_image(): cfg = bm_config.load() if not cfg.get("primarySparkHost"): raise HTTPException(400, "No Spark configured.") threading.Thread(target=lambda: _safe_build(cfg), daemon=True).start() - return {"ok": True, "message": "Building reviewer image on the head Spark — watch the activity log."} + return {"ok": True, "message": "Building grader image on the head Spark — watch the activity log."} def _safe_build(cfg: dict): try: - rev_mod.ensure_reviewer_image(cfg, runner.log) + fn = getattr(grader_mod, "ensure_grader_image", None) or grader_mod.ensure_reviewer_image + fn(cfg, runner.log) except Exception as e: - runner.log(f"[reviewers] image build failed: {e}") + runner.log(f"[graders] image build failed: {e}") @app.post("/api/stop") @@ -167,21 +221,182 @@ def stop(): return {"ok": True} -# ----------------------------------------------------------------------- reports +# ----------------------------------------------------------------------- companies / ledger +def _config_companies(cfg: dict) -> list[dict]: + """Companies registered in the StartOS config (may have zero graded decks).""" + out = [] + for c in (cfg.get("companies") or []): + if isinstance(c, dict): + name = (c.get("name") or c.get("company") or c.get("slug") or "").strip() + else: + name = str(c).strip() + if name: + out.append({"slug": decks.slugify(name), "name": name}) + return out + + +def _ledger_companies() -> list[dict]: + try: + led = ledger_mod.Ledger(LEDGER_DIR) + return led.all_companies() or [] + except Exception: + return [] + + +@app.get("/api/companies") +def companies(): + cfg = bm_config.load() + rows, seen = [], set() + for c in _ledger_companies(): + try: + hist = [{"period": h.get("period"), "composite": h.get("composite")} + for h in (c.get("history") or [])] + latest = None + if hist: + delta = None + cur, prev = hist[-1]["composite"], (hist[-2]["composite"] if len(hist) >= 2 else None) + if isinstance(cur, (int, float)) and isinstance(prev, (int, float)): + delta = round(cur - prev, 2) + latest = {"period": hist[-1]["period"], "composite": cur, "delta": delta} + slug = c.get("slug") or decks.slugify(c.get("name") or "") + seen.add(slug) + rows.append({"slug": slug, "name": c.get("name") or slug, + "auto_created": bool(c.get("auto_created")), + "deck_count": len(hist), "latest": latest, "history": hist}) + except Exception: + continue + for c in _config_companies(cfg): + if c["slug"] not in seen: + seen.add(c["slug"]) + rows.append({"slug": c["slug"], "name": c["name"], "auto_created": False, + "deck_count": 0, "latest": None, "history": []}) + rows.sort(key=lambda r: (r["name"] or "").lower()) + return {"companies": rows} + + +def _kpi_hit_rate(records: list[dict]) -> dict: + """Per canonical KPI across records (oldest→newest): + attempts, hits (credit>=0.999), streak of hits ending at the latest attempt, + last_credit and profitability tag.""" + stats: dict[str, dict] = {} + for rec in records: + for k in (rec.get("kpi_results") or []): + cn = k.get("canonical_name") or k.get("name") + if not cn: + continue + s = stats.setdefault(cn, {"name": k.get("name") or cn, "attempts": 0, "hits": 0, + "streak": 0, "last_credit": None, "profitability": False}) + if k.get("name"): + s["name"] = k["name"] + s["profitability"] = bool(k.get("profitability", s["profitability"])) + credit = k.get("credit") + if not isinstance(credit, (int, float)): + continue # no target matched — not an attempt + s["attempts"] += 1 + s["last_credit"] = credit + if credit >= 0.999: + s["hits"] += 1 + s["streak"] += 1 + else: + s["streak"] = 0 + return stats + + +def _categories_latest(records: list[dict]) -> dict: + if not records: + return {} + def cats(rec): + return ((rec.get("qual") or {}).get("categories")) or {} + latest = cats(records[-1]) + prev = cats(records[-2]) if len(records) >= 2 else {} + out = {} + for cid in "ABCDEFGH": + cur = latest.get(cid) + if cur is None and prev.get(cid) is None: + continue + out[cid] = {"latest_adjusted": (cur or {}).get("adjusted"), + "previous_adjusted": (prev.get(cid) or {}).get("adjusted")} + return out + + +@app.get("/api/companies/{slug}") +def company_detail(slug: str): + slug = os.path.basename(slug) + company, records = None, [] + try: + led = ledger_mod.Ledger(LEDGER_DIR) + company = led.get_company(slug) + if company is not None: + records = led.deck_records(slug) or [] + except Exception: + company, records = None, [] + if company is None: + cfg = bm_config.load() + match = next((c for c in _config_companies(cfg) if c["slug"] == slug), None) + if not match: + raise HTTPException(404, "no such company") + company = {"slug": slug, "name": match["name"], "auto_created": False, + "kpi_aliases": {}, "pinned_targets": [], "extracted_targets": {}, + "history": []} + open_flags = (((records[-1].get("penalties") or {}).get("flags")) or []) if records else [] + return { + "company": company, + "records": records, + "kpi_hit_rate": _kpi_hit_rate(records), + "categories_latest": _categories_latest(records), + "open_flags": open_flags, + } + + +@app.get("/api/companies/{slug}/scorecard", response_class=PlainTextResponse) +def company_scorecard(slug: str): + slug = os.path.basename(slug) + path = os.path.join(LEDGER_DIR, slug, "SCORECARD.md") + if not os.path.exists(path): + raise HTTPException(404, "no scorecard yet for this company") + return open(path, errors="replace").read() + + +@app.get("/api/companies/{slug}/decks/{deck_id}") +def deck_record(slug: str, deck_id: str): + slug, deck_id = os.path.basename(slug), os.path.basename(deck_id) + rec = None + try: + led = ledger_mod.Ledger(LEDGER_DIR) + rec = led.deck_record(slug, deck_id) + except Exception: + rec = None + if rec is None: + raise HTTPException(404, "no such deck record") + return JSONResponse(rec) + + +@app.get("/api/companies/{slug}/decks/{deck_id}/report", response_class=PlainTextResponse) +def deck_report(slug: str, deck_id: str): + slug, deck_id = os.path.basename(slug), os.path.basename(deck_id) + if deck_id.endswith(".md"): + deck_id = deck_id[:-3] + path = os.path.join(LEDGER_DIR, slug, "decks", f"{deck_id}.md") + if not os.path.exists(path): + raise HTTPException(404, "no report for this deck") + return open(path, errors="replace").read() + + +# ----------------------------------------------------------------------- reports (legacy job reports) @app.get("/api/reports") def list_reports(): if not os.path.isdir(REPORTS_DIR): return {"reports": []} - jobs = sorted((d for d in os.listdir(REPORTS_DIR) - if os.path.isdir(os.path.join(REPORTS_DIR, d))), reverse=True) - return {"reports": jobs} + jobs_ = sorted((d for d in os.listdir(REPORTS_DIR) + if os.path.isdir(os.path.join(REPORTS_DIR, d))), reverse=True) + return {"reports": jobs_} @app.get("/api/report", response_class=PlainTextResponse) def latest_report(): path = os.path.join(REPORTS_DIR, "latest.md") if not os.path.exists(path): - return "(no report yet — drop documents in the inbox and run a review)" + return "(no report yet — drop decks in a company folder and run a grading job)" return open(path, errors="replace").read().strip() or "(empty report)" diff --git a/orchestrator/bdef.md b/orchestrator/bdef.md new file mode 100644 index 0000000..d517d7d --- /dev/null +++ b/orchestrator/bdef.md @@ -0,0 +1,103 @@ +# Board Deck Evaluation Framework (BDEF v1.1) +Inch Wide, Mile Deep — Girdley traits integrated with Munger & Buffett principles. + +You are grading a portfolio-company board deck. The deck should let an owner's +representative answer, with high confidence: Are incentives aligned with long-term +owners? Has management inverted the problem and built in margin of safety? Are they +inside (and rationally expanding) their circle of competence? Is capital allocated +with owner-like patience, or is activity masquerading as progress? Would this +company survive a Lollapalooza of bad incentives, biases, and external shocks? + +Score each category 1–5. A score above or below 3 REQUIRES verbatim evidence +quotes from the deck. Judge what the deck actually shows — absence of evidence on +a category is itself information (score 2–3 with the absence noted, not a guess). + +## A. Incentive Alignment & Skin in the Game +Probes: Does compensation/promotion demonstrably reward rational long-term capital +allocation and owner-like behavior? Visible misalignments (short-term bonus +weighting, option reloads, metrics that invite channel stuffing or earnings +management)? Does management have skin in the game that survives a multi-year +downturn? Munger test: if I changed the incentives, would behavior change predictably? +- 1: Incentives invisible or visibly perverse. 3: Headcount/engagement shown but no + comp structure or ownership data. 5: Comp, promotion criteria, and ownership shown + and clearly aligned with long-term owners. + +## B. Inversion Discipline & Margin of Safety +Probes: Are plausible failure modes explicitly modeled for major initiatives and +forecasts? Visible conservatism in assumptions, capital buffers, competitive-response +planning? Does the deck show what the company would NOT do even if it looked attractive? +- 1: Only upside shown; hockey-stick forecasts with no falsifiers. 3: Generic risk + slide, no quantified margin of safety. 5: Explicit inversion — what breaks the + thesis, how much buffer exists, and pre-committed "we won't do X" boundaries. + +## C. Circle of Competence & Rational Learning +Probes: Does management accurately describe the boundaries of what they know well? +Disciplined expansion of the circle rather than overreach into new areas? Is learning +from mistakes visible and systematic? +- 1: Confident claims in adjacencies with no demonstrated competence. 3: Competent in + core but boundaries unstated. 5: Explicit "we know / we don't know", postmortems, + and disciplined expansion criteria. + +## D. Capital Allocation Quality +Probes: Is every significant capital decision framed as opportunity cost vs long-term +owner return (including returning capital)? Patience ("sit on your ass") vs activity +bias? Are buybacks, dividends, M&A, and reinvestment held to the same owner rigor? +- 1: Growth for its own sake; projects listed without expected returns. 3: Budgets + shown but no alternatives comparison. 5: Every major incremental dollar shown with + expected return vs alternatives, including the do-nothing/return-it option. + +## E. Moat Durability & Competitive Reality +Probes: Is the moat described in specific, testable terms (cost, switching costs, +network effects, brand) rather than generic "great team" language? What is management +actively doing to widen/defend it, and which threats are acknowledged? Buffett test: +would an intelligent owner buy this business at a fair price today based on the +durability shown? +- 1: "Great team / huge TAM" hand-waving. 3: Moat named but not evidenced or + threatened realistically. 5: Specific, testable moat with widening actions and + honestly acknowledged threats. + +## F. Psychological & Cultural Health +Probes: Does the deck's framing reward early surfacing of problems, or filter +information upward? Evidence of Lollapalooza effects (multiple biases/misaligned +incentives compounding)? Does "no drama" reflect genuine psychological safety or +suppressed dissent? Do problem employees move on quickly; do values drive hiring/firing? +- 1: Only good news; problems appear late and pre-spun. 3: Engagement scores without + bad-news examples. 5: Bad news travels fast and visibly; the deck itself surfaces + problems early with owners' candor. + +## G. Simplicity, Clarity & Decision Velocity +Probes: Does the deck avoid unnecessary complexity ("simple stays simple")? Are +repeatable processes and decision frameworks visible, or is the company reliant on +heroic individual effort? Is the board asked to judge the few things that matter +enormously rather than many that matter little? +- 1: Impressively complex deck obscuring weak economics. 3: Clear but unfocused. + 5: A model of clarity an intelligent owner could absorb in one sitting, focused on + the 2–3 decisions that matter. + +## H. Board Value-Add & Governance Quality +Probes: Does the deck position the board to pull (high-leverage questions on +incentives, inversion, capital allocation, moat) rather than rubber-stamp? Evidence +the board functions as owners' representatives rather than management's advisors? +Clear asks with recommendations and the inversion of those decisions? +- 1: No asks, or trivia; board presides rather than governs. 3: Asks listed without + recommendation or inversion. 5: The few decisions that matter, each with a clear + recommendation and what would make it wrong. + +## Red-flag taxonomy +Use these codes (severity 1–5; suggest severity per guidance): +- `adjusted_metrics` (2–4): heavy reliance on adjusted/non-GAAP numbers without bridges. +- `metric_redefinition` (3–5): a KPI's definition changed between periods. +- `kpi_dropped` (2–3): a previously reported KPI silently disappeared. +- `hockey_stick_forecast` (2–4): forecast with no inversion or margin of safety. +- `channel_stuffing_risk` (3–5): incentives/metrics that invite pull-forward behavior. +- `short_term_comp` (2–4): compensation heavily weighted to short-term outcomes. +- `related_party` (3–5): related-party transactions or conflicts. +- `governance_gap` (2–4): big questions (succession, major bets, incentive redesign) get superficial treatment while minutiae fill the deck. +- `cash_runway_silence` (3–5): cash/runway/burn not clearly disclosed. +- `no_profitability_visibility` (3): no profit/margin/cash KPI reported at all. +- `overreach_adjacency` (2–4): confident expansion outside demonstrated competence. +- `activity_bias` (2–3): busy project lists without linkage to moat or owner returns. +- `complexity_smokescreen` (2–4): complexity that appears designed to obscure economics. +- `suppressed_dissent` (3–5): signs bad news is filtered before reaching the board. + +Do NOT compute totals or a composite score. Numbers are computed elsewhere. diff --git a/orchestrator/bm_config.py b/orchestrator/bm_config.py index 2091b7c..abc4c3c 100644 --- a/orchestrator/bm_config.py +++ b/orchestrator/bm_config.py @@ -12,13 +12,23 @@ import spark_client as sc DATA_DIR = os.environ.get("BM_DATA_DIR", "/data") HF_TOKEN_PATH = os.path.join(DATA_DIR, "secrets", "hf_token") +BDEF_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "bdef.md") -DEFAULT_RUBRIC = ( - "Review the attached document(s). Produce a structured report: a 3-5 sentence " - "summary, the key findings and insights, risks or red flags, open questions, " - "and concrete recommendations. Cite the document and section for each point. " - "Be honest about uncertainty; never invent facts not present in the documents." -) +# Scoring weights (composite 0-100 = quant 60 + qual 40 - penalties). +# Every knob the deterministic scorer uses lives here so the user can retune +# without a rebuild. Keep flat: StartOS action inputs are flat number fields. +WEIGHTS_DEFAULTS = { + "profitabilityKpi": 30, # profitability KPI attainment bucket + "otherKpi": 20, # non-profitability measurable KPI bucket + "forecastIntegrity": 10, # deck N actuals vs deck N-1 stated targets + "qualCategoryMax": 5, # each BDEF category A-H maxes at this (8x5=40) + "redFlagCap": 15, # max total penalty + "kpiCreditFloor": 0.5, # actual/target ratio below which credit = 0 + "droppedKpiPenalty": 2, # severity of a KPI that silently disappeared + "droppedKpiMax": 3, # count at most this many dropped-KPI flags + "evidenceFullCredit": 400, # quote chars for full qualitative weight + "singleSourceFlagFactor": 0.5, # damping for flags raised by one source only +} CONFIG_DEFAULTS = { # Spark connection @@ -39,22 +49,29 @@ CONFIG_DEFAULTS = { "proxyPort": 4000, "maxConcurrentModels": 1, "models": [ - {"alias": "reviewer-a", "hfModel": "Qwen/Qwen3-32B-FP8", "spark": "primary", "port": 8001}, + {"alias": "grader-a", "hfModel": "Qwen/Qwen3-32B-FP8", "spark": "primary", "port": 8001}, ], - # Review panel - "reviewers": [ - {"name": "reviewer-1", "model": "reviewer-a", "persona": "", "temperature": None}, + # Grading panel + "graders": [ + {"name": "munger-lens", "model": "grader-a", "persona": "", "temperature": None}, ], - # Review job settings - "reviewInstructions": DEFAULT_RUBRIC, + # Which catalog model runs the stage-1 structured extractor ("" = first model) + "extractorModel": "", + # Grading job settings + "bdefOverride": "", # non-empty replaces the baked-in bdef.md rubric + "weights": dict(WEIGHTS_DEFAULTS), "networkMode": "airgapped", "searxngUrl": "", - "synthesisEnabled": True, - "synthesisModel": "", - "synthesisPersona": "", + "adjudicatorEnabled": True, + "adjudicatorModel": "", + "adjudicatorPersona": "", "wipeRemoteDocs": True, "autoRunOnDrop": False, "networkName": "boardroom-net", + # Portfolio companies (authoritative source of pinned targets / aliases). + # pinnedTargets: [{kpi, target, unit, direction: gte|lte, profitability}] + # kpiAliases: newline-separated "canonical=alias1;alias2" lines. + "companies": [], # Flags "hfTokenSet": False, } @@ -68,9 +85,22 @@ def load() -> dict: except FileNotFoundError: return merged merged.update({k: v for k, v in saved.items() if v is not None}) + # weights merge key-by-key so a partially-saved weights object keeps defaults + w = dict(WEIGHTS_DEFAULTS) + w.update({k: v for k, v in (merged.get("weights") or {}).items() if v is not None}) + merged["weights"] = w return merged +def bdef_text(cfg: dict) -> str: + """The grading rubric: config override if set, else the baked-in spec.""" + override = (cfg.get("bdefOverride") or "").strip() + if override: + return override + with open(BDEF_PATH, encoding="utf-8") as f: + return f.read() + + def hf_token() -> str | None: if os.path.exists(HF_TOKEN_PATH): t = open(HF_TOKEN_PATH).read().strip() diff --git a/orchestrator/decks.py b/orchestrator/decks.py new file mode 100644 index 0000000..c783367 --- /dev/null +++ b/orchestrator/decks.py @@ -0,0 +1,122 @@ +"""Deck discovery — map the inbox onto (company, period) grading units. + +The inbox is organized by company: /data/inbox//. The +subfolder name is the company slug; the reporting period is parsed from each +filename (2026-Q2, Q2 2026, 2026-H1, 2026-05, FY2026 ...). Files whose period +cannot be parsed form a period-less unit that the extractor's own deck.period +can later fill in. Files at the inbox root are skipped (we would not know the +company) and reported so the UI can nag the operator. +""" +from __future__ import annotations + +import os +import re + +SUPPORTED_EXTS = {".pdf", ".pptx", ".docx", ".txt", ".md", ".text"} + +# Canonical period forms: "2026-Q2", "2026-H1", "2026-05", "FY2026". +_PERIOD_PATTERNS = [ + # 2026-Q2 / 2026Q2 / 2026_Q2 / 2026 Q2 + (re.compile(r"(? str: + """Lowercase [a-z0-9-] slug for a company folder name.""" + s = re.sub(r"[^a-z0-9]+", "-", (name or "").strip().lower()).strip("-") + return s or "company" + + +def parse_period_from_name(filename: str) -> str | None: + """Canonical period parsed from a filename, or None.""" + base = os.path.basename(filename) + for pat, canon in _PERIOD_PATTERNS: + m = pat.search(base) + if m: + return canon(m) + return None + + +def period_sort_key(period: str | None) -> tuple: + """(year, start_month, granularity_rank); unknown/None sorts last.""" + if not period: + return _UNKNOWN_KEY + m = re.fullmatch(r"((?:19|20)\d{2})-Q([1-4])", period) + if m: + return (int(m.group(1)), (int(m.group(2)) - 1) * 3 + 1, _GRAN_Q) + m = re.fullmatch(r"((?:19|20)\d{2})-H([12])", period) + if m: + return (int(m.group(1)), (int(m.group(2)) - 1) * 6 + 1, _GRAN_H) + m = re.fullmatch(r"((?:19|20)\d{2})-(0[1-9]|1[0-2])", period) + if m: + return (int(m.group(1)), int(m.group(2)), _GRAN_M) + m = re.fullmatch(r"FY((?:19|20)\d{2})", period) + if m: + return (int(m.group(1)), 1, _GRAN_FY) + return _UNKNOWN_KEY + + +def discover(inbox_dir: str) -> dict: + """Scan the inbox into grading units. + + Returns {"units": [{company_slug, period, period_source, files, ignored}], + "skipped": [root-level file names]}. Units are grouped by (company, parsed + period), sorted by company then oldest period first (period-less last).""" + units: dict[tuple, dict] = {} + skipped: list[str] = [] + if not os.path.isdir(inbox_dir): + return {"units": [], "skipped": []} + for entry in sorted(os.listdir(inbox_dir)): + if entry.startswith("."): + continue + path = os.path.join(inbox_dir, entry) + if os.path.isfile(path): + skipped.append(entry) + continue + if not os.path.isdir(path): + continue + company = slugify(entry) + for fn in sorted(os.listdir(path)): + if fn.startswith("."): + continue + fpath = os.path.join(path, fn) + if not os.path.isfile(fpath): + continue + period = parse_period_from_name(fn) + key = (company, period_sort_key(period), period) + unit = units.setdefault(key, { + "company_slug": company, + "period": period, + "period_source": "filename" if period else "unknown", + "files": [], + "ignored": [], + }) + ext = os.path.splitext(fn)[1].lower() + if ext in SUPPORTED_EXTS: + unit["files"].append(os.path.abspath(fpath)) + else: + unit["ignored"].append(fn) + out = [u for _, u in sorted(units.items(), key=lambda kv: (kv[0][0], kv[0][1])) + if u["files"]] + for u in out: + u["files"].sort() + u["ignored"].sort() + return {"units": out, "skipped": skipped} diff --git a/orchestrator/extraction.py b/orchestrator/extraction.py index 40b8931..8b34311 100644 --- a/orchestrator/extraction.py +++ b/orchestrator/extraction.py @@ -5,6 +5,7 @@ to the Sparks, we extract plain text here so that only normalized text (never th original binaries) crosses to the review containers. Supported formats: .pdf -> pypdf + .pptx -> python-pptx (text frames, tables, chart data, notes) .docx -> python-docx .txt .md .text -> read as UTF-8 @@ -17,7 +18,7 @@ from __future__ import annotations import os TEXT_EXTS = {".txt", ".md", ".text", ".markdown"} -SUPPORTED = TEXT_EXTS | {".pdf", ".docx"} +SUPPORTED = TEXT_EXTS | {".pdf", ".docx", ".pptx"} def _extract_pdf(path: str) -> str: @@ -48,6 +49,73 @@ def _extract_docx(path: str) -> str: return "\n".join(lines).strip() +def _pptx_chart_text(shape) -> list[str]: + """Best-effort chart title + series names/values (chart XML varies wildly).""" + lines: list[str] = [] + try: + chart = shape.chart + try: + if chart.has_title and chart.chart_title.has_text_frame: + lines.append(f"[chart] {chart.chart_title.text_frame.text}") + except Exception: + pass + for plot in chart.plots: + try: + cats = [str(c) for c in (plot.categories or [])] + except Exception: + cats = [] + for series in plot.series: + try: + name = str(series.name) + except Exception: + name = "(series)" + try: + vals = ["" if v is None else f"{v:g}" for v in series.values] + except Exception: + vals = [] + if cats and len(cats) == len(vals): + pairs = ", ".join(f"{c}={v}" for c, v in zip(cats, vals)) + else: + pairs = ", ".join(vals) + lines.append(f"[chart series] {name}: {pairs}") + except Exception: + pass + return lines + + +def _extract_pptx(path: str) -> str: + from pptx import Presentation + + prs = Presentation(path) + parts: list[str] = [] + for n, slide in enumerate(prs.slides, 1): + body: list[str] = [] + for shape in slide.shapes: + if getattr(shape, "has_text_frame", False): + txt = shape.text_frame.text.strip() + if txt: + body.append(txt) + if getattr(shape, "has_table", False): + for row in shape.table.rows: + cells = [c.text.strip() for c in row.cells] + if any(cells): + body.append(" | ".join(cells)) + if getattr(shape, "has_chart", False): + body.extend(_pptx_chart_text(shape)) + notes = "" + try: + if slide.has_notes_slide: + notes = (slide.notes_slide.notes_text_frame.text or "").strip() + except Exception: + pass + if notes: + body.append(f"--- notes ---\n{notes}") + if len("".join(body)) < 20: + body.append(f"[low_text_slide: slide {n}]") + parts.append(f"\n\n===== slide {n} =====\n" + "\n".join(body)) + return "".join(parts).strip() + + def _extract_text(path: str) -> str: with open(path, errors="replace") as f: return f.read().strip() @@ -57,6 +125,8 @@ def extract_file(path: str) -> str: ext = os.path.splitext(path)[1].lower() if ext == ".pdf": return _extract_pdf(path) + if ext == ".pptx": + return _extract_pptx(path) if ext == ".docx": return _extract_docx(path) if ext in TEXT_EXTS: @@ -70,20 +140,16 @@ def _safe_name(name: str) -> str: return (keep or "document").replace(" ", "_") -def extract_inbox(inbox: str, out_dir: str, log=print) -> list[dict]: - """Extract every supported file in `inbox` to a .txt in `out_dir`. +def extract_files(files: list[str], out_dir: str, log=print) -> list[dict]: + """Extract an explicit list of files (a deck unit) to .txt files in `out_dir`. Returns a manifest: [{source, out, chars, ok, error}]. Skips unsupported files (recorded with ok=False) rather than failing the whole job.""" os.makedirs(out_dir, exist_ok=True) manifest: list[dict] = [] - if not os.path.isdir(inbox): - return manifest seen: dict[str, int] = {} - for fn in sorted(os.listdir(inbox)): - src = os.path.join(inbox, fn) - if not os.path.isfile(src): - continue + for src in files: + fn = os.path.basename(src) ext = os.path.splitext(fn)[1].lower() rec = {"source": fn, "out": None, "chars": 0, "ok": False, "error": ""} if ext not in SUPPORTED: @@ -113,3 +179,16 @@ def extract_inbox(inbox: str, out_dir: str, log=print) -> list[dict]: log(f"[extract] {fn} -> {out_name} ({len(text)} chars)") manifest.append(rec) return manifest + + +def extract_inbox(inbox: str, out_dir: str, log=print) -> list[dict]: + """Extract every supported file in `inbox` to a .txt in `out_dir`. + + Returns a manifest: [{source, out, chars, ok, error}]. Skips unsupported + files (recorded with ok=False) rather than failing the whole job.""" + if not os.path.isdir(inbox): + os.makedirs(out_dir, exist_ok=True) + return [] + files = [os.path.join(inbox, fn) for fn in sorted(os.listdir(inbox)) + if os.path.isfile(os.path.join(inbox, fn))] + return extract_files(files, out_dir, log) diff --git a/orchestrator/graders.py b/orchestrator/graders.py index f717fa5..e66372d 100644 --- a/orchestrator/graders.py +++ b/orchestrator/graders.py @@ -1,67 +1,87 @@ -"""Launch the reviewer panel on the head Spark over SSH. +"""Launch the grading panel (and the stage-1 extractor) on the head Spark. -Each reviewer is a ONE-SHOT, hardened, read-only container: it reads the document -text mounted at /docs, runs its model (through the on-Spark proxy) under its -persona + the shared rubric, writes a single report to /out/.md, and exits. -There is no shared writable workspace and no git — reviewers cannot alter the -documents or each other's reports. +Each role agent is a ONE-SHOT, hardened, read-only container running +sandbox/grader_agent.py: it reads the deck text mounted at /docs, runs its model +(through the on-Spark proxy) under its persona + the BDEF rubric, writes exactly +one output file to /out, and exits. There is no tool loop and no shared writable +workspace — agents cannot alter the deck text or each other's reports. + +Per-deck mount contract (remote paths under {remoteWorkDir}/jobs///): + docs/ -> /docs:ro + BDEF.md -> /BDEF.md:ro + schemas/.schema.json -> /schema.json:ro (extractor|grades) + personas/.md -> /persona/PERSONA.md:ro + out/ -> /out:rw Sandbox (per the operator's confidentiality requirement): * non-root, --cap-drop ALL, --security-opt no-new-privileges - * read-only rootfs + small writable tmpfs; only /docs (ro), /persona (ro), and - /out (rw, this reviewer's report dir) are mounted + * read-only rootfs + small writable tmpfs (/tmp, /home/rev) * NO docker socket, cpu/mem/pid caps * attached to the per-job network — in airgapped mode that network is - --internal, so the reviewer can reach ONLY the model proxy, never the internet + --internal, so the agent can reach ONLY the model proxy, never the internet -Reviewers hold no credentials beyond a dummy proxy key. +Agents hold no credentials beyond a dummy proxy key. """ from __future__ import annotations import os import re import shlex +import shutil -import spark_client as sc +import bm_config +import prompts import serving +import spark_client as sc -SANDBOX_SRC = os.path.join(os.path.dirname(os.path.abspath(__file__)), "sandbox") +ORCH_DIR = os.path.dirname(os.path.abspath(__file__)) +SANDBOX_SRC = os.path.join(ORCH_DIR, "sandbox") +SCHEMAS_SRC = os.path.join(ORCH_DIR, "schemas") +SCHEMA_FILES = ("extraction.schema.json", "grades.schema.json") + +HARDEN = ( + "--user 1000:1000 --security-opt no-new-privileges --cap-drop ALL " + "--read-only --tmpfs /tmp:size=256m --tmpfs /home/rev:size=128m " + "--pids-limit 256 --memory 6g --cpus 4" +) -def ensure_reviewer_image(cfg: dict, log) -> None: - """Build the reviewer image on the head Spark if missing (aarch64, native).""" +# ------------------------------------------------------------------ image +def ensure_grader_image(cfg: dict, log) -> None: + """Build the grader image on the head Spark if missing (aarch64, native).""" head = sc.head(cfg) image = cfg["graderImage"] q = shlex.quote r = sc.run(head, f"docker image inspect {q(image)} >/dev/null 2>&1 && echo PRESENT || echo MISSING", timeout=30) if "PRESENT" in (r.stdout or ""): - log(f"[reviewers] image {image} already present on {head.host}") + log(f"[graders] image {image} already present on {head.host}") return if not os.path.isdir(SANDBOX_SRC): - raise RuntimeError(f"reviewer build context missing at {SANDBOX_SRC} (image not baked in?)") + raise RuntimeError(f"grader build context missing at {SANDBOX_SRC} (image not baked in?)") remote_dir = f"{cfg['remoteWorkDir']}/sandbox-build" - log(f"[reviewers] building reviewer image {image} on {head.host} (first run; a few minutes)…") + log(f"[graders] building grader image {image} on {head.host} (first run; a few minutes)…") push = sc.push_dir(head, SANDBOX_SRC, remote_dir) if push.returncode != 0: - raise RuntimeError(f"rsync reviewer build context to {head.host} failed: {push.stderr}") + raise RuntimeError(f"rsync grader build context to {head.host} failed: {push.stderr}") b = sc.run(head, f"cd {q(remote_dir)} && IMAGE={q(image)} bash build.sh", timeout=1800) if b.returncode != 0: - raise RuntimeError(f"reviewer image build failed on {head.host}: {b.stderr or b.stdout}") - log(f"[reviewers] reviewer image built: {image}") + raise RuntimeError(f"grader image build failed on {head.host}: {b.stderr or b.stdout}") + log(f"[graders] grader image built: {image}") +# ------------------------------------------------------------------ roster def slug(name: str) -> str: s = re.sub(r"[^A-Za-z0-9]+", "-", (name or "").strip().lower()).strip("-") - return s or "reviewer" + return s or "grader" def roster(cfg: dict) -> list[dict]: - """Reviewer roster from config. Each: {rid, name, model alias, persona, temperature}.""" + """Grader roster from config. Each: {rid, name, model alias, persona, temperature}.""" out: list[dict] = [] seen: dict[str, int] = {} - for w in (cfg.get("reviewers") or []): - name = (w.get("name") or "reviewer").strip() + for w in (cfg.get("graders") or []): + name = (w.get("name") or "grader").strip() rid = slug(name) if rid in seen: seen[rid] += 1 @@ -76,87 +96,143 @@ def roster(cfg: dict) -> list[dict]: return out -def _container(cfg: dict, jobdir: str, rid: str, name: str, model: str, persona: str, - temperature, role: str, extra_mounts: str = "") -> str: - """docker run command for one reviewer/synthesizer container (detached, one-shot).""" - q = shlex.quote - net = serving.net_name(cfg) - base = serving.reviewer_proxy_base(cfg) - searxng = (cfg.get("searxngUrl") or "").strip() - persona_path = f"{jobdir}/personas/{rid}.md" +# ------------------------------------------------------------------ staging +def stage_deck_files(cfg: dict, local_deck_dir: str, panel: list[dict]) -> None: + """Write everything the role containers need into the LOCAL per-deck staging + dir (jobs.py rsyncs the whole dir to the Spark afterwards): the BDEF rubric, + both schemas, and one persona file per role agent (extractor + each grader + + adjudicator). Also pre-creates out/ and adjudicator-out/ so the rsync creates + them remotely with the SSH user's ownership (uid 1000 = the container user).""" + for sub in ("personas", "schemas", "out", "adjudicator-out"): + os.makedirs(os.path.join(local_deck_dir, sub), exist_ok=True) + with open(os.path.join(local_deck_dir, "BDEF.md"), "w") as f: + f.write(bm_config.bdef_text(cfg)) + for fn in SCHEMA_FILES: + shutil.copyfile(os.path.join(SCHEMAS_SRC, fn), + os.path.join(local_deck_dir, "schemas", fn)) + def persona(rid: str, text: str) -> None: + with open(os.path.join(local_deck_dir, "personas", f"{rid}.md"), "w") as f: + f.write((text or "").strip() + "\n") + + persona("extractor", prompts.extractor_persona()) + for r in panel: + persona(r["rid"], r["persona"] or prompts.default_grader_persona(r["name"])) + persona("adjudicator", + (cfg.get("adjudicatorPersona") or "").strip() or prompts.adjudicator_persona()) + + +# ------------------------------------------------------------------ containers +def container_suffix(deckdir: str) -> str: + """Short, docker-safe name suffix from the deck dir (…//).""" + parts = deckdir.rstrip("/").split("/") + tail = "-".join(parts[-2:]) if len(parts) >= 2 else parts[-1] + return slug(tail)[:48] or "deck" + + +def base_env(cfg: dict, rid: str, name: str, role: str, model: str, temperature) -> str: + """The env-var block shared by every role container (also used by adjudicator.py).""" + q = shlex.quote + base = serving.reviewer_proxy_base(cfg) env = ( - f"-e BM_REVIEWER_ID={q(rid)} -e BM_REVIEWER_NAME={q(name)} -e BM_ROLE={q(role)} " + f"-e BM_ROLE={q(role)} -e BM_GRADER_ID={q(rid)} -e BM_GRADER_NAME={q(name)} " f"-e BM_MODEL={q(model)} -e BM_LLM_BASE={q(base)} -e BM_LLM_KEY=sk-local " f"-e BM_MAX_MODEL_LEN={q(str(int(cfg.get('maxModelLen', 32768))))} " f"-e HOME=/home/rev " ) - if temperature is not None: + if role == "extractor": + env += "-e BM_TEMPERATURE=0.0 " + elif temperature is not None: env += f"-e BM_TEMPERATURE={q(str(temperature))} " - # web_search is offered ONLY in local_services mode with a SearXNG URL. - if cfg.get("networkMode") == "local_services" and searxng: - env += f"-e BM_SEARXNG_URL={q(searxng)} " - - harden = ( - "--user 1000:1000 --security-opt no-new-privileges --cap-drop ALL " - "--read-only --tmpfs /tmp:size=256m --tmpfs /home/rev:size=128m " - "--pids-limit 256 --memory 6g --cpus 4" - ) - mounts = ( - f"-v {q(jobdir)}/docs:/docs:ro " - f"-v {q(jobdir)}/out:/out " - f"-v {q(persona_path)}:/persona/PERSONA.md:ro " - + extra_mounts - ) - cname = f"bm-grader-{rid}" - return ( - f"docker rm -f {cname} >/dev/null 2>&1; " - f"docker run -d --name {cname} --network {q(net)} {harden} {env} {mounts} {q(cfg['graderImage'])}" - ) + return env -def _write_persona(cfg: dict, jobdir: str, rid: str, persona: str) -> None: +def _container(cfg: dict, deckdir: str, rid: str, name: str, model: str, + temperature, role: str) -> tuple[str, str]: + """(container name, docker run command) for one extractor/grader container.""" q = shlex.quote - sc.run(sc.head(cfg), - f"mkdir -p {q(jobdir)}/personas && printf '%s' {q(persona)} > {q(jobdir)}/personas/{rid}.md", - timeout=30) + net = serving.net_name(cfg) + schema_file = "extraction.schema.json" if role == "extractor" else "grades.schema.json" + env = base_env(cfg, rid, name, role, model, temperature) + mounts = ( + f"-v {q(deckdir)}/docs:/docs:ro " + f"-v {q(deckdir)}/BDEF.md:/BDEF.md:ro " + f"-v {q(deckdir)}/schemas/{schema_file}:/schema.json:ro " + f"-v {q(deckdir)}/personas/{rid}.md:/persona/PERSONA.md:ro " + f"-v {q(deckdir)}/out:/out " + ) + cname = f"bm-grader-{rid}-{container_suffix(deckdir)}" + cmd = ( + f"docker rm -f {cname} >/dev/null 2>&1; " + f"docker run -d --name {cname} --network {q(net)} {HARDEN} {env} {mounts} " + f"{q(cfg['graderImage'])}" + ) + return cname, cmd -def run_wave_reviewers(cfg: dict, jobdir: str, panel: list[dict], rubric: str, log, - wait_timeout: int = 1800) -> list[dict]: - """Launch every reviewer in `panel` (already filtered to this wave's models), - wait for them to finish, and report status. Reports land in /out.""" +def _wait_for(cfg: dict, cname: str, out_file: str, log, wait_timeout: int) -> tuple[str, bool]: + """docker-wait a launched container, check its output file, remove it.""" head = sc.head(cfg) q = shlex.quote - # Rubric is shared; write it once into the job dir, mounted into every container. - sc.run(head, f"mkdir -p {q(jobdir)}/out && printf '%s' {q(rubric)} > {q(jobdir)}/RUBRIC.md", timeout=30) + w = sc.run(head, f"docker wait {cname}", timeout=wait_timeout) + code = (w.stdout or "").strip() + chk = sc.run(head, f"test -s {q(out_file)} && echo OK || echo MISSING", timeout=30) + wrote = "OK" in (chk.stdout or "") + sc.run(head, f"docker rm -f {cname} 2>/dev/null; true", timeout=30) + return code, wrote + + +# ------------------------------------------------------------------ runs +def run_extractor(cfg: dict, remote_deck_dir: str, model_alias: str, log, + wait_timeout: int = 1800) -> dict: + """Launch the stage-1 extractor for one deck and wait for out/extraction.json.""" + head = sc.head(cfg) + cname, cmd = _container(cfg, remote_deck_dir, "extractor", "extractor", + model_alias, None, role="extractor") + r = sc.run(head, cmd, timeout=120) + if r.returncode != 0: + log(f"[graders] extractor launch FAILED: {r.stderr or r.stdout}") + return {"rid": "extractor", "model": model_alias, "ok": False, + "error": (r.stderr or r.stdout)[:300], "report": False} + log(f"[graders] up: extractor -> {model_alias}") + code, wrote = _wait_for(cfg, cname, f"{remote_deck_dir}/out/extraction.json", + log, wait_timeout) + log(f"[graders] extractor exited (code={code or '?'}), " + f"extraction.json={'written' if wrote else 'MISSING'}") + return {"rid": "extractor", "model": model_alias, "ok": True, "error": "", + "exit": code, "report": wrote} + + +def run_wave_graders(cfg: dict, remote_deck_dir: str, wave_panel: list[dict], log, + wait_timeout: int = 1800) -> list[dict]: + """Launch every grader in `wave_panel` (already filtered to this wave's + models) against one deck, wait for them, and report status. Grade JSONs land + in /out/.json.""" + head = sc.head(cfg) launched = [] - for r in panel: - _write_persona(cfg, jobdir, r["rid"], r["persona"]) - cmd = _container(cfg, jobdir, r["rid"], r["name"], r["model"], r["persona"], - r.get("temperature"), role="reviewer", - extra_mounts=f"-v {q(jobdir)}/RUBRIC.md:/RUBRIC.md:ro ") + for r in wave_panel: + cname, cmd = _container(cfg, remote_deck_dir, r["rid"], r["name"], r["model"], + r.get("temperature"), role="grader") res = sc.run(head, cmd, timeout=120) if res.returncode != 0: - log(f"[reviewers] launch {r['rid']} FAILED: {res.stderr or res.stdout}") - launched.append({**r, "ok": False, "error": (res.stderr or res.stdout)[:300]}) + log(f"[graders] launch {r['rid']} FAILED: {res.stderr or res.stdout}") + launched.append({**r, "ok": False, "error": (res.stderr or res.stdout)[:300], + "cname": cname}) continue - log(f"[reviewers] up: {r['rid']} -> {r['model']}") - launched.append({**r, "ok": True, "error": ""}) + log(f"[graders] up: {r['rid']} -> {r['model']}") + launched.append({**r, "ok": True, "error": "", "cname": cname}) # Wait for each launched container to exit (they run in parallel; waiting # sequentially still finishes when the slowest does). results = [] for r in launched: if not r["ok"]: - results.append(r) + results.append({k: v for k, v in r.items() if k != "cname"}) continue - cname = f"bm-grader-{r['rid']}" - w = sc.run(head, f"docker wait {cname}", timeout=wait_timeout) - code = (w.stdout or "").strip() - out_check = sc.run(head, f"test -s {q(jobdir)}/out/{q(r['rid'])}.md && echo OK || echo MISSING", timeout=30) - wrote = "OK" in (out_check.stdout or "") - log(f"[reviewers] {r['rid']} exited (code={code or '?'}), report={'written' if wrote else 'MISSING'}") - sc.run(head, f"docker rm -f {cname} 2>/dev/null; true", timeout=30) - results.append({**r, "exit": code, "report": wrote}) + code, wrote = _wait_for(cfg, r["cname"], f"{remote_deck_dir}/out/{r['rid']}.json", + log, wait_timeout) + log(f"[graders] {r['rid']} exited (code={code or '?'}), " + f"grades={'written' if wrote else 'MISSING'}") + results.append({k: v for k, v in r.items() if k != "cname"} + | {"exit": code, "report": wrote}) return results diff --git a/orchestrator/jobs.py b/orchestrator/jobs.py index a60ddfa..432f309 100644 --- a/orchestrator/jobs.py +++ b/orchestrator/jobs.py @@ -1,40 +1,61 @@ -"""The Boardroom Map job runner — convenes the review panel over dropped documents. +"""The Boardroom Map job runner — grades dropped board decks against the BDEF. -Runs as a background thread inside the FastAPI app. It does NOT run on a clock -like Nightshift; it reacts to triggers: +Runs as a background thread inside the FastAPI app. It does NOT run on a clock; +it reacts to triggers: - * an explicit "Run Review" (drops /data/state/run_request), or - * autoRunOnDrop: files landing in /data/inbox, once the inbox is stable. + * an explicit "Grade Decks" run request (drops /data/state/run_request), or + * autoRunOnDrop: files landing under /data/inbox//, once the + inbox is stable across two ticks. -One job at a time. A job: - 1. extract text from the inbox locally (CPU) — only text crosses to the Sparks - 2. rsync the text to a per-job dir on the head Spark - 3. serve the needed models in WAVES; run the reviewers for each wave - 4. optionally run the local lead-reviewer synthesis - 5. pull the reports back to /data/reports/, assemble latest.md - 6. wipe the documents from the Sparks (unless disabled) and tear serving down +One job at a time. A job iterates the discovered deck units OLDEST FIRST per +company, and for each deck: -All state (phase, current job, per-reviewer status, last report) is mirrored to + 1. extract text locally (CPU) — only text crosses to the Sparks + 2. rsync the per-deck bundle (docs/, BDEF.md, personas/, schemas/, out/, + adjudicator-out/) to {remoteWorkDir}/jobs//// + 3. serve the needed models in WAVES (graders' models ∪ the extractor model); + the extractor runs when its model's wave is up, graders in their waves + 4. pull out/, validate: extraction.json invalid => the DECK fails (the job + continues); grader JSONs are validated individually, invalid ones dropped; + fewer than 2 valid grade reports => the deck fails + 5. adjudicate (optional, non-fatal): a local model weighs the panel's evidence + 6. score deterministically (scoring.score_deck) against the company's pinned + targets + the prior deck's forward targets, and record it in the ledger + 7. render DECK_REPORT.md + refresh the company SCORECARD.md and the + /data/reports copies + +Then it wipes the remote job dir (unless disabled), tears serving down, and +moves the graded originals to /data/processed/// (the company folder +stays in the inbox for reuse). One deck's failure never kills the job: the job +ends "done" if at least one deck was graded. + +All state (phase, per-deck status, panel status, last report) is mirrored to /data/state/runtime.json so the Web UI can render it. """ from __future__ import annotations import json import os +import re import shutil import threading import time import traceback from collections import deque -from datetime import datetime +from datetime import datetime, timezone +import adjudicator as adj_mod import bm_config +import decks import extraction +import graders as gr_mod +import ledger as ledger_mod import preflight -import reviewers as rev_mod +import scorecard +import scoring import serving import spark_client as sc -import synthesis as synth_mod +import validate DATA_DIR = os.environ.get("BM_DATA_DIR", "/data") INBOX = os.path.join(DATA_DIR, "inbox") @@ -42,40 +63,77 @@ PROCESSED = os.path.join(DATA_DIR, "processed") STATE_DIR = os.path.join(DATA_DIR, "state") JOBS_DIR = os.path.join(STATE_DIR, "jobs") REPORTS_DIR = os.path.join(DATA_DIR, "reports") +LEDGER_DIR = os.path.join(DATA_DIR, "ledger") RUNTIME_PATH = os.path.join(STATE_DIR, "runtime.json") REQUEST_PATH = os.path.join(STATE_DIR, "run_request") TICK_SECONDS = 10 +RUNNING_PHASES = ("extracting", "grading", "adjudicating", "scoring", "collecting") + +# Canonical-ish reporting periods: 2026-Q2, 2026-H1, FY2026, 2026-05, 2026. +_PERIOD_RE = re.compile( + r"^(?:FY\s?-?\d{4}|\d{4}(?:[-/ ]?(?:Q[1-4]|H[12]|0[1-9]|1[0-2]))?)$", re.IGNORECASE) def _inbox_signature() -> tuple[int, str]: - """(count, signature) of supported files in the inbox, for stability checks.""" + """(count, signature) of supported files anywhere in the inbox tree, for + autoRunOnDrop stability checks (decks live in per-company subfolders).""" if not os.path.isdir(INBOX): return (0, "") items = [] - for fn in sorted(os.listdir(INBOX)): - p = os.path.join(INBOX, fn) - if os.path.isfile(p) and os.path.splitext(fn)[1].lower() in extraction.SUPPORTED: - items.append(f"{fn}:{os.path.getsize(p)}:{int(os.path.getmtime(p))}") + for root, _dirs, files in os.walk(INBOX): + for fn in files: + p = os.path.join(root, fn) + if os.path.splitext(fn)[1].lower() in extraction.SUPPORTED: + try: + items.append(f"{os.path.relpath(p, INBOX)}:{os.path.getsize(p)}:" + f"{int(os.path.getmtime(p))}") + except OSError: + pass + items.sort() return (len(items), "|".join(items)) +def _token(s: str) -> str: + t = re.sub(r"[^a-z0-9]+", "-", (s or "").lower()).strip("-") + return t or "deck" + + +def _composite(record) -> float | None: + """Best-effort composite lookup on the scoring record (shape owned by scoring.py).""" + if not isinstance(record, dict): + return None + for k in ("composite", "composite_score"): + v = record.get(k) + if isinstance(v, (int, float)): + return v + for parent in ("scores", "totals", "score"): + d = record.get(parent) + if isinstance(d, dict) and isinstance(d.get("composite"), (int, float)): + return d["composite"] + return None + + class JobRunner: def __init__(self): self._events = deque(maxlen=500) self._lock = threading.Lock() - self.phase = "idle" # idle | extracting | reviewing | synthesizing | collecting | done | error + self.phase = "idle" # idle | extracting | grading | adjudicating | scoring | collecting | done | error self.job_id = None self.message = "" self.panel: list[dict] = [] self.waves_total = 0 self.wave_index = 0 + self.decks_total = 0 + self.deck_index = 0 + self.company = None + self.period = None + self.decks: list[dict] = [] self.last_report_path = None self._thread = None self._last_sig = None - self._stable_sig = None self._last_done_sig = None - for d in (STATE_DIR, JOBS_DIR, REPORTS_DIR, INBOX, PROCESSED): + for d in (STATE_DIR, JOBS_DIR, REPORTS_DIR, LEDGER_DIR, INBOX, PROCESSED): os.makedirs(d, exist_ok=True) self._restore() @@ -108,11 +166,12 @@ class JobRunner: self.job_id = d.get("job_id") self.message = d.get("message", "") self.panel = d.get("panel", []) + self.decks = d.get("decks", []) self.last_report_path = d.get("last_report_path") for e in d.get("events", []): self._events.append(e) # A job can't survive a restart; reset a stuck running phase. - if self.phase in ("extracting", "reviewing", "synthesizing", "collecting"): + if self.phase in RUNNING_PHASES: self.phase = "idle" except Exception: pass @@ -125,6 +184,11 @@ class JobRunner: "panel": self.panel, "waves_total": self.waves_total, "wave_index": self.wave_index, + "decks_total": self.decks_total, + "deck_index": self.deck_index, + "company": self.company, + "period": self.period, + "decks": self.decks, "last_report_path": self.last_report_path, } @@ -136,7 +200,7 @@ class JobRunner: self._thread.start() def request_run(self): - """Public hook (used by the API) to request a review immediately.""" + """Public hook (used by the API) to request a grading run immediately.""" try: with open(REQUEST_PATH, "w") as f: f.write(str(time.time())) @@ -160,23 +224,17 @@ class JobRunner: if os.path.exists(REQUEST_PATH): os.remove(REQUEST_PATH) triggered = True - self.log("[runner] review requested") + self.log("[runner] grading run requested") elif cfg.get("autoRunOnDrop"): count, sig = _inbox_signature() if count and sig == self._last_sig and sig != self._last_done_sig: # stable across two ticks and not the batch we last processed triggered = True - self.log("[runner] inbox stable — auto-running review") + self.log("[runner] inbox stable — auto-running grading") self._last_sig = sig if not triggered: return - count, _ = _inbox_signature() - if not count: - self.log("[runner] nothing to review (inbox empty of supported files)") - self.phase = "idle" - self._persist() - return self._run_job(cfg) # ------------------------------------------------------------- the job @@ -186,114 +244,124 @@ class JobRunner: self.message = "" self.waves_total = 0 self.wave_index = 0 + self.decks_total = 0 + self.deck_index = 0 + self.company = None + self.period = None + self.decks = [] self.panel = [] - local_job = os.path.join(JOBS_DIR, job_id) - local_docs = os.path.join(local_job, "docs") - remote_job = f"{cfg['remoteWorkDir'].rstrip('/')}/jobs/{job_id}" - rubric = cfg.get("reviewInstructions") or bm_config.DEFAULT_RUBRIC - self.log(f"=== Review job {job_id} begins ===") + remote_root = f"{cfg['remoteWorkDir'].rstrip('/')}/jobs/{job_id}" + self.log(f"=== Grading job {job_id} begins ===") try: - # 1. Extract locally (only text crosses to the Sparks). - self.phase = "extracting"; self._persist() - manifest = extraction.extract_inbox(INBOX, local_docs, self.log) - ok_docs = [m for m in manifest if m["ok"]] - if not ok_docs: - raise RuntimeError("no documents could be extracted (unsupported or empty inbox)") - self.log(f"[runner] extracted {len(ok_docs)} document(s)") + # 1. Discover deck units (per-company folders; oldest first). + disc = decks.discover(INBOX) + units = disc.get("units") or [] + for fn in disc.get("skipped") or []: + self.log(f"[runner] WARNING: skipping root-level inbox file '{fn}' — " + "decks belong in /data/inbox//") + if not units: + raise RuntimeError("nothing to grade — drop decks into /data/inbox//") - # 2. Resolve the panel against the model catalog. - catalog = {m["alias"] for m in (cfg.get("models") or [])} - panel = rev_mod.roster(cfg) + # 2. Ledger, merged with the configured portfolio companies. + led = ledger_mod.Ledger(LEDGER_DIR) + led.merge_config_companies(cfg.get("companies") or []) + + # Resolve the grader panel + extractor model against the catalog. + models = cfg.get("models") or [] + catalog = {m["alias"] for m in models} + panel = gr_mod.roster(cfg) valid = [r for r in panel if r["model"] in catalog] - invalid = [r for r in panel if r["model"] not in catalog] - for r in invalid: - self.log(f"[runner] WARNING: reviewer '{r['name']}' uses unknown model '{r['model']}' — skipped") - if not valid: - raise RuntimeError("no reviewers reference a configured model (see Configure Models/Reviewers)") - self.panel = [{"name": r["name"], "model": r["model"], "status": "pending"} for r in valid] + for r in panel: + if r["model"] not in catalog: + self.log(f"[runner] WARNING: grader '{r['name']}' uses unknown model " + f"'{r['model']}' — skipped") + if len(valid) < 2: + raise RuntimeError( + "need at least 2 graders referencing configured models — every deck " + "requires >= 2 valid grade reports (see Configure Models/Graders)") + extractor_model = (cfg.get("extractorModel") or "").strip() or \ + (models[0]["alias"] if models else "") + if extractor_model not in catalog: + raise RuntimeError(f"extractor model '{extractor_model}' is not in the model catalog") - needed = {r["model"] for r in valid} - if cfg.get("synthesisEnabled"): - sm = synth_mod.pick_model(cfg) - if sm: - needed.add(sm) + needed = {r["model"] for r in valid} | {extractor_model} + adjudicate = bool(cfg.get("adjudicatorEnabled")) + adj_model = adj_mod.pick_model(cfg) if adjudicate else "" + needed_all = needed | ({adj_model} if adjudicate and adj_model else set()) # Air-gapped mode can't route to second-Spark models (internal net). if cfg.get("networkMode") == "airgapped": - cat = {m["alias"]: m for m in (cfg.get("models") or [])} - offenders = [a for a in needed if cat.get(a, {}).get("spark") == "secondary"] + cat = {m["alias"]: m for m in models} + offenders = [a for a in needed_all if cat.get(a, {}).get("spark") == "secondary"] if offenders: raise RuntimeError( "air-gapped mode requires all models on the head Spark, but these are " f"on the secondary: {', '.join(sorted(offenders))}. Move them to the " "primary Spark or switch to local-services mode.") - # 3. Ship text to the Spark + ensure infra. - self.phase = "reviewing"; self._persist() - push = sc.push_dir(sc.head(cfg), local_docs, f"{remote_job}/docs") - if push.returncode != 0: - raise RuntimeError(f"shipping documents to the Spark failed: {push.stderr}") - rev_mod.ensure_reviewer_image(cfg, self.log) + # 3. Infra once per job. + gr_mod.ensure_grader_image(cfg, self.log) serving.ensure_network(cfg, self.log) - # In local_services mode, warn early if the optional web_search backend - # (SearXNG, self-signed HTTPS) is unreachable — non-fatal. preflight.check_searxng(cfg, self.log) - - # 4. Run the panel in waves (synthesis is handled separately, below). - review_aliases = {r["model"] for r in valid} - waves = serving.plan_waves(cfg, review_aliases) - self.waves_total = len(waves) hf = bm_config.hf_token() - collected = [] - for i, wave in enumerate(waves, 1): - self.wave_index = i - wave_aliases = {m["alias"] for m in wave} - wpanel = [r for r in valid if r["model"] in wave_aliases] - self.log(f"[runner] wave {i}/{len(waves)}: models={sorted(wave_aliases)} " - f"reviewers={[r['name'] for r in wpanel]}") - serving.bring_up_wave(cfg, wave, hf, self.log) - self._await_serving(cfg, wave) - preflight.check_wave(cfg, wave, self.log) - res = rev_mod.run_wave_reviewers(cfg, remote_job, wpanel, rubric, self.log) - collected.extend(res) - self._mark_panel(res) - serving.tear_down_wave(cfg, wave, self.log) - # 5. Synthesis (its own single-model wave). - synth_ok = False - if cfg.get("synthesisEnabled"): - self.phase = "synthesizing"; self._persist() - sm = synth_mod.pick_model(cfg) - swave = serving.plan_waves(cfg, {sm}) - for wave in swave: - serving.bring_up_wave(cfg, wave, hf, self.log) - self._await_serving(cfg, wave) - preflight.check_wave(cfg, wave, self.log) - sres = synth_mod.run_synthesis(cfg, remote_job, rubric, self.log) - synth_ok = bool(sres.get("report")) - serving.tear_down_wave(cfg, wave, self.log) + # 4. Grade each deck unit, oldest first. One deck's failure never + # kills the job. + self.decks_total = len(units) + succeeded = 0 + for idx, unit in enumerate(units, 1): + self.deck_index = idx + self.company = unit["company_slug"] + self.period = unit.get("period") + entry = {"company": unit["company_slug"], "period": unit.get("period"), + "status": "running"} + self.decks.append(entry) + self.panel = [{"name": r["name"], "model": r["model"], "status": "pending"} + for r in valid] + self._persist() + try: + result = self._grade_deck(cfg, led, job_id, remote_root, unit, idx, + valid, extractor_model, adjudicate, hf) + entry.update({"status": "done", "period": result["period"], + "composite": result["composite"]}) + succeeded += 1 + comp = result["composite"] + self.log(f"[runner] deck done: {unit['company_slug']} {result['period']}" + f" composite={comp if comp is not None else '?'}") + except Exception as e: + entry.update({"status": "failed", "error": str(e)[:300]}) + self.log(f"[runner] DECK FAILED ({unit['company_slug']} " + f"{unit.get('period') or '?'}): {e}") + self.log(traceback.format_exc().splitlines()[-1]) + self._persist() - # 6. Collect reports + assemble. + # 5. Job-level report. self.phase = "collecting"; self._persist() - self._collect(cfg, job_id, remote_job, local_job, valid, manifest, synth_ok) + self._write_job_report(job_id) - # 7. Confidentiality: wipe the documents from the Spark. + # 6. Confidentiality: wipe the deck text from the Spark + teardown. if cfg.get("wipeRemoteDocs", True): - sc.run(sc.head(cfg), f"rm -rf {remote_job}", timeout=60) - self.log("[runner] wiped document text from the Spark") + sc.run(sc.head(cfg), f"rm -rf {remote_root}", timeout=120) + self.log("[runner] wiped deck text from the Spark") serving.tear_down_all(cfg, self.log) - # 8. Clear the inbox (move originals aside so they aren't re-reviewed). - self._drain_inbox(job_id) + # 7. Move the graded originals aside (the company folders stay). + self._drain_inbox(job_id, units) self._last_done_sig = _inbox_signature()[1] - self.phase = "done" - self.message = f"Reviewed {len(ok_docs)} document(s) with {len(valid)} reviewer(s)." - self.log(f"=== Review job {job_id} complete ===") + + if succeeded: + self.phase = "done" + self.message = (f"Graded {succeeded}/{len(units)} deck(s) with " + f"{len(valid)} grader(s).") + else: + self.phase = "error" + self.message = f"All {len(units)} deck(s) failed — see the activity log." + self.log(f"=== Grading job {job_id} complete ({succeeded}/{len(units)} decks) ===") self._persist() except Exception as e: self.phase = "error" - self.message = f"Review failed: {e}" + self.message = f"Grading failed: {e}" self.log(f"[runner] JOB FAILED — {e}") self.log(traceback.format_exc().splitlines()[-1]) try: @@ -302,6 +370,205 @@ class JobRunner: pass self._persist() + # ------------------------------------------------------------- one deck + def _grade_deck(self, cfg: dict, led, job_id: str, remote_root: str, unit: dict, + idx: int, panel: list[dict], extractor_model: str, + adjudicate: bool, hf) -> dict: + """Grade one deck unit end to end. Raises on deck failure (caller continues).""" + slug_c = unit["company_slug"] + # The staging/remote dir token; the final deck_id is the resolved period. + token = _token(unit["period"]) if unit.get("period") else f"deck-{idx:02d}" + local_deck = os.path.join(JOBS_DIR, job_id, slug_c, token) + remote_deck = f"{remote_root}/{slug_c}/{token}" + head = sc.head(cfg) + + # --- extract locally + stage + push ----------------------------------- + self.phase = "extracting"; self._persist() + manifest = extraction.extract_files(unit["files"], os.path.join(local_deck, "docs"), + self.log) + ok_docs = [m for m in manifest if m["ok"]] + for m in manifest: + if not m["ok"]: + self.log(f"[runner] WARNING: {m['source']}: {m['error']}") + if not ok_docs: + raise RuntimeError("no document text could be extracted from this deck") + gr_mod.stage_deck_files(cfg, local_deck, panel) + push = sc.push_dir(head, local_deck, remote_deck) + if push.returncode != 0: + raise RuntimeError(f"shipping deck text to the Spark failed: {push.stderr}") + + # --- serve in waves; extractor + graders run in their model's wave ---- + self.phase = "grading"; self._persist() + waves = serving.plan_waves(cfg, {r["model"] for r in panel} | {extractor_model}) + self.waves_total = len(waves) + for i, wave in enumerate(waves, 1): + self.wave_index = i + aliases = {m["alias"] for m in wave} + wpanel = [r for r in panel if r["model"] in aliases] + self.log(f"[runner] wave {i}/{len(waves)}: models={sorted(aliases)} " + f"graders={[r['name'] for r in wpanel]}" + f"{' +extractor' if extractor_model in aliases else ''}") + serving.bring_up_wave(cfg, wave, hf, self.log) + try: + self._await_serving(cfg, wave) + preflight.check_wave(cfg, wave, self.log) + if extractor_model in aliases: + er = gr_mod.run_extractor(cfg, remote_deck, extractor_model, self.log) + if not er.get("report"): + raise RuntimeError("extractor produced no extraction.json") + if wpanel: + res = gr_mod.run_wave_graders(cfg, remote_deck, wpanel, self.log) + self._mark_panel(res) + finally: + serving.tear_down_wave(cfg, wave, self.log) + + # --- pull the panel outputs + validate -------------------------------- + local_out = os.path.join(local_deck, "out") + pull = sc.pull_dir(head, f"{remote_deck}/out", local_out) + if pull.returncode != 0: + raise RuntimeError(f"pulling panel outputs from the Spark failed: {pull.stderr}") + + ext_path = os.path.join(local_out, "extraction.json") + ext_obj, ext_err = validate.validate_file(ext_path, "extraction") + if ext_obj is None or ext_err: + raise RuntimeError(f"extraction.json invalid: {ext_err or 'missing'}") + + period, deck_id = self._resolve_period(unit, ext_obj, ext_path, token) + self.period = period; self._persist() + + grades, panel_meta = [], [] + for r in panel: + gpath = os.path.join(local_out, f"{r['rid']}.json") + gobj, gerr = (None, "no output file") + if os.path.exists(gpath) and not os.path.exists(gpath + ".invalid"): + gobj, gerr = validate.validate_file(gpath, "grades") + ok = gobj is not None and not gerr + if ok: + grades.append(gobj) + else: + self.log(f"[runner] grader {r['rid']} report dropped: {gerr}") + panel_meta.append({"rid": r["rid"], "model": r["model"], "valid": ok}) + if len(grades) < 2: + raise RuntimeError(f"only {len(grades)} valid grade report(s) (need >= 2)") + + # --- adjudication (non-fatal) ------------------------------------------ + adjudication_md = None + if adjudicate: + self.phase = "adjudicating"; self._persist() + try: + adj_model = adj_mod.pick_model(cfg) + for wave in serving.plan_waves(cfg, {adj_model}): + serving.bring_up_wave(cfg, wave, hf, self.log) + try: + self._await_serving(cfg, wave) + preflight.check_wave(cfg, wave, self.log) + adj_mod.run_adjudication(cfg, remote_deck, self.log) + finally: + serving.tear_down_wave(cfg, wave, self.log) + local_adj = os.path.join(local_deck, "adjudicator-out") + sc.pull_dir(head, f"{remote_deck}/adjudicator-out", local_adj) + apath = os.path.join(local_adj, "ADJUDICATION.md") + if os.path.exists(apath): + adjudication_md = open(apath, errors="replace").read().strip() or None + if not adjudication_md: + self.log("[runner] WARNING: no adjudication produced (continuing without)") + except Exception as e: + self.log(f"[runner] WARNING: adjudication failed (non-fatal): {e}") + + # --- deterministic scoring + ledger ------------------------------------ + self.phase = "scoring"; self._persist() + company = led.ensure_company(slug_c) + pinned = company.get("pinned_targets") or [] + aliases_map = company.get("kpi_aliases") or {} + prior = led.prior_targets(slug_c, period) + report_deck_dir = os.path.join(REPORTS_DIR, job_id, slug_c, deck_id) + meta = { + "company": slug_c, + "period": period, + "deck_id": deck_id, + "job_id": job_id, + "graded_at": datetime.now(timezone.utc).isoformat(), + "panel": panel_meta, + "artifacts": { + "report_dir": report_deck_dir, + "extraction": os.path.join(report_deck_dir, "extraction.json"), + "grades": [os.path.join(report_deck_dir, f"{p['rid']}.json") + for p in panel_meta if p["valid"]], + "adjudication": (os.path.join(report_deck_dir, "ADJUDICATION.md") + if adjudication_md else None), + }, + } + record = scoring.score_deck(ext_obj, grades, pinned, prior, aliases_map, + cfg["weights"], meta) + rec_path = led.record_deck(slug_c, record, ext_obj.get("forward_targets") or []) + self.log(f"[runner] ledger updated: {rec_path}") + + # --- reports ------------------------------------------------------------ + self.phase = "collecting"; self._persist() + deck_md = scorecard.render_deck_report(record, ext_obj, adjudication_md) + if rec_path and str(rec_path).endswith(".json"): + ledger_md = os.path.splitext(str(rec_path))[0] + ".md" + else: + ledger_md = os.path.join(LEDGER_DIR, slug_c, "decks", f"{deck_id}.md") + os.makedirs(os.path.dirname(ledger_md), exist_ok=True) + with open(ledger_md, "w") as f: + f.write(deck_md) + + os.makedirs(report_deck_dir, exist_ok=True) + with open(os.path.join(report_deck_dir, "DECK_REPORT.md"), "w") as f: + f.write(deck_md) + for fn in sorted(os.listdir(local_out)): # extraction + raw grader jsons (+ .invalid) + src = os.path.join(local_out, fn) + if os.path.isfile(src): + shutil.copyfile(src, os.path.join(report_deck_dir, fn)) + if adjudication_md: + with open(os.path.join(report_deck_dir, "ADJUDICATION.md"), "w") as f: + f.write(adjudication_md + "\n") + + # Refresh the company scorecard + the /data/reports latest copy. + sc_md = scorecard.render_scorecard(led.get_company(slug_c), led.deck_records(slug_c)) + sc_path = os.path.join(LEDGER_DIR, slug_c, "SCORECARD.md") + os.makedirs(os.path.dirname(sc_path), exist_ok=True) + with open(sc_path, "w") as f: + f.write(sc_md) + with open(os.path.join(REPORTS_DIR, "latest-scorecard.md"), "w") as f: + f.write(sc_md) + self.log(f"[runner] reports saved to {report_deck_dir}") + + return {"period": period, "deck_id": deck_id, "composite": _composite(record), + "report_dir": report_deck_dir} + + def _resolve_period(self, unit: dict, ext_obj: dict, ext_path: str, + token: str) -> tuple[str, str]: + """(period, deck_id) for this deck. Filename-derived period wins; else the + extractor's deck.period if canonical-ish; else the file's mtime month + (flagged as period_inferred in the extraction's red-flag candidates).""" + if unit.get("period"): + return unit["period"], _token(unit["period"]) + p = ((ext_obj.get("deck") or {}).get("period") or "").strip() + if p and _PERIOD_RE.match(p): + self.log(f"[runner] period '{p}' taken from the deck text") + return p, _token(p) + try: + mtime = os.path.getmtime(unit["files"][0]) + except OSError: + mtime = time.time() + period = time.strftime("%Y-%m", time.localtime(mtime)) + ext_obj.setdefault("red_flag_candidates", []).append({ + "code": "period_inferred", + "description": ("Reporting period was not stated in the filename or the deck " + f"text; inferred from the file's modification time as {period}."), + "severity": 2, + "evidence": "", + }) + try: + with open(ext_path, "w") as f: + json.dump(ext_obj, f, indent=2) + except Exception: + pass + self.log(f"[runner] WARNING: period inferred from file mtime: {period}") + return period, (_token(period) or token) + # ------------------------------------------------------------- helpers def _await_serving(self, cfg, wave, timeout=900): self.log("[runner] waiting for wave serving to come online…") @@ -329,50 +596,55 @@ class JobRunner: p["status"] = "no-report" self._persist() - def _collect(self, cfg, job_id, remote_job, local_job, valid, manifest, synth_ok): - out_local = os.path.join(REPORTS_DIR, job_id) - os.makedirs(out_local, exist_ok=True) - sc.pull_dir(sc.head(cfg), f"{remote_job}/out", os.path.join(out_local, "reviewers")) - if synth_ok: - sc.pull_dir(sc.head(cfg), f"{remote_job}/synth-out", os.path.join(out_local, "synthesis")) - - # Assemble a single latest.md: the consolidated report if present, else a - # concatenation of the individual reports. - parts = [f"# Boardroom Map review — {job_id}\n", - "Documents reviewed: " + ", ".join(m["source"] for m in manifest if m["ok"]) + "\n", - "Panel: " + ", ".join(f"{r['name']} ({r['model']})" for r in valid) + "\n"] - consolidated = os.path.join(out_local, "synthesis", "CONSOLIDATED_REPORT.md") - if synth_ok and os.path.exists(consolidated): - parts.append("\n---\n\n## Consolidated report (lead reviewer)\n\n") - parts.append(open(consolidated, errors="replace").read()) - parts.append("\n\n---\n") - parts.append("\n## Individual reviewer reports\n") - rev_local = os.path.join(out_local, "reviewers") - if os.path.isdir(rev_local): - for fn in sorted(os.listdir(rev_local)): - if fn.endswith(".md"): - parts.append(f"\n### {fn[:-3]}\n\n") - parts.append(open(os.path.join(rev_local, fn), errors="replace").read()) - parts.append("\n") - assembled = "".join(parts) - with open(os.path.join(out_local, "report.md"), "w") as f: + def _write_job_report(self, job_id: str): + """latest.md — one job summary across every deck graded (or failed).""" + lines = [f"# Boardroom Map grading job — {job_id}\n"] + done = [d for d in self.decks if d.get("status") == "done"] + failed = [d for d in self.decks if d.get("status") == "failed"] + lines.append(f"Decks graded: {len(done)}/{len(self.decks)}\n") + lines.append("\n## Results\n") + for d in self.decks: + if d.get("status") == "done": + comp = d.get("composite") + comp_s = f"{comp:.1f}" if isinstance(comp, (int, float)) else "?" + lines.append(f"- **{d['company']}** — {d.get('period') or '?'}: " + f"composite **{comp_s}** / 100\n") + else: + lines.append(f"- **{d['company']}** — {d.get('period') or '?'}: " + f"FAILED — {d.get('error', 'unknown error')}\n") + if done: + lines.append("\nPer-deck reports (DECK_REPORT.md, extraction, raw grades, " + f"adjudication): `/data/reports/{job_id}///`.\n") + lines.append("Company scorecards: `/data/ledger//SCORECARD.md` " + "(latest copy at `/data/reports/latest-scorecard.md`).\n") + if failed: + lines.append("\nFailed decks were still moved to " + f"`/data/processed/{job_id}/` — re-drop them to regrade.\n") + assembled = "".join(lines) + out_dir = os.path.join(REPORTS_DIR, job_id) + os.makedirs(out_dir, exist_ok=True) + with open(os.path.join(out_dir, "report.md"), "w") as f: f.write(assembled) with open(os.path.join(REPORTS_DIR, "latest.md"), "w") as f: f.write(assembled) - self.last_report_path = os.path.join(out_local, "report.md") - self.log(f"[runner] reports saved to {out_local}") + self.last_report_path = os.path.join(out_dir, "report.md") - def _drain_inbox(self, job_id): - dest = os.path.join(PROCESSED, job_id) - os.makedirs(dest, exist_ok=True) - for fn in os.listdir(INBOX): - src = os.path.join(INBOX, fn) - if os.path.isfile(src): - try: - shutil.move(src, os.path.join(dest, fn)) - except Exception: - pass - self.log(f"[runner] inbox cleared (originals moved to processed/{job_id})") + def _drain_inbox(self, job_id: str, units: list[dict]): + """Move each unit's ORIGINAL files to /data/processed///. The + per-company inbox folders are kept — the user reuses them next quarter.""" + moved = 0 + for unit in units: + dest = os.path.join(PROCESSED, job_id, unit["company_slug"]) + os.makedirs(dest, exist_ok=True) + for src in unit["files"]: + if os.path.isfile(src): + try: + shutil.move(src, os.path.join(dest, os.path.basename(src))) + moved += 1 + except Exception: + pass + self.log(f"[runner] inbox cleared ({moved} file(s) moved to processed/{job_id}; " + "company folders kept)") # Module-level singleton used by app.py diff --git a/orchestrator/ledger.py b/orchestrator/ledger.py new file mode 100644 index 0000000..9f7255a --- /dev/null +++ b/orchestrator/ledger.py @@ -0,0 +1,180 @@ +"""The per-company running score ledger under /data/ledger. + +Layout: + /data/ledger//company.json identity, aliases, pinned targets, + extracted forward targets, history + /data/ledger//decks/.json full scoring record per graded deck + +Config (bm_config `companies`) is the source of truth for name/aliases/pinned +targets; extracted_targets and history are owned by the grading pipeline. +Re-graded decks supersede (rename, never delete) the previous record. +""" +from __future__ import annotations + +import json +import os +from datetime import datetime, timezone + +import decks as decks_mod + + +def atomic_write_json(path: str, obj) -> None: + """Write JSON via temp file + os.replace so readers never see a torn file.""" + tmp = f"{path}.tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump(obj, f, indent=2, sort_keys=False) + f.write("\n") + os.replace(tmp, path) + + +def parse_alias_lines(text: str) -> dict: + """Parse newline-separated "canonical=alias1;alias2" lines into a dict.""" + out: dict[str, list[str]] = {} + for line in (text or "").splitlines(): + line = line.strip() + if not line or "=" not in line: + continue + canonical, _, rest = line.partition("=") + canonical = canonical.strip().lower() + aliases = [a.strip() for a in rest.split(";") if a.strip()] + if canonical and aliases: + out[canonical] = aliases + return out + + +class Ledger: + def __init__(self, base_dir: str): + self.base = base_dir + os.makedirs(self.base, exist_ok=True) + + # ------------------------------------------------------------- paths + def _company_dir(self, slug: str) -> str: + return os.path.join(self.base, slug) + + def _company_path(self, slug: str) -> str: + return os.path.join(self._company_dir(slug), "company.json") + + def _decks_dir(self, slug: str) -> str: + return os.path.join(self._company_dir(slug), "decks") + + # ------------------------------------------------------------- companies + def ensure_company(self, slug: str, name: str | None = None) -> dict: + """Load the company, creating a skeleton entry on first sight.""" + existing = self.get_company(slug) + if existing is not None: + return existing + company = { + "schema_version": 1, + "slug": slug, + "name": name or slug, + "auto_created": name is None, + "kpi_aliases": {}, + "pinned_targets": [], + "extracted_targets": {}, + "history": [], + } + os.makedirs(self._company_dir(slug), exist_ok=True) + atomic_write_json(self._company_path(slug), company) + return company + + def get_company(self, slug: str) -> dict | None: + try: + with open(self._company_path(slug), encoding="utf-8") as f: + return json.load(f) + except (FileNotFoundError, json.JSONDecodeError): + return None + + def all_slugs(self) -> list[str]: + if not os.path.isdir(self.base): + return [] + return sorted(d for d in os.listdir(self.base) + if os.path.isfile(self._company_path(d))) + + def all_companies(self) -> list[dict]: + return [c for c in (self.get_company(s) for s in self.all_slugs()) if c] + + def merge_config_companies(self, companies_cfg: list) -> None: + """Config wins for name/aliases/pinned targets; ledger keeps the rest.""" + for cc in companies_cfg or []: + name = (cc.get("name") or "").strip() + slug = (cc.get("slug") or "").strip() or decks_mod.slugify(name) + company = self.ensure_company(slug, name or slug) + company["name"] = name or company["name"] + company["auto_created"] = False + company["kpi_aliases"] = parse_alias_lines(cc.get("kpiAliases") or "") + company["pinned_targets"] = list(cc.get("pinnedTargets") or []) + atomic_write_json(self._company_path(slug), company) + + # ------------------------------------------------------------- targets + def prior_targets(self, slug: str, period: str) -> list[dict]: + """Forward targets an earlier deck set for `period` (this deck's exam).""" + company = self.get_company(slug) + if not company or not period: + return [] + return (company.get("extracted_targets", {}).get(period) or {}).get("targets", []) + + # ------------------------------------------------------------- decks + def record_deck(self, slug: str, record: dict, forward_targets: list[dict]) -> str: + """Persist a scoring record + its forward targets. Returns the deck path. + + A re-graded deck_id supersedes (renames) the old record; the history + entry for the same period is replaced; a target period's forward + targets are replaced wholesale when set by a newer (or same) deck.""" + company = self.ensure_company(slug) + deck_id = record["deck_id"] + ddir = self._decks_dir(slug) + os.makedirs(ddir, exist_ok=True) + deck_path = os.path.join(ddir, f"{deck_id}.json") + if os.path.exists(deck_path): + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + os.replace(deck_path, os.path.join(ddir, f"{deck_id}.superseded-{stamp}.json")) + atomic_write_json(deck_path, record) + + period = record.get("period") + entry = {"period": period, "composite": record.get("composite"), + "graded_at": record.get("graded_at"), + "deck_file": f"decks/{deck_id}.json"} + history = [h for h in company.get("history", []) if h.get("period") != period] + history.append(entry) + history.sort(key=lambda h: decks_mod.period_sort_key(h.get("period"))) + company["history"] = history + + extracted = company.setdefault("extracted_targets", {}) + from_key = decks_mod.period_sort_key(period) + by_period: dict[str, list[dict]] = {} + for ft in forward_targets or []: + tp = ft.get("target_period") + if tp: + by_period.setdefault(tp, []).append(ft) + for tp, targets in by_period.items(): + cur = extracted.get(tp) + if cur is None or from_key >= decks_mod.period_sort_key(cur.get("from_deck")): + extracted[tp] = {"from_deck": period, "targets": targets} + + atomic_write_json(self._company_path(slug), company) + return deck_path + + def deck_record(self, slug: str, deck_id: str) -> dict | None: + try: + with open(os.path.join(self._decks_dir(slug), f"{deck_id}.json"), + encoding="utf-8") as f: + return json.load(f) + except (FileNotFoundError, json.JSONDecodeError): + return None + + def deck_records(self, slug: str) -> list[dict]: + """All live (non-superseded) deck records, oldest period first.""" + ddir = self._decks_dir(slug) + if not os.path.isdir(ddir): + return [] + out: list[dict] = [] + for fn in sorted(os.listdir(ddir)): + if not fn.endswith(".json") or ".superseded-" in fn: + continue + try: + with open(os.path.join(ddir, fn), encoding="utf-8") as f: + out.append(json.load(f)) + except (json.JSONDecodeError, OSError): + continue + out.sort(key=lambda r: decks_mod.period_sort_key(r.get("period"))) + return out diff --git a/orchestrator/prompts.py b/orchestrator/prompts.py new file mode 100644 index 0000000..c1d6fb0 --- /dev/null +++ b/orchestrator/prompts.py @@ -0,0 +1,80 @@ +"""Persona texts for the extractor, graders, and adjudicator. + +These are the PERSONA.md contents written into each per-job dir and mounted +into the one-shot sandbox containers. The mechanical role instructions (read +/docs, emit JSON matching the schema at /schema.json, write to /out) live in +the sandbox agent itself — these texts only shape judgment and voice. +""" +from __future__ import annotations + + +def extractor_persona() -> str: + """Stage-1 structured extractor: a forensic analyst, never a calculator.""" + return ( + "You are a forensic financial analyst extracting structured data from a " + "portfolio-company board deck. You are exhaustive and literal.\n\n" + "Extract:\n" + "- EVERY quantitative KPI actual reported for the deck's period: revenue, " + "ARR, margins, burn, cash, churn, NRR, headcount, pipeline — anything with " + "a number attached to a metric.\n" + "- Every stated forward target or guidance, with the exact period it " + "applies to (target_period).\n" + "- Red-flag candidates, using ONLY the taxonomy codes from the BDEF rubric " + "(adjusted_metrics, metric_redefinition, kpi_dropped, hockey_stick_forecast, " + "channel_stuffing_risk, short_term_comp, related_party, governance_gap, " + "cash_runway_silence, no_profitability_visibility, overreach_adjacency, " + "activity_bias, complexity_smokescreen, suppressed_dissent).\n" + "- Deck metadata: company hint, reporting period as printed, meeting date, " + "title.\n\n" + "Rules:\n" + "- canonical_name is lower_snake_case, GENERIC, and stable across quarters: " + "arr, ebitda_margin, churn_rate — not q2_arr_2026 or acme_revenue.\n" + "- profitability=true ONLY for profit/margin/cash metrics (EBITDA, net " + "margin, FCF, burn, runway) — never growth or activity metrics.\n" + "- NEVER compute, derive, or infer a number that is not printed in the " + "deck. If a margin is not printed, do not divide two numbers to get it.\n" + "- Copy the source location for every item (e.g. 'slide 6, financial " + "summary').\n" + "- direction: gte when higher is better, lte when lower is better " + "(churn, burn, CAC).\n" + "- target_in_deck is only a target printed NEXT TO the actual for the SAME " + "period; guidance for future periods goes in forward_targets." + ) + + +def default_grader_persona(name: str) -> str: + """Neutral BDEF lens for graders the operator has not customized.""" + return ( + f"You are '{name}', an owner's representative on the board grading this " + "deck strictly against the BDEF rubric provided.\n\n" + "- Score each category A-H from 1 to 5. A score above or below 3 REQUIRES " + "verbatim evidence quotes from the deck, with locations.\n" + "- Judge what the deck actually shows. Absence of evidence on a category " + "is itself information: score 2-3 and note the absence — never guess in " + "management's favor.\n" + "- Quote exactly; do not paraphrase inside quotes.\n" + "- Raise red flags only with the rubric's taxonomy codes, each with the " + "evidence that triggered it.\n" + "- Be specific and terse in rationales; write for a board member with " + "five minutes.\n" + "- Do NOT compute totals or a composite score; numbers are computed " + "elsewhere." + ) + + +def adjudicator_persona() -> str: + """Panel chair: consolidates the graders' verdicts, adds no new scores.""" + return ( + "You are the panel chair. You did not grade the deck yourself — you read " + "the graders' completed evaluations and adjudicate.\n\n" + "Produce a short markdown memo covering:\n" + "1. Consensus: what the panel agrees on, in one tight paragraph.\n" + "2. Disagreements: where graders diverge, which grader's evidence is " + "stronger and why (judge the quotes, not the adjectives).\n" + "3. Red flags: confirm or dismiss each raised flag based on the cited " + "evidence; say which deserve board attention.\n" + "4. Exactly 3 questions the board should ask management next quarter — " + "high-leverage, inversion-minded, answerable with data.\n\n" + "Attribute points to the grader(s) who raised them. Do not invent " + "findings, do not re-grade, and do NOT produce scores or totals." + ) diff --git a/orchestrator/requirements.txt b/orchestrator/requirements.txt index 84524a6..c80ee0d 100644 --- a/orchestrator/requirements.txt +++ b/orchestrator/requirements.txt @@ -5,3 +5,5 @@ python-multipart==0.0.20 pyyaml==6.0.2 pypdf==5.1.0 python-docx==1.1.2 +python-pptx==1.0.2 +jsonschema==4.23.0 diff --git a/orchestrator/schemas/extraction.schema.json b/orchestrator/schemas/extraction.schema.json new file mode 100644 index 0000000..9bc6362 --- /dev/null +++ b/orchestrator/schemas/extraction.schema.json @@ -0,0 +1,83 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "boardroom_extraction", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "deck", "kpis", "forward_targets", "red_flag_candidates", "narrative"], + "properties": { + "schema_version": {"type": "integer"}, + "deck": { + "type": "object", + "additionalProperties": false, + "required": ["period"], + "properties": { + "company_hint": {"type": ["string", "null"]}, + "period": {"type": ["string", "null"], "description": "Reporting period as printed on the deck, e.g. 2026-Q2, 2026-H1, FY2026, 2026-05"}, + "meeting_date": {"type": ["string", "null"]}, + "title": {"type": ["string", "null"]}, + "truncated": {"type": "boolean", "default": false} + } + }, + "kpis": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["name", "canonical_name", "actual", "direction", "profitability", "source"], + "properties": { + "name": {"type": "string", "description": "KPI label exactly as printed in the deck"}, + "canonical_name": {"type": "string", "pattern": "^[a-z0-9_]+$", "description": "lower_snake_case, generic, stable across quarters (arr, ebitda_margin, churn_rate...)"}, + "actual": {"type": "number"}, + "unit": {"type": "string", "default": ""}, + "period": {"type": ["string", "null"]}, + "direction": {"type": "string", "enum": ["gte", "lte"], "description": "gte = higher is better, lte = lower is better"}, + "profitability": {"type": "boolean", "description": "true only for profit/margin/cash metrics (EBITDA, net margin, FCF, burn...)"}, + "target_in_deck": {"type": ["number", "null"], "description": "Target/plan value printed NEXT TO the actual for the SAME period, if any"}, + "source": {"type": "string", "description": "Where in the deck, e.g. 'slide 6, financial summary'"}, + "notes": {"type": "string", "default": ""} + } + } + }, + "forward_targets": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["name", "canonical_name", "target", "target_period", "direction", "profitability", "source"], + "properties": { + "name": {"type": "string"}, + "canonical_name": {"type": "string", "pattern": "^[a-z0-9_]+$"}, + "target": {"type": "number"}, + "unit": {"type": "string", "default": ""}, + "target_period": {"type": "string", "description": "Period this guidance applies to, e.g. 2026-Q3"}, + "direction": {"type": "string", "enum": ["gte", "lte"]}, + "profitability": {"type": "boolean"}, + "source": {"type": "string"} + } + } + }, + "red_flag_candidates": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["code", "description", "severity"], + "properties": { + "code": {"type": "string", "pattern": "^[a-z0-9_]+$"}, + "description": {"type": "string"}, + "severity": {"type": "integer", "minimum": 1, "maximum": 5}, + "evidence": {"type": "string", "default": ""} + } + } + }, + "narrative": { + "type": "object", + "additionalProperties": false, + "required": ["summary"], + "properties": { + "summary": {"type": "string"}, + "asks": {"type": "array", "items": {"type": "string"}, "default": []} + } + } + } +} diff --git a/orchestrator/schemas/grades.schema.json b/orchestrator/schemas/grades.schema.json new file mode 100644 index 0000000..540c106 --- /dev/null +++ b/orchestrator/schemas/grades.schema.json @@ -0,0 +1,53 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "boardroom_grades", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "grader", "categories", "red_flags", "overall_comment"], + "properties": { + "schema_version": {"type": "integer"}, + "grader": {"type": "string"}, + "categories": { + "type": "array", + "minItems": 8, + "maxItems": 8, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["id", "score", "evidence", "rationale"], + "properties": { + "id": {"type": "string", "enum": ["A", "B", "C", "D", "E", "F", "G", "H"]}, + "score": {"type": "integer", "minimum": 1, "maximum": 5}, + "evidence": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["quote", "location"], + "properties": { + "quote": {"type": "string", "description": "Verbatim text from the deck"}, + "location": {"type": "string", "description": "e.g. 'slide 3'"} + } + } + }, + "rationale": {"type": "string"} + } + } + }, + "red_flags": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["code", "description", "severity"], + "properties": { + "code": {"type": "string", "pattern": "^[a-z0-9_]+$"}, + "description": {"type": "string"}, + "severity": {"type": "integer", "minimum": 1, "maximum": 5}, + "evidence": {"type": "string", "default": ""} + } + } + }, + "overall_comment": {"type": "string"} + } +} diff --git a/orchestrator/scorecard.py b/orchestrator/scorecard.py new file mode 100644 index 0000000..4c47bda --- /dev/null +++ b/orchestrator/scorecard.py @@ -0,0 +1,265 @@ +"""Markdown renderers: per-deck DECK_REPORT and per-company SCORECARD. + +Pure string builders over the scoring record shape (scoring.score_deck) and +ledger deck records — no I/O here; jobs.py decides where the files land. +""" +from __future__ import annotations + +_CATEGORIES = "ABCDEFGH" +_CATEGORY_TITLES = { + "A": "Incentive Alignment & Skin in the Game", + "B": "Inversion Discipline & Margin of Safety", + "C": "Circle of Competence & Rational Learning", + "D": "Capital Allocation Quality", + "E": "Moat Durability & Competitive Reality", + "F": "Psychological & Cultural Health", + "G": "Simplicity, Clarity & Decision Velocity", + "H": "Board Value-Add & Governance Quality", +} + + +def _fmt(x, digits: int = 1) -> str: + if x is None: + return "—" + if isinstance(x, bool): + return "yes" if x else "no" + if isinstance(x, float): + return f"{x:.{digits}f}" + return str(x) + + +def _num(x, unit: str = "") -> str: + if x is None: + return "—" + s = f"{x:g}" if isinstance(x, (int, float)) else str(x) + return f"{s}{unit}" if unit and unit in ("%",) else (f"{s} {unit}".strip() if unit else s) + + +def _bucket_row(name: str, b: dict) -> str: + if b.get("na"): + return f"| {name} | — | — | {b.get('kpi_count', 0)} | NA — weight redistributed |" + return (f"| {name} | {_fmt(b.get('weight'))} | {_fmt(b.get('score'))} " + f"| {b.get('kpi_count', 0)} | |") + + +def render_deck_report(record: dict, extraction: dict, adjudication_md: str | None = None) -> str: + """One deck's full markdown report.""" + lines: list[str] = [] + company = record.get("company") or "?" + period = record.get("period") or "unknown period" + lines.append(f"# Deck report — {company} · {period}") + lines.append("") + lines.append(f"## Composite: **{_fmt(record.get('composite'))} / 100**") + lines.append("") + q = record.get("quant", {}) + lines.append(f"Quant {_fmt(q.get('score'))} · Qual {_fmt(record.get('qual', {}).get('score'))}" + f" · Penalties −{_fmt(record.get('penalties', {}).get('total'))}" + f" · graded {record.get('graded_at') or '?'} (job {record.get('job_id') or '?'})") + lines.append("") + + # --- quantitative buckets + lines.append("## Quantitative (max 60)") + lines.append("") + lines.append("| Bucket | Weight | Score | KPIs | Note |") + lines.append("|---|---|---|---|---|") + lines.append(_bucket_row("Profitability KPIs", q.get("profitability", {}))) + lines.append(_bucket_row("Other KPIs", q.get("other", {}))) + lines.append(_bucket_row("Forecast integrity", q.get("forecast_integrity", {}))) + lines.append("") + + kpi_results = record.get("kpi_results") or [] + if kpi_results: + lines.append("### KPI results") + lines.append("") + lines.append("| KPI | Actual | Target | Source | Credit |") + lines.append("|---|---|---|---|---|") + for r in kpi_results: + name = r.get("name") or r.get("canonical_name") or "?" + if r.get("matched_via") == "fuzzy": + name += " (≈ matched via fuzzy)" + src = r.get("target_source") or "—" + credit = "—" if r.get("credit") is None else _fmt(r.get("credit"), 2) + lines.append(f"| {name} | {_num(r.get('actual'), r.get('unit') or '')} " + f"| {_num(r.get('target'), r.get('unit') or '')} | {src} | {credit} |") + lines.append("") + + fresults = record.get("forecast_results") or [] + if fresults: + lines.append("### Forecast integrity (prior targets vs this period's actuals)") + lines.append("") + lines.append("| KPI | Prior target | Actual | Accuracy |") + lines.append("|---|---|---|---|") + for r in fresults: + lines.append(f"| {r.get('canonical_name')} | {_num(r.get('target'))} " + f"| {_num(r.get('actual'))} | {_fmt(r.get('accuracy'), 2)} |") + lines.append("") + + # --- qualitative + lines.append("## Qualitative (max 40)") + lines.append("") + lines.append("| Category | Median | Evidence quality | Adjusted | Points |") + lines.append("|---|---|---|---|---|") + cats = record.get("qual", {}).get("categories", {}) + for cid in _CATEGORIES: + c = cats.get(cid, {}) + lines.append(f"| {cid}. {_CATEGORY_TITLES[cid]} | {_fmt(c.get('median'))} " + f"| {_fmt(c.get('evidence_quality'), 2)} | {_fmt(c.get('adjusted'), 2)} " + f"| {_fmt(c.get('points'), 2)} |") + lines.append("") + for cid in _CATEGORIES: + c = cats.get(cid, {}) + rats = c.get("rationales") or [] + if not rats: + continue + best = max(rats, key=lambda r: sum(len(e.get("quote") or "") for e in r.get("evidence") or [])) + lines.append(f"### {cid}. {_CATEGORY_TITLES[cid]}") + lines.append("") + lines.append(f"**{best.get('grader')}**: {best.get('rationale')}") + for ev in best.get("evidence") or []: + loc = f" — {ev.get('location')}" if ev.get("location") else "" + lines.append(f"> \"{ev.get('quote')}\"{loc}") + lines.append("") + + # --- red flags + flags = record.get("penalties", {}).get("flags") or [] + lines.append(f"## Red flags (penalty −{_fmt(record.get('penalties', {}).get('total'))})") + lines.append("") + if flags: + lines.append("| Code | Severity | Points | Sources | Description |") + lines.append("|---|---|---|---|---|") + for f in flags: + lines.append(f"| `{f.get('code')}` | {f.get('severity')} | {_fmt(f.get('points'))} " + f"| {', '.join(f.get('sources') or [])} | {f.get('description')} |") + else: + lines.append("None raised.") + lines.append("") + + # --- narrative + narrative = record.get("narrative") or extraction.get("narrative") or {} + if narrative.get("summary"): + lines.append("## Narrative") + lines.append("") + lines.append(narrative["summary"]) + lines.append("") + asks = narrative.get("asks") or [] + if asks: + lines.append("### Asks") + lines.append("") + for a in asks: + lines.append(f"- {a}") + lines.append("") + + if adjudication_md: + lines.append("## Panel adjudication") + lines.append("") + lines.append(adjudication_md.strip()) + lines.append("") + return "\n".join(lines).rstrip() + "\n" + + +def _arrow(delta: float) -> str: + if delta > 0: + return "↑" + if delta < 0: + return "↓" + return "→" + + +def render_scorecard(company: dict, records: list[dict]) -> str: + """Company SCORECARD.md across all live deck records (oldest first).""" + name = company.get("name") or company.get("slug") or "?" + lines: list[str] = [f"# Scorecard — {name}", ""] + if not records: + lines.append("No graded decks yet.") + return "\n".join(lines) + "\n" + + latest = records[-1] + prev = records[-2] if len(records) > 1 else None + comp = latest.get("composite") or 0.0 + if prev is not None: + delta = round(comp - (prev.get("composite") or 0.0), 1) + lines.append(f"## Latest composite: **{_fmt(comp)}** ({latest.get('period')}) " + f"{_arrow(delta)} {'+' if delta > 0 else ''}{_fmt(delta)} vs {prev.get('period')}") + else: + lines.append(f"## Latest composite: **{_fmt(comp)}** ({latest.get('period')}) — first graded deck") + lines.append("") + + # --- composite history + lines.append("## Composite history") + lines.append("") + lines.append("| Period | Composite | Quant | Qual | Penalties |") + lines.append("|---|---|---|---|---|") + for r in records: + lines.append(f"| {r.get('period') or '?'} | {_fmt(r.get('composite'))} " + f"| {_fmt(r.get('quant', {}).get('score'))} " + f"| {_fmt(r.get('qual', {}).get('score'))} " + f"| −{_fmt(r.get('penalties', {}).get('total'))} |") + lines.append("") + + # --- categories latest vs previous + lines.append("## BDEF categories (latest vs previous)") + lines.append("") + lines.append("| Category | Latest | Previous | Δ |") + lines.append("|---|---|---|---|") + lcats = latest.get("qual", {}).get("categories", {}) + pcats = (prev or {}).get("qual", {}).get("categories", {}) + for cid in _CATEGORIES: + lp = lcats.get(cid, {}).get("points") + pp = pcats.get(cid, {}).get("points") + if lp is not None and pp is not None: + d = round(lp - pp, 2) + dcol = f"{_arrow(d)} {'+' if d > 0 else ''}{_fmt(d, 2)}" + else: + dcol = "—" + lines.append(f"| {cid}. {_CATEGORY_TITLES[cid]} | {_fmt(lp, 2)} | {_fmt(pp, 2)} | {dcol} |") + lines.append("") + + # --- KPI hit-rate across records + order: list[str] = [] + per_kpi: dict[str, list] = {} + for r in records: + for k in r.get("kpi_results") or []: + cn = k.get("canonical_name") or k.get("name") or "?" + if cn not in per_kpi: + per_kpi[cn] = [] + order.append(cn) + per_kpi[cn].append(k.get("credit")) + lines.append("## KPI hit-rate") + lines.append("") + lines.append("| KPI | Attempts | Hits | Streak | Last credit |") + lines.append("|---|---|---|---|---|") + for cn in order: + credits = [c for c in per_kpi[cn] if c is not None] + if not credits: + continue + hits = sum(1 for c in credits if c >= 1) + streak = 0 + for c in reversed(credits): + if c >= 1: + streak += 1 + else: + break + lines.append(f"| {cn} | {len(credits)} | {hits} | {streak} | {_fmt(credits[-1], 2)} |") + lines.append("") + + # --- open flags on the latest deck + flags = latest.get("penalties", {}).get("flags") or [] + lines.append(f"## Open flags ({latest.get('period')})") + lines.append("") + if flags: + lines.append("| Code | Severity | Points | Sources | Description |") + lines.append("|---|---|---|---|---|") + for f in flags: + lines.append(f"| `{f.get('code')}` | {f.get('severity')} | {_fmt(f.get('points'))} " + f"| {', '.join(f.get('sources') or [])} | {f.get('description')} |") + else: + lines.append("None.") + lines.append("") + + # --- deck reports + lines.append("## Deck reports") + lines.append("") + for r in records: + lines.append(f"- {r.get('period') or '?'} — composite {_fmt(r.get('composite'))} — " + f"decks/{r.get('deck_id')}.json / DECK_REPORT.md") + return "\n".join(lines).rstrip() + "\n" diff --git a/orchestrator/scoring.py b/orchestrator/scoring.py new file mode 100644 index 0000000..fedb7ac --- /dev/null +++ b/orchestrator/scoring.py @@ -0,0 +1,327 @@ +"""Deterministic BDEF scoring — pure functions, stdlib only, no I/O. + +Everything numeric happens here, never in a model: the panel supplies 1-5 +category scores with verbatim evidence, the extractor supplies KPI actuals and +targets, and this module turns them into the 0-100 composite: + + composite = quant (60) + qualitative (40) - red-flag penalties (cap 15) + +All knobs come from the `weights` dict (bm_config.WEIGHTS_DEFAULTS shape) so +the operator can retune without a rebuild. +""" +from __future__ import annotations + +import difflib +import re +import statistics + +FUZZY_THRESHOLD = 0.85 +_CATEGORIES = "ABCDEFGH" + + +# ---------------------------------------------------------------- KPI matching +def _norm(name: str) -> str: + return re.sub(r"[^a-z0-9]+", " ", (name or "").lower()).strip() + + +def match_kpi(canonical: str, candidates: list[dict], aliases: dict) -> tuple[dict | None, str | None]: + """Match a canonical KPI name against candidate dicts ({canonical_name, name}). + + Exact canonical -> alias map (either direction) -> fuzzy ratio >= 0.85. + Returns (candidate, via) with via in exact|alias|fuzzy, or (None, None).""" + canon = (canonical or "").strip().lower() + if not canon: + return None, None + for c in candidates: + if (c.get("canonical_name") or "").strip().lower() == canon: + return c, "exact" + amap = {(k or "").strip().lower(): {(a or "").strip().lower() for a in (v or [])} + for k, v in (aliases or {}).items()} + ours = amap.get(canon, set()) + for c in candidates: + cn = (c.get("canonical_name") or "").strip().lower() + nm = (c.get("name") or "").strip().lower() + if cn in ours or nm in ours or canon in amap.get(cn, set()): + return c, "alias" + best, best_r = None, 0.0 + for c in candidates: + for other in (c.get("canonical_name") or "", c.get("name") or ""): + r = difflib.SequenceMatcher(None, _norm(canon), _norm(other)).ratio() + if r > best_r: + best, best_r = c, r + if best is not None and best_r >= FUZZY_THRESHOLD: + return best, "fuzzy" + return None, None + + +# ---------------------------------------------------------------- KPI credit +def _passes(actual: float, target: float, direction: str) -> bool: + return actual <= target if direction == "lte" else actual >= target + + +def _credit(actual: float, target: float, direction: str, floor: float) -> float: + """Partial credit for a targeted KPI: 0 below floor, linear to 1 at target.""" + if target == 0 or (actual < 0) != (target < 0) or (direction == "lte" and actual == 0): + return 1.0 if _passes(actual, target, direction) else 0.0 + r = target / actual if direction == "lte" else actual / target + if actual < 0 and target < 0: + # Both negative (EBITDA margin target -2, actual -3): the plain ratio + # inverts the ordering, so flip it back. + r = 1.0 / r + if r >= 1: + return 1.0 + if floor >= 1 or r < floor: + return 0.0 + return max(0.0, min(1.0, (r - floor) / (1 - floor))) + + +def _resolve_target(kpi: dict, pinned_targets: list[dict], prior_targets: list[dict], + aliases: dict) -> tuple[dict | None, str | None, str | None]: + """(target dict {target, direction}, target_source, matched_via) for one KPI. + + Precedence: pinned config target > prior deck's extracted forward target + for this period > target printed in the deck itself.""" + canon = kpi.get("canonical_name") or "" + pinned_cands = [{"canonical_name": p.get("kpi"), "name": p.get("kpi"), "_src": p} + for p in (pinned_targets or [])] + cand, via = match_kpi(canon, pinned_cands, aliases) + if cand is not None: + p = cand["_src"] + return {"target": p.get("target"), "direction": p.get("direction") or kpi.get("direction")}, "pinned", via + cand, via = match_kpi(canon, prior_targets or [], aliases) + if cand is not None: + return {"target": cand.get("target"), + "direction": cand.get("direction") or kpi.get("direction")}, "extracted", via + if kpi.get("target_in_deck") is not None: + return {"target": kpi["target_in_deck"], "direction": kpi.get("direction")}, "in_deck", None + return None, None, None + + +# ---------------------------------------------------------------- scoring +def _bucket(results: list[dict], weight: float) -> dict: + if not results: + return {"score": 0.0, "weight": 0.0, "na": True, "kpi_count": 0} + mean = sum(r["credit"] for r in results) / len(results) + return {"score": mean, "weight": weight, "na": False, "kpi_count": len(results)} + + +def _qual(grades: list[dict], weights: dict) -> tuple[float, dict]: + """(qual score, per-category detail). Evidence regresses medians toward 3.""" + full_credit = max(1, int(weights.get("evidenceFullCredit", 400))) + cat_max = float(weights.get("qualCategoryMax", 5)) + out: dict[str, dict] = {} + total = 0.0 + for cid in _CATEGORIES: + scores: list[int] = [] + equalities: list[float] = [] + rationales: list[dict] = [] + for g in grades or []: + cat = next((c for c in (g.get("categories") or []) if c.get("id") == cid), None) + if cat is None: + continue + evidence = cat.get("evidence") or [] + quote_chars = sum(min(len(ev.get("quote") or ""), 200) for ev in evidence) + scores.append(int(cat.get("score", 3))) + equalities.append(min(1.0, quote_chars / full_credit)) + rationales.append({ + "grader": g.get("grader") or "grader", + "rationale": cat.get("rationale") or "", + "evidence": [{"quote": ev.get("quote") or "", "location": ev.get("location") or ""} + for ev in evidence], + }) + if scores: + median = float(statistics.median(scores)) + e_mean = sum(equalities) / len(equalities) + else: + median, e_mean = 3.0, 0.0 + adjusted = 3.0 + (median - 3.0) * e_mean + points = adjusted * cat_max / 5.0 + total += points + out[cid] = { + "panel_scores": scores, + "median": round(median, 2), + "evidence_quality": round(e_mean, 4), + "adjusted": round(adjusted, 4), + "points": round(points, 4), + "rationales": rationales, + } + return total, out + + +def score_deck(extraction: dict, grades: list[dict], pinned_targets: list[dict], + prior_targets: list[dict], kpi_aliases: dict, weights: dict, + meta: dict) -> dict: + """Score one graded deck into the canonical ledger record. Pure.""" + kpis = extraction.get("kpis") or [] + floor = float(weights.get("kpiCreditFloor", 0.5)) + scoring_flags: list[dict] = [] + + # --- per-KPI target resolution + credit + kpi_results: list[dict] = [] + for k in kpis: + tgt, source, via = _resolve_target(k, pinned_targets, prior_targets, kpi_aliases) + credit = None + target = None + if tgt is not None and tgt.get("target") is not None: + target = float(tgt["target"]) + credit = _credit(float(k.get("actual", 0)), target, + tgt.get("direction") or k.get("direction") or "gte", floor) + kpi_results.append({ + "canonical_name": k.get("canonical_name"), "name": k.get("name"), + "actual": k.get("actual"), "unit": k.get("unit") or "", + "direction": k.get("direction"), "profitability": bool(k.get("profitability")), + "target": target, "target_source": source, "matched_via": via, + "credit": None if credit is None else round(credit, 4), + }) + + prof_all = [r for r in kpi_results if r["profitability"]] + prof_hit = [r for r in prof_all if r["credit"] is not None] + other_hit = [r for r in kpi_results if not r["profitability"] and r["credit"] is not None] + + # --- forecast integrity: this deck's actuals vs the prior deck's targets + forecast_results: list[dict] = [] + dropped: list[str] = [] + for pt in prior_targets or []: + cand, _via = match_kpi(pt.get("canonical_name") or "", kpis, kpi_aliases) + if cand is None: + dropped.append(pt.get("canonical_name") or pt.get("name") or "kpi") + continue + actual = float(cand.get("actual", 0)) + target = float(pt.get("target", 0)) + direction = pt.get("direction") or "gte" + if target == 0: + acc = 1.0 if _passes(actual, target, direction) else 0.0 + else: + err = (actual - target) / abs(target) + if direction == "lte": + err = -err + e = abs(err) if err < 0 else abs(err) / 2.0 # overshoot penalized half + acc = 1.0 - min(1.0, e) + forecast_results.append({ + "canonical_name": pt.get("canonical_name"), "target": target, + "actual": actual, "accuracy": round(acc, 4), + }) + + # Pinned targets that no reported actual matches count as dropped too. + for p in pinned_targets or []: + cand, _via = match_kpi(p.get("kpi") or "", kpis, kpi_aliases) + if cand is None: + dropped.append(p.get("kpi") or "kpi") + seen: set[str] = set() + dropped_unique = [d for d in dropped + if not (d.strip().lower() in seen or seen.add(d.strip().lower()))] + for name in dropped_unique[: int(weights.get("droppedKpiMax", 3))]: + scoring_flags.append({ + "code": "kpi_dropped", + "description": f"previously targeted KPI '{name}' is not reported this period", + "severity": int(weights.get("droppedKpiPenalty", 2)), + }) + + # --- quant buckets + renormalization (NA weight redistributes pro-rata) + prof = _bucket(prof_hit, float(weights.get("profitabilityKpi", 30))) + other = _bucket(other_hit, float(weights.get("otherKpi", 20))) + fmean = (sum(f["accuracy"] for f in forecast_results) / len(forecast_results) + if forecast_results else 0.0) + if forecast_results: + forecast = {"score": fmean, "weight": float(weights.get("forecastIntegrity", 10)), + "na": False, "kpi_count": len(forecast_results)} + else: + forecast = {"score": 0.0, "weight": 0.0, "na": True, "kpi_count": 0} + + if not prof_all: + prof["na"] = True + prof["weight"] = 0.0 + scoring_flags.append({ + "code": "no_profitability_visibility", + "description": "no profit/margin/cash KPI reported at all", + "severity": 3, + }) + + total_quant_w = (float(weights.get("profitabilityKpi", 30)) + + float(weights.get("otherKpi", 20)) + + float(weights.get("forecastIntegrity", 10))) + present = [b for b in (prof, other, forecast) if not b["na"]] + if present: + scale = total_quant_w / sum(b["weight"] for b in present) + for b in present: + b["weight"] = round(b["weight"] * scale, 4) + b["score"] = round(b["score"] * b["weight"], 4) + quant_score = sum(b["score"] for b in present) + all_quant_na = False + else: + quant_score = 0.0 + all_quant_na = True + scoring_flags.append({ + "code": "no_quantitative_kpis", + "description": "no quantitative bucket could be scored (no targeted KPIs, " + "no prior targets)", + "severity": 4, + }) + + # --- qualitative + qual_score, categories = _qual(grades, weights) + qual_max = 8.0 * float(weights.get("qualCategoryMax", 5)) + + # --- red flags: extractor + graders + scoring; dedup, damp single-source + damp = float(weights.get("singleSourceFlagFactor", 0.5)) + cap = float(weights.get("redFlagCap", 15)) + flag_map: dict[str, dict] = {} + + def add_flag(f: dict, source: str, scoring_flag: bool = False): + code = (f.get("code") or "flag").strip().lower() + key = f"{code}:{f.get('description', '')}" if scoring_flag and code == "kpi_dropped" else code + sev = int(f.get("severity", 1)) + cur = flag_map.get(key) + if cur is None: + flag_map[key] = {"code": code, "description": f.get("description") or "", + "severity": sev, "sources": {source}, "scoring": scoring_flag} + else: + if sev > cur["severity"]: + cur["severity"] = sev + cur["description"] = f.get("description") or cur["description"] + cur["sources"].add(source) + cur["scoring"] = cur["scoring"] or scoring_flag + + for f in extraction.get("red_flag_candidates") or []: + add_flag(f, "extractor") + for g in grades or []: + for f in g.get("red_flags") or []: + add_flag(f, g.get("grader") or "grader") + for f in scoring_flags: + add_flag(f, "scoring", scoring_flag=True) + + flags: list[dict] = [] + for f in flag_map.values(): + full = f["scoring"] or len(f["sources"]) >= 2 + points = float(f["severity"]) if full else float(f["severity"]) * damp + flags.append({"code": f["code"], "description": f["description"], + "severity": f["severity"], "points": round(points, 4), + "sources": sorted(f["sources"])}) + flags.sort(key=lambda f: (-f["points"], f["code"])) + penalty_total = round(min(cap, sum(f["points"] for f in flags)), 4) + + # --- composite + if all_quant_na: + base = (qual_score / qual_max * 100.0) if qual_max else 0.0 + else: + base = quant_score + qual_score + composite = round(max(0.0, min(100.0, base - penalty_total)), 1) + + return { + "schema_version": 1, + "company": meta.get("company"), + "period": meta.get("period"), + "deck_id": meta.get("deck_id"), + "job_id": meta.get("job_id"), + "graded_at": meta.get("graded_at"), + "composite": composite, + "quant": {"score": round(quant_score, 4), "profitability": prof, + "other": other, "forecast_integrity": forecast}, + "qual": {"score": round(qual_score, 4), "categories": categories}, + "penalties": {"total": penalty_total, "flags": flags}, + "kpi_results": kpi_results, + "forecast_results": forecast_results, + "panel": meta.get("panel") or [], + "artifacts": meta.get("artifacts", {}), + "narrative": extraction.get("narrative", {}), + } diff --git a/orchestrator/templates/index.html b/orchestrator/templates/index.html index c2fc48f..e935093 100644 --- a/orchestrator/templates/index.html +++ b/orchestrator/templates/index.html @@ -6,7 +6,7 @@ Boardroom Map @@ -48,16 +90,56 @@
+
+

Portfolio

+
Loading…
+
+ + +
-

Documents

-
Drop confidential documents here, or click to choose
- PDF · DOCX · TXT · MD
+

Drop decks

+
+ + + +
+
Drop board decks here, or click to choose
+ PDF · PPTX · DOCX · TXT · MD — period parsed from filename (2026-Q2, FY2026…)
- + - +
@@ -67,15 +149,15 @@
-

Panel

-
No reviewers configured.
+

Grading panel

+
No graders configured.

Serving / Job

—
- +
@@ -84,72 +166,350 @@

Activity log

loading…
- -
-

Latest report

- 
—
-
diff --git a/orchestrator/tests/fixtures/extraction_q1.json b/orchestrator/tests/fixtures/extraction_q1.json new file mode 100644 index 0000000..1c3d1d6 --- /dev/null +++ b/orchestrator/tests/fixtures/extraction_q1.json @@ -0,0 +1,29 @@ +{ + "schema_version": 1, + "deck": { + "company_hint": "Acme Robotics", + "period": "2026-Q1", + "meeting_date": "2026-04-15", + "title": "Acme Robotics — Q1 2026 Board Deck", + "truncated": false + }, + "kpis": [ + {"name": "ARR", "canonical_name": "arr", "actual": 10.0, "unit": "$M", "period": "2026-Q1", "direction": "gte", "profitability": false, "target_in_deck": 9.5, "source": "slide 3, financial summary", "notes": ""}, + {"name": "EBITDA Margin", "canonical_name": "ebitda_margin", "actual": -5.0, "unit": "%", "period": "2026-Q1", "direction": "gte", "profitability": true, "target_in_deck": -6.0, "source": "slide 4, P&L bridge", "notes": ""}, + {"name": "Logo Churn", "canonical_name": "churn_rate", "actual": 4.0, "unit": "%", "period": "2026-Q1", "direction": "lte", "profitability": false, "target_in_deck": 5.0, "source": "slide 5, retention", "notes": ""}, + {"name": "Cash Balance", "canonical_name": "cash_balance", "actual": 12.0, "unit": "$M", "period": "2026-Q1", "direction": "gte", "profitability": true, "target_in_deck": null, "source": "slide 4, balance sheet", "notes": ""} + ], + "forward_targets": [ + {"name": "ARR", "canonical_name": "arr", "target": 12.0, "unit": "$M", "target_period": "2026-Q2", "direction": "gte", "profitability": false, "source": "slide 9, guidance"}, + {"name": "Logo Churn", "canonical_name": "churn_rate", "target": 4.0, "unit": "%", "target_period": "2026-Q2", "direction": "lte", "profitability": false, "source": "slide 9, guidance"}, + {"name": "EBITDA Margin", "canonical_name": "ebitda_margin", "target": -2.0, "unit": "%", "target_period": "2026-Q2", "direction": "gte", "profitability": true, "source": "slide 9, guidance"}, + {"name": "Qualified Pipeline", "canonical_name": "qualified_pipeline", "target": 30.0, "unit": "$M", "target_period": "2026-Q2", "direction": "gte", "profitability": false, "source": "slide 10, pipeline build"} + ], + "red_flag_candidates": [ + {"code": "hockey_stick_forecast", "description": "H2 revenue ramp shown with no downside case or stated falsifiers", "severity": 3, "evidence": "slide 9 guidance chart"} + ], + "narrative": { + "summary": "Solid Q1: ARR beat plan at $10.0M, EBITDA margin improved to -5%, churn under plan. The H2 story rests entirely on the $30M qualified pipeline building as projected.", + "asks": ["Approve $2M expansion of the Austin integration facility"] + } +} diff --git a/orchestrator/tests/fixtures/extraction_q2.json b/orchestrator/tests/fixtures/extraction_q2.json new file mode 100644 index 0000000..ee97a9c --- /dev/null +++ b/orchestrator/tests/fixtures/extraction_q2.json @@ -0,0 +1,26 @@ +{ + "schema_version": 1, + "deck": { + "company_hint": "Acme Robotics", + "period": "2026-Q2", + "meeting_date": "2026-07-14", + "title": "Acme Robotics — Q2 2026 Board Deck", + "truncated": false + }, + "kpis": [ + {"name": "ARR", "canonical_name": "arr", "actual": 11.0, "unit": "$M", "period": "2026-Q2", "direction": "gte", "profitability": false, "target_in_deck": null, "source": "slide 3, financial summary", "notes": ""}, + {"name": "Logo Churn", "canonical_name": "churn_rate", "actual": 3.5, "unit": "%", "period": "2026-Q2", "direction": "lte", "profitability": false, "target_in_deck": null, "source": "slide 5, retention", "notes": ""}, + {"name": "EBITDA Margin", "canonical_name": "ebitda_margin", "actual": -3.0, "unit": "%", "period": "2026-Q2", "direction": "gte", "profitability": true, "target_in_deck": null, "source": "slide 4, P&L bridge", "notes": ""}, + {"name": "Cash Balance", "canonical_name": "cash_balance", "actual": 13.0, "unit": "$M", "period": "2026-Q2", "direction": "gte", "profitability": true, "target_in_deck": null, "source": "slide 4, balance sheet", "notes": ""} + ], + "forward_targets": [ + {"name": "ARR", "canonical_name": "arr", "target": 14.0, "unit": "$M", "target_period": "2026-Q3", "direction": "gte", "profitability": false, "source": "slide 9, guidance"} + ], + "red_flag_candidates": [ + {"code": "adjusted_metrics", "description": "EBITDA presented on an adjusted basis with no bridge to GAAP", "severity": 2, "evidence": "slide 4 footnote"} + ], + "narrative": { + "summary": "Mixed Q2: ARR missed guidance at $11.0M vs $12.0M, churn beat, margin improved but missed the -2% target. Pipeline metric no longer reported.", + "asks": ["Approve revised FY2026 hiring plan"] + } +} diff --git a/orchestrator/tests/fixtures/grade_a.json b/orchestrator/tests/fixtures/grade_a.json new file mode 100644 index 0000000..8da2fa0 --- /dev/null +++ b/orchestrator/tests/fixtures/grade_a.json @@ -0,0 +1,103 @@ +{ + "schema_version": 1, + "grader": "grader-a", + "categories": [ + { + "id": "A", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category A: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "B", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category B: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "C", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category C: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "D", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category D: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "E", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category E: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "F", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category F: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "G", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category G: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "H", + "score": 2, + "evidence": [ + { + "quote": "We ask the board to approve the revised hiring plan as presented; supporting detail is available from management upon request after the meeting, and we recommend approval without further discussion given the compressed agenda for this session.", + "location": "slide 11" + } + ], + "rationale": "Asks are listed without recommendations or the inversion of the decision." + } + ], + "red_flags": [ + { + "code": "governance_gap", + "description": "succession and incentive redesign get one bullet while product minutiae fill nine slides", + "severity": 2, + "evidence": "slides 12-20" + } + ], + "overall_comment": "Strong disclosure discipline overall; governance asks remain the weak spot." +} diff --git a/orchestrator/tests/fixtures/grade_b.json b/orchestrator/tests/fixtures/grade_b.json new file mode 100644 index 0000000..2d485f5 --- /dev/null +++ b/orchestrator/tests/fixtures/grade_b.json @@ -0,0 +1,98 @@ +{ + "schema_version": 1, + "grader": "grader-b", + "categories": [ + { + "id": "A", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category A: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "B", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category B: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "C", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category C: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "D", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category D: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "E", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category E: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "F", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category F: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "G", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category G: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "H", + "score": 3, + "evidence": [], + "rationale": "Asks are listed without recommendations or the inversion of the decision." + } + ], + "red_flags": [ + { + "code": "governance_gap", + "description": "board asks lack recommendations and inversion", + "severity": 3, + "evidence": "slide 11" + } + ], + "overall_comment": "Strong disclosure discipline overall; governance asks remain the weak spot." +} diff --git a/orchestrator/tests/fixtures/grade_c.json b/orchestrator/tests/fixtures/grade_c.json new file mode 100644 index 0000000..61db192 --- /dev/null +++ b/orchestrator/tests/fixtures/grade_c.json @@ -0,0 +1,96 @@ +{ + "schema_version": 1, + "grader": "grader-c", + "categories": [ + { + "id": "A", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category A: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "B", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category B: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "C", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category C: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "D", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category D: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "E", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category E: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "F", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category F: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "G", + "score": 4, + "evidence": [ + { + "quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.", + "location": "slide 7" + } + ], + "rationale": "Category G: specific, quantified disclosure with owner-aligned framing." + }, + { + "id": "H", + "score": 2, + "evidence": [ + { + "quote": "We ask the board to approve the revised hiring plan as presented; supporting detail is available from management upon request after the meeting, and we recommend approval without further discussion given the compressed agenda for this session.", + "location": "slide 11" + } + ], + "rationale": "Asks are listed without recommendations or the inversion of the decision." + } + ], + "red_flags": [], + "overall_comment": "Strong disclosure discipline overall; governance asks remain the weak spot." +} diff --git a/orchestrator/tests/test_decks.py b/orchestrator/tests/test_decks.py new file mode 100644 index 0000000..03bf69c --- /dev/null +++ b/orchestrator/tests/test_decks.py @@ -0,0 +1,118 @@ +"""Tests for decks.py: slugify, period parsing/sorting, inbox discovery.""" +import os +import sys +import tempfile +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +import decks + + +class TestSlugify(unittest.TestCase): + def test_basic(self): + self.assertEqual(decks.slugify("Acme Robotics"), "acme-robotics") + self.assertEqual(decks.slugify(" Acme, Inc. (US) "), "acme-inc-us") + self.assertEqual(decks.slugify("ALLCAPS"), "allcaps") + self.assertEqual(decks.slugify(""), "company") + self.assertEqual(decks.slugify("---"), "company") + + +class TestParsePeriod(unittest.TestCase): + def test_quarters(self): + self.assertEqual(decks.parse_period_from_name("acme_2026-Q2_board.pdf"), "2026-Q2") + self.assertEqual(decks.parse_period_from_name("2026Q4 deck.pptx"), "2026-Q4") + self.assertEqual(decks.parse_period_from_name("Q3 2025 update.pptx"), "2025-Q3") + self.assertEqual(decks.parse_period_from_name("q1_2024_board.docx"), "2024-Q1") + + def test_halves(self): + self.assertEqual(decks.parse_period_from_name("board-2026-H1.docx"), "2026-H1") + self.assertEqual(decks.parse_period_from_name("2025H2-review.pdf"), "2025-H2") + + def test_months(self): + self.assertEqual(decks.parse_period_from_name("acme 2026-05 board.pdf"), "2026-05") + self.assertEqual(decks.parse_period_from_name("2026_12_flash.txt"), "2026-12") + self.assertIsNone(decks.parse_period_from_name("2026-13 notes.pdf")) + self.assertIsNone(decks.parse_period_from_name("2026-00 notes.pdf")) + + def test_fiscal_year(self): + self.assertEqual(decks.parse_period_from_name("FY2025 review.pdf"), "FY2025") + self.assertEqual(decks.parse_period_from_name("fy-2024 plan.txt"), "FY2024") + self.assertEqual(decks.parse_period_from_name("FY 2026 budget.docx"), "FY2026") + + def test_no_period(self): + self.assertIsNone(decks.parse_period_from_name("notes.txt")) + self.assertIsNone(decks.parse_period_from_name("budget_2027.xlsx")) + self.assertIsNone(decks.parse_period_from_name("Q5 2026.pdf")) + + def test_quarter_wins_over_month(self): + # "2026-Q2" must not be misread; Q pattern is checked before YYYY-MM. + self.assertEqual(decks.parse_period_from_name("2026-Q2 and 2026-05.pdf"), "2026-Q2") + + def test_not_inside_digit_runs(self): + self.assertIsNone(decks.parse_period_from_name("doc-20261-05.pdf")) + + +class TestPeriodSortKey(unittest.TestCase): + def test_ordering_mixed_granularities(self): + ordered = ["FY2025", "2025-Q4", "2026-H1", "2026-Q1", "2026-01", + "2026-Q2", "2026-05", "2026-H2", "2026-Q4"] + self.assertEqual(sorted(ordered, key=decks.period_sort_key), ordered) + + def test_start_months(self): + self.assertEqual(decks.period_sort_key("2026-Q2")[:2], (2026, 4)) + self.assertEqual(decks.period_sort_key("2026-H2")[:2], (2026, 7)) + self.assertEqual(decks.period_sort_key("2026-11")[:2], (2026, 11)) + self.assertEqual(decks.period_sort_key("FY2026")[:2], (2026, 1)) + + def test_unknown_sorts_last(self): + keys = [decks.period_sort_key(p) for p in ("2026-Q4", None, "garbage", "FY2026")] + self.assertEqual(max(keys), decks.period_sort_key(None)) + self.assertEqual(decks.period_sort_key(None), decks.period_sort_key("garbage")) + self.assertGreater(decks.period_sort_key(None), decks.period_sort_key("2099-Q4")) + + +class TestDiscover(unittest.TestCase): + def _touch(self, *parts): + path = os.path.join(*parts) + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w") as f: + f.write("x") + + def test_discover(self): + with tempfile.TemporaryDirectory() as tmp: + inbox = os.path.join(tmp, "inbox") + self._touch(inbox, "Acme Robotics", "acme-2026-Q1.pdf") + self._touch(inbox, "Acme Robotics", "acme-2026-Q1-appendix.txt") + self._touch(inbox, "Acme Robotics", "acme-2026-Q2.pptx") + self._touch(inbox, "Acme Robotics", "chart-2026-Q2.png") + self._touch(inbox, "Acme Robotics", "notes.txt") + self._touch(inbox, "Acme Robotics", ".DS_Store") + self._touch(inbox, "beta-corp", "deck 2026-H1.docx") + self._touch(inbox, "stray.pdf") + + out = decks.discover(inbox) + self.assertEqual(out["skipped"], ["stray.pdf"]) + units = out["units"] + keys = [(u["company_slug"], u["period"], u["period_source"]) for u in units] + self.assertEqual(keys, [ + ("acme-robotics", "2026-Q1", "filename"), + ("acme-robotics", "2026-Q2", "filename"), + ("acme-robotics", None, "unknown"), + ("beta-corp", "2026-H1", "filename"), + ]) + q1 = units[0] + self.assertEqual([os.path.basename(f) for f in q1["files"]], + ["acme-2026-Q1-appendix.txt", "acme-2026-Q1.pdf"]) + self.assertTrue(all(os.path.isabs(f) for f in q1["files"])) + q2 = units[1] + self.assertEqual([os.path.basename(f) for f in q2["files"]], ["acme-2026-Q2.pptx"]) + self.assertEqual(q2["ignored"], ["chart-2026-Q2.png"]) + self.assertEqual([os.path.basename(f) for f in units[2]["files"]], ["notes.txt"]) + + def test_missing_inbox(self): + self.assertEqual(decks.discover("/nonexistent/inbox"), {"units": [], "skipped": []}) + + +if __name__ == "__main__": + unittest.main() diff --git a/orchestrator/tests/test_ledger.py b/orchestrator/tests/test_ledger.py new file mode 100644 index 0000000..5889a90 --- /dev/null +++ b/orchestrator/tests/test_ledger.py @@ -0,0 +1,116 @@ +"""Tests for ledger.py: company lifecycle, deck records, forward targets.""" +import glob +import os +import sys +import tempfile +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +import ledger as ledger_mod + + +def _record(deck_id, period, composite=70.0): + return {"schema_version": 1, "deck_id": deck_id, "period": period, + "composite": composite, "graded_at": "2026-07-01T00:00:00Z"} + + +def _ft(canonical, target, target_period, direction="gte"): + return {"name": canonical, "canonical_name": canonical, "target": target, + "unit": "", "target_period": target_period, "direction": direction, + "profitability": False, "source": "slide 9"} + + +class TestLedger(unittest.TestCase): + def setUp(self): + self._tmp = tempfile.TemporaryDirectory() + self.ledger = ledger_mod.Ledger(os.path.join(self._tmp.name, "ledger")) + + def tearDown(self): + self._tmp.cleanup() + + def test_ensure_company_auto_created(self): + c = self.ledger.ensure_company("acme") + self.assertTrue(c["auto_created"]) + self.assertEqual(c["name"], "acme") + c2 = self.ledger.ensure_company("acme", name="Acme Robotics") + self.assertEqual(c2["name"], "acme") # existing entry wins + named = self.ledger.ensure_company("beta", name="Beta Corp") + self.assertFalse(named["auto_created"]) + self.assertEqual(named["name"], "Beta Corp") + self.assertEqual(self.ledger.all_slugs(), ["acme", "beta"]) + self.assertEqual(len(self.ledger.all_companies()), 2) + + def test_merge_config_companies(self): + self.ledger.ensure_company("acme") + self.ledger.merge_config_companies([{ + "slug": "acme", "name": "Acme Robotics", + "kpiAliases": "arr=annual recurring revenue;run_rate_arr\nchurn_rate=logo_churn", + "pinnedTargets": [{"kpi": "cash_balance", "target": 12.0, "unit": "$M", + "direction": "gte", "profitability": True}], + }]) + c = self.ledger.get_company("acme") + self.assertFalse(c["auto_created"]) + self.assertEqual(c["name"], "Acme Robotics") + self.assertEqual(c["kpi_aliases"], + {"arr": ["annual recurring revenue", "run_rate_arr"], + "churn_rate": ["logo_churn"]}) + self.assertEqual(c["pinned_targets"][0]["kpi"], "cash_balance") + # slug derived from name when absent + self.ledger.merge_config_companies([{"name": "Beta Corp", "kpiAliases": "", + "pinnedTargets": []}]) + self.assertIsNotNone(self.ledger.get_company("beta-corp")) + + def test_record_supersede_prior_targets_roundtrip(self): + path = self.ledger.record_deck("acme", _record("2026-Q1", "2026-Q1", 82.7), + [_ft("arr", 12.0, "2026-Q2"), + _ft("arr", 15.0, "2026-Q3")]) + self.assertTrue(os.path.isfile(path)) + self.assertEqual( + [t["target"] for t in self.ledger.prior_targets("acme", "2026-Q2")], [12.0]) + self.assertEqual(self.ledger.prior_targets("acme", "2026-Q4"), []) + self.assertEqual(self.ledger.prior_targets("nobody", "2026-Q2"), []) + + # Re-grade the same deck: old record superseded (renamed), one live record. + self.ledger.record_deck("acme", _record("2026-Q1", "2026-Q1", 80.0), + [_ft("arr", 12.5, "2026-Q2")]) + ddir = os.path.dirname(path) + self.assertEqual(len(glob.glob(os.path.join(ddir, "*.superseded-*.json"))), 1) + live = self.ledger.deck_records("acme") + self.assertEqual(len(live), 1) + self.assertEqual(live[0]["composite"], 80.0) + self.assertEqual( + [t["target"] for t in self.ledger.prior_targets("acme", "2026-Q2")], [12.5]) + # history keeps one entry per period + c = self.ledger.get_company("acme") + self.assertEqual([h["period"] for h in c["history"]], ["2026-Q1"]) + self.assertEqual(c["history"][0]["composite"], 80.0) + + def test_newer_deck_replaces_targets_older_does_not(self): + self.ledger.record_deck("acme", _record("2026-Q1", "2026-Q1"), + [_ft("arr", 15.0, "2026-Q3")]) + self.ledger.record_deck("acme", _record("2026-Q2", "2026-Q2"), + [_ft("arr", 16.0, "2026-Q3"), + _ft("churn_rate", 3.0, "2026-Q3", "lte")]) + targets = self.ledger.prior_targets("acme", "2026-Q3") + self.assertEqual(sorted(t["target"] for t in targets), [3.0, 16.0]) + c = self.ledger.get_company("acme") + self.assertEqual(c["extracted_targets"]["2026-Q3"]["from_deck"], "2026-Q2") + # Re-recording the OLDER deck must not clobber the newer deck's targets. + self.ledger.record_deck("acme", _record("2026-Q1", "2026-Q1"), + [_ft("arr", 15.0, "2026-Q3")]) + targets = self.ledger.prior_targets("acme", "2026-Q3") + self.assertEqual(sorted(t["target"] for t in targets), [3.0, 16.0]) + # History is sorted oldest first. + c = self.ledger.get_company("acme") + self.assertEqual([h["period"] for h in c["history"]], ["2026-Q1", "2026-Q2"]) + + def test_deck_record_lookup(self): + self.ledger.record_deck("acme", _record("2026-Q1", "2026-Q1"), []) + self.assertEqual(self.ledger.deck_record("acme", "2026-Q1")["period"], "2026-Q1") + self.assertIsNone(self.ledger.deck_record("acme", "2026-Q9")) + self.assertEqual(self.ledger.deck_records("nobody"), []) + + +if __name__ == "__main__": + unittest.main() diff --git a/orchestrator/tests/test_scoring.py b/orchestrator/tests/test_scoring.py new file mode 100644 index 0000000..f069a01 --- /dev/null +++ b/orchestrator/tests/test_scoring.py @@ -0,0 +1,423 @@ +"""Tests for scoring.py (pure scorer), validate.py, and the fixture-driven +Q1 -> Q2 end-to-end flow through the ledger and scorecard renderers.""" +import json +import os +import sys +import tempfile +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +import ledger as ledger_mod +import scorecard +import scoring +import validate + +FIXTURES = os.path.join(os.path.dirname(os.path.abspath(__file__)), "fixtures") + +WEIGHTS = { + "profitabilityKpi": 30, "otherKpi": 20, "forecastIntegrity": 10, + "qualCategoryMax": 5, "redFlagCap": 15, "kpiCreditFloor": 0.5, + "droppedKpiPenalty": 2, "droppedKpiMax": 3, "evidenceFullCredit": 400, + "singleSourceFlagFactor": 0.5, +} + +PINNED_CASH = [{"kpi": "cash_balance", "target": 12.0, "unit": "$M", + "direction": "gte", "profitability": True}] + + +def _fixture(name): + with open(os.path.join(FIXTURES, name), encoding="utf-8") as f: + return json.load(f) + + +def _kpi(canonical, actual, direction="gte", prof=False, tid=None, name=None, unit=""): + return {"name": name or canonical, "canonical_name": canonical, "actual": actual, + "unit": unit, "period": None, "direction": direction, "profitability": prof, + "target_in_deck": tid, "source": "slide 1", "notes": ""} + + +def _ft(canonical, target, direction="gte", target_period="2026-Q2", prof=False): + return {"name": canonical, "canonical_name": canonical, "target": target, + "unit": "", "target_period": target_period, "direction": direction, + "profitability": prof, "source": "slide 9"} + + +def _extraction(kpis=None, forward=None, flags=None, period="2026-Q2"): + return {"schema_version": 1, "deck": {"period": period}, + "kpis": kpis or [], "forward_targets": forward or [], + "red_flag_candidates": flags or [], + "narrative": {"summary": "test deck", "asks": []}} + + +def _grade(grader="grader-a", score=3, quote_chars=0, red_flags=None, overrides=None): + cats = [] + for cid in "ABCDEFGH": + s, qc = score, quote_chars + if overrides and cid in overrides: + s, qc = overrides[cid] + ev = [{"quote": "q" * qc, "location": "slide 1"}] if qc else [] + cats.append({"id": cid, "score": s, "evidence": ev, "rationale": f"cat {cid}"}) + return {"schema_version": 1, "grader": grader, "categories": cats, + "red_flags": red_flags or [], "overall_comment": "ok"} + + +def _meta(period="2026-Q2", deck_id="d1"): + return {"company": "acme", "period": period, "deck_id": deck_id, "job_id": "job-1", + "graded_at": "2026-07-06T12:00:00Z", + "panel": [{"rid": "grader-a", "model": "grader-a", "valid": True}], + "artifacts": {"extraction": "extraction.json"}} + + +def _score(extraction, grades=None, pinned=None, prior=None, aliases=None, meta=None): + return scoring.score_deck(extraction, grades if grades is not None else [_grade()], + pinned or [], prior or [], aliases or {}, WEIGHTS, + meta or _meta()) + + +def _flag(rec, code): + return [f for f in rec["penalties"]["flags"] if f["code"] == code] + + +class TestMatchKpi(unittest.TestCase): + def test_exact(self): + cand, via = scoring.match_kpi("arr", [{"canonical_name": "arr", "name": "ARR"}], {}) + self.assertEqual(via, "exact") + self.assertEqual(cand["name"], "ARR") + + def test_alias_forward_and_reverse(self): + aliases = {"arr": ["Annual Recurring Revenue", "run_rate_arr"]} + cand, via = scoring.match_kpi( + "arr", [{"canonical_name": "revenue_annualized", + "name": "Annual Recurring Revenue"}], aliases) + self.assertEqual(via, "alias") + cand, via = scoring.match_kpi( + "run_rate_arr", [{"canonical_name": "arr", "name": "ARR"}], aliases) + self.assertEqual(via, "alias") + + def test_fuzzy(self): + cand, via = scoring.match_kpi( + "ebitda_margin", [{"canonical_name": "ebitda_margins", "name": "x"}], {}) + self.assertEqual(via, "fuzzy") + + def test_no_match(self): + self.assertEqual( + scoring.match_kpi("arr", [{"canonical_name": "cash_balance", "name": "Cash"}], {}), + (None, None)) + self.assertEqual(scoring.match_kpi("", [{"canonical_name": "arr"}], {}), (None, None)) + + +class TestCredit(unittest.TestCase): + def test_lte_credit(self): + rec = _score(_extraction([_kpi("churn_rate", 6.0, "lte", tid=5.0)])) + self.assertAlmostEqual(rec["kpi_results"][0]["credit"], 0.6667, places=4) + rec = _score(_extraction([_kpi("churn_rate", 4.0, "lte", tid=5.0)])) + self.assertEqual(rec["kpi_results"][0]["credit"], 1.0) + + def test_floor(self): + rec = _score(_extraction([_kpi("arr", 4.0, tid=10.0)])) # r=0.4 < floor + self.assertEqual(rec["kpi_results"][0]["credit"], 0.0) + rec = _score(_extraction([_kpi("arr", 7.5, tid=10.0)])) # r=0.75 -> 0.5 + self.assertAlmostEqual(rec["kpi_results"][0]["credit"], 0.5, places=4) + + def test_guards(self): + self.assertEqual(scoring._credit(5, 0, "gte", 0.5), 1.0) # zero target, passes + self.assertEqual(scoring._credit(-5, 0, "gte", 0.5), 0.0) # zero target, fails + self.assertEqual(scoring._credit(-1, 1, "gte", 0.5), 0.0) # sign mismatch, fails + self.assertEqual(scoring._credit(1, -1, "gte", 0.5), 1.0) # sign mismatch, passes + self.assertEqual(scoring._credit(0, 5, "lte", 0.5), 1.0) # lte zero actual + + def test_negative_targets(self): + # EBITDA margin: target -2, actual -3 -> two thirds of the way -> 0.3333 + self.assertAlmostEqual(scoring._credit(-3, -2, "gte", 0.5), 1 / 3, places=4) + self.assertEqual(scoring._credit(-1, -2, "gte", 0.5), 1.0) + + +class TestQuantBuckets(unittest.TestCase): + def test_first_deck_renormalization(self): + # No prior targets -> forecast NA -> its 10 points redistribute 36/24. + rec = _score(_extraction([_kpi("ebitda_margin", 5.0, prof=True, tid=5.0), + _kpi("arr", 10.0, tid=10.0)])) + q = rec["quant"] + self.assertTrue(q["forecast_integrity"]["na"]) + self.assertAlmostEqual(q["profitability"]["weight"], 36.0) + self.assertAlmostEqual(q["profitability"]["score"], 36.0) + self.assertAlmostEqual(q["other"]["weight"], 24.0) + self.assertAlmostEqual(q["other"]["score"], 24.0) + self.assertAlmostEqual(q["score"], 60.0) + self.assertEqual(rec["penalties"]["flags"], []) + self.assertAlmostEqual(rec["composite"], 84.0) # 60 quant + 24 qual (all 3s) + + def test_forecast_integrity_second_deck(self): + prior = [_ft("arr", 12.0), _ft("churn_rate", 4.0, "lte")] + rec = _score(_extraction([_kpi("arr", 11.0), _kpi("churn_rate", 3.5, "lte"), + _kpi("fcf", 1.0, prof=True, tid=1.0)]), + prior=prior) + fi = rec["quant"]["forecast_integrity"] + self.assertFalse(fi["na"]) + self.assertEqual(fi["weight"], 10.0) + self.assertEqual(fi["kpi_count"], 2) + accs = {f["canonical_name"]: f["accuracy"] for f in rec["forecast_results"]} + self.assertAlmostEqual(accs["arr"], 0.9167, places=4) # 1/12 undershoot + self.assertAlmostEqual(accs["churn_rate"], 0.9375, places=4) # overshoot halved + self.assertAlmostEqual(fi["score"], (0.9167 + 0.9375) / 2 * 10, places=3) + + def test_no_profitability_flag_and_redistribution(self): + rec = _score(_extraction([_kpi("arr", 10.0, tid=10.0)])) + q = rec["quant"] + self.assertTrue(q["profitability"]["na"]) + self.assertTrue(q["forecast_integrity"]["na"]) + self.assertAlmostEqual(q["other"]["weight"], 60.0) + self.assertAlmostEqual(q["score"], 60.0) + flags = _flag(rec, "no_profitability_visibility") + self.assertEqual(len(flags), 1) + self.assertEqual(flags[0]["points"], 3.0) # scoring flags never damped + self.assertEqual(flags[0]["sources"], ["scoring"]) + + def test_profitability_kpis_without_targets_na_no_flag(self): + rec = _score(_extraction([_kpi("ebitda_margin", -5.0, prof=True), + _kpi("arr", 10.0, tid=10.0)])) + self.assertTrue(rec["quant"]["profitability"]["na"]) + self.assertEqual(_flag(rec, "no_profitability_visibility"), []) + + def test_all_quant_na_scales_qual(self): + rec = _score(_extraction([])) + # qual 24 (all 3s) scaled to 60, minus no_profitability(3) + no_quantitative(4) + self.assertTrue(all(rec["quant"][b]["na"] for b in + ("profitability", "other", "forecast_integrity"))) + self.assertEqual(len(_flag(rec, "no_quantitative_kpis")), 1) + self.assertAlmostEqual(rec["composite"], 53.0) + + +class TestTargetPrecedence(unittest.TestCase): + def test_pinned_beats_extracted_beats_in_deck(self): + kpis = [_kpi("arr", 11.0, tid=9.0)] + pinned = [{"kpi": "arr", "target": 10.0, "unit": "$M", + "direction": "gte", "profitability": False}] + prior = [_ft("arr", 12.0)] + r = _score(_extraction(kpis), pinned=pinned, prior=prior)["kpi_results"][0] + self.assertEqual((r["target"], r["target_source"], r["matched_via"]), + (10.0, "pinned", "exact")) + self.assertEqual(r["credit"], 1.0) + r = _score(_extraction(kpis), prior=prior)["kpi_results"][0] + self.assertEqual((r["target"], r["target_source"]), (12.0, "extracted")) + self.assertAlmostEqual(r["credit"], 0.8333, places=4) + r = _score(_extraction(kpis))["kpi_results"][0] + self.assertEqual((r["target"], r["target_source"], r["matched_via"]), + (9.0, "in_deck", None)) + + def test_untargeted_kpi_reported_with_none(self): + r = _score(_extraction([_kpi("nps", 40.0)]))["kpi_results"][0] + self.assertIsNone(r["target"]) + self.assertIsNone(r["credit"]) + self.assertIsNone(r["target_source"]) + + +class TestQualitative(unittest.TestCase): + def test_evidence_regression_both_directions(self): + # Median 5 with no quotes regresses to 3; so does median 1. + rec = _score(_extraction([]), grades=[_grade(score=5, quote_chars=0)]) + self.assertEqual(rec["qual"]["categories"]["A"]["adjusted"], 3.0) + rec = _score(_extraction([]), grades=[_grade(score=1, quote_chars=0)]) + self.assertEqual(rec["qual"]["categories"]["A"]["adjusted"], 3.0) + self.assertAlmostEqual(rec["qual"]["score"], 24.0) + + def test_full_evidence_keeps_extreme_scores(self): + # Per-quote chars cap at 200, so full credit (400) needs two quotes. + g = _grade(score=5, quote_chars=200) + for cat in g["categories"]: + cat["evidence"].append({"quote": "q" * 200, "location": "slide 2"}) + rec = _score(_extraction([]), grades=[g]) + cat = rec["qual"]["categories"]["A"] + self.assertEqual(cat["evidence_quality"], 1.0) + self.assertEqual(cat["adjusted"], 5.0) + self.assertEqual(cat["points"], 5.0) + + def test_quote_chars_capped_at_200_each(self): + # One 1000-char quote counts as 200 -> e = 0.5 -> adjusted 4. + rec = _score(_extraction([]), grades=[_grade(score=5, quote_chars=1000)]) + self.assertEqual(rec["qual"]["categories"]["A"]["evidence_quality"], 0.5) + self.assertEqual(rec["qual"]["categories"]["A"]["adjusted"], 4.0) + + def test_panel_median_and_rationales(self): + grades = [_grade("g1", score=4, quote_chars=400), + _grade("g2", score=4, quote_chars=400), + _grade("g3", score=2, quote_chars=400)] + rec = _score(_extraction([]), grades=grades) + cat = rec["qual"]["categories"]["B"] + self.assertEqual(cat["panel_scores"], [4, 4, 2]) + self.assertEqual(cat["median"], 4.0) + self.assertEqual(len(cat["rationales"]), 3) + self.assertEqual(cat["rationales"][0]["grader"], "g1") + + +class TestPenalties(unittest.TestCase): + def test_single_source_damping(self): + rec = _score(_extraction([], flags=[{"code": "adjusted_metrics", + "description": "d", "severity": 4}])) + f = _flag(rec, "adjusted_metrics")[0] + self.assertEqual(f["points"], 2.0) + self.assertEqual(f["sources"], ["extractor"]) + + def test_two_sources_full_severity_max_wins(self): + grades = [_grade("g1", red_flags=[{"code": "governance_gap", + "description": "weak", "severity": 2}]), + _grade("g2", red_flags=[{"code": "governance_gap", + "description": "worse", "severity": 3}])] + rec = _score(_extraction([]), grades=grades) + f = _flag(rec, "governance_gap")[0] + self.assertEqual(f["severity"], 3) + self.assertEqual(f["points"], 3.0) + self.assertEqual(f["sources"], ["g1", "g2"]) + + def test_penalty_cap(self): + codes = ["related_party", "channel_stuffing_risk", "suppressed_dissent", + "metric_redefinition"] + flags = [{"code": c, "description": c, "severity": 5} for c in codes] + rec = _score(_extraction([_kpi("fcf", 1.0, prof=True, tid=1.0)], flags=flags), + grades=[_grade("g1", red_flags=flags)]) + self.assertEqual(rec["penalties"]["total"], 15.0) # 4x5=20 capped + + def test_dropped_kpi_flags_capped(self): + prior = [_ft(c, 1.0) for c in ("alpha_metric", "beta_metric", "gamma_metric", + "delta_metric", "epsilon_metric")] + rec = _score(_extraction([]), prior=prior) + dropped = _flag(rec, "kpi_dropped") + self.assertEqual(len(dropped), 3) # droppedKpiMax + for f in dropped: + self.assertEqual(f["points"], 2.0) # droppedKpiPenalty, never damped + + +class TestValidate(unittest.TestCase): + def test_parse_json_text(self): + self.assertEqual(validate.parse_json_text('{"a": 1}'), {"a": 1}) + salvaged = validate.parse_json_text( + 'Sure! Here is the JSON:\n```json\n{"a": {"b": "}"}}\n```\ntrailing prose') + self.assertEqual(salvaged, {"a": {"b": "}"}}) + self.assertIsNone(validate.parse_json_text("no json here")) + self.assertIsNone(validate.parse_json_text("[1, 2, 3]")) + self.assertIsNone(validate.parse_json_text("")) + + def test_schemas_load_and_fixtures_validate(self): + self.assertIn("properties", validate.load_schema("extraction")) + self.assertIn("properties", validate.load_schema("grades")) + for name, schema in (("extraction_q1.json", "extraction"), + ("extraction_q2.json", "extraction"), + ("grade_a.json", "grades"), ("grade_b.json", "grades"), + ("grade_c.json", "grades")): + err = validate.validate_obj(_fixture(name), schema) + self.assertIsNone(err, f"{name}: {err}") + + def test_validate_obj_rejects_bad(self): + self.assertIsNotNone(validate.validate_obj({"schema_version": 1}, "grades")) + + def test_validate_file(self): + obj, err = validate.validate_file(os.path.join(FIXTURES, "grade_a.json"), "grades") + self.assertIsNone(err) + self.assertEqual(obj["grader"], "grader-a") + obj, err = validate.validate_file("/nonexistent.json", "grades") + self.assertIsNone(obj) + self.assertIsNotNone(err) + + +class TestEndToEnd(unittest.TestCase): + """Fixture-driven Q1 -> Q2 flow: score, ledger round-trip, rendering.""" + + def setUp(self): + self._tmp = tempfile.TemporaryDirectory() + self.ledger = ledger_mod.Ledger(os.path.join(self._tmp.name, "ledger")) + self.grades = [_fixture("grade_a.json"), _fixture("grade_b.json"), + _fixture("grade_c.json")] + self.q1 = _fixture("extraction_q1.json") + self.q2 = _fixture("extraction_q2.json") + + def tearDown(self): + self._tmp.cleanup() + + def _score_q1(self): + return scoring.score_deck(self.q1, self.grades, PINNED_CASH, [], {}, WEIGHTS, + _meta("2026-Q1", "2026-Q1")) + + def test_q1_first_deck(self): + rec = self._score_q1() + q = rec["quant"] + self.assertTrue(q["forecast_integrity"]["na"]) + self.assertAlmostEqual(q["score"], 60.0) # every KPI at/above target + # qual: A-G 3.5 pts each (median 4, evidence 0.5), H 2.6667 + self.assertAlmostEqual(rec["qual"]["score"], 27.1667, places=3) + self.assertAlmostEqual(rec["qual"]["categories"]["H"]["points"], 2.6667, places=3) + # hockey_stick (extractor only, sev 3 -> 1.5) + governance_gap (2 graders -> 3) + self.assertAlmostEqual(rec["penalties"]["total"], 4.5) + self.assertAlmostEqual(rec["composite"], 82.7) + cash = next(k for k in rec["kpi_results"] if k["canonical_name"] == "cash_balance") + self.assertEqual(cash["target_source"], "pinned") + + def test_q2_against_q1_targets(self): + rec1 = self._score_q1() + self.ledger.record_deck("acme", rec1, self.q1["forward_targets"]) + prior = self.ledger.prior_targets("acme", "2026-Q2") + self.assertEqual(len(prior), 4) + + rec2 = scoring.score_deck(self.q2, self.grades, PINNED_CASH, prior, {}, WEIGHTS, + _meta("2026-Q2", "2026-Q2")) + by_name = {k["canonical_name"]: k for k in rec2["kpi_results"]} + self.assertAlmostEqual(by_name["arr"]["credit"], 0.8333, places=4) + self.assertEqual(by_name["churn_rate"]["credit"], 1.0) + self.assertAlmostEqual(by_name["ebitda_margin"]["credit"], 0.3333, places=4) + self.assertEqual(by_name["cash_balance"]["target_source"], "pinned") + self.assertEqual(by_name["cash_balance"]["credit"], 1.0) + + q = rec2["quant"] + self.assertAlmostEqual(q["profitability"]["score"], 20.0, places=2) + self.assertAlmostEqual(q["other"]["score"], 18.333, places=2) + self.assertAlmostEqual(q["forecast_integrity"]["score"], 7.847, places=2) + self.assertEqual(len(rec2["forecast_results"]), 3) + + # qualified_pipeline guided in Q1 but not reported in Q2 -> dropped flag + dropped = _flag(rec2, "kpi_dropped") + self.assertEqual(len(dropped), 1) + self.assertIn("qualified_pipeline", dropped[0]["description"]) + # adjusted_metrics 1.0 + governance_gap 3.0 + kpi_dropped 2.0 + self.assertAlmostEqual(rec2["penalties"]["total"], 6.0) + self.assertAlmostEqual(rec2["composite"], 67.3) + self.assertAlmostEqual( + rec2["composite"], + round(q["score"] + rec2["qual"]["score"] - rec2["penalties"]["total"], 1)) + + # ledger round-trip + rendering + self.ledger.record_deck("acme", rec2, self.q2["forward_targets"]) + records = self.ledger.deck_records("acme") + self.assertEqual([r["period"] for r in records], ["2026-Q1", "2026-Q2"]) + + report = scorecard.render_deck_report(rec2, self.q2, adjudication_md="Chair memo.") + self.assertIn("67.3", report) + self.assertIn("pinned", report) + self.assertIn("## Panel adjudication", report) + self.assertIn("Chair memo.", report) + self.assertIn("kpi_dropped", report) + + card = scorecard.render_scorecard(self.ledger.get_company("acme"), records) + self.assertIn("2026-Q1", card) + self.assertIn("2026-Q2", card) + self.assertIn("↓", card) # composite fell Q1 -> Q2 + self.assertIn("KPI hit-rate", card) + self.assertIn("arr", card) + + def test_meta_passthrough_and_record_shape(self): + rec = self._score_q1() + self.assertEqual(rec["company"], "acme") + self.assertEqual(rec["deck_id"], "2026-Q1") + self.assertEqual(rec["job_id"], "job-1") + self.assertEqual(rec["panel"][0]["rid"], "grader-a") + self.assertEqual(rec["artifacts"], {"extraction": "extraction.json"}) + self.assertEqual(rec["schema_version"], 1) + self.assertIn("summary", rec["narrative"]) + for key in ("composite", "quant", "qual", "penalties", "kpi_results", + "forecast_results"): + self.assertIn(key, rec) + # the record must be JSON-serializable as produced + json.dumps(rec) + + +if __name__ == "__main__": + unittest.main() diff --git a/orchestrator/validate.py b/orchestrator/validate.py new file mode 100644 index 0000000..e4c0db5 --- /dev/null +++ b/orchestrator/validate.py @@ -0,0 +1,89 @@ +"""JSON parsing + schema validation for extractor/grader outputs. + +Local models occasionally wrap their JSON in prose or fences; parse_json_text +salvages the first brace-balanced top-level object before we give up. Schemas +live in orchestrator/schemas/ and are the single contract between the sandbox +agents and the deterministic scorer. +""" +from __future__ import annotations + +import json +import os + +SCHEMAS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "schemas") +_cache: dict[str, dict] = {} + + +def load_schema(name: str) -> dict: + """Load "extraction" or "grades" schema (cached).""" + if name not in _cache: + path = os.path.join(SCHEMAS_DIR, f"{name}.schema.json") + with open(path, encoding="utf-8") as f: + _cache[name] = json.load(f) + return _cache[name] + + +def parse_json_text(text: str) -> dict | None: + """Parse `text` as a JSON object; salvage the first balanced {...} block.""" + if not text: + return None + try: + obj = json.loads(text) + return obj if isinstance(obj, dict) else None + except Exception: + pass + start = text.find("{") + while start != -1: + depth = 0 + in_str = False + esc = False + for i in range(start, len(text)): + c = text[i] + if esc: + esc = False + elif in_str: + if c == "\\": + esc = True + elif c == '"': + in_str = False + elif c == '"': + in_str = True + elif c == "{": + depth += 1 + elif c == "}": + depth -= 1 + if depth == 0: + try: + obj = json.loads(text[start:i + 1]) + if isinstance(obj, dict): + return obj + except Exception: + pass + break + start = text.find("{", start + 1) + return None + + +def validate_obj(obj, schema_name: str) -> str | None: + """Validate against the named schema; error message or None if valid.""" + import jsonschema + + try: + jsonschema.validate(obj, load_schema(schema_name)) + return None + except jsonschema.ValidationError as e: + path = ".".join(str(p) for p in e.absolute_path) or "(root)" + return f"{path}: {e.message}"[:500] + + +def validate_file(path: str, schema_name: str) -> tuple[dict | None, str | None]: + """Read + parse (with salvage) + validate a JSON file -> (obj, error).""" + try: + with open(path, encoding="utf-8", errors="replace") as f: + text = f.read() + except Exception as e: + return None, f"read failed: {e}" + obj = parse_json_text(text) + if obj is None: + return None, "no parseable JSON object found" + return obj, validate_obj(obj, schema_name) diff --git a/package.json b/package.json index 7003a9f..1c0b568 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "boardroom-map-startos", "version": "0.1.0", - "description": "StartOS service: drop confidential documents in and have a panel of local LLMs on your DGX Sparks review them — fully air-gappable, no frontier oversight", + "description": "StartOS service: grade portfolio-company board decks with a panel of local LLMs on your DGX Sparks — BDEF v1.1 scoring, per-company running scorecards, fully air-gappable", "scripts": { "build": "rm -rf ./javascript && ncc build startos/index.ts -o ./javascript", "check": "tsc --noEmit", diff --git a/sandbox/build.sh b/sandbox/build.sh index a9dd77d..d79ac0a 100644 --- a/sandbox/build.sh +++ b/sandbox/build.sh @@ -1,17 +1,17 @@ #!/usr/bin/env bash -# Build the Boardroom Map reviewer image ON THE HEAD SPARK (aarch64/GB10). +# Build the Boardroom Map grader image ON THE HEAD SPARK (aarch64/GB10). # The orchestrator does this automatically over SSH, but you can also run it by # hand: copy this sandbox/ directory to the Spark and run: # # IMAGE=boardroom-grader:latest bash build.sh # -# The tag must match the service's "Reviewer Image Tag" (Configure Sparks). +# The tag must match the service's "Grader Image Tag" (Configure Sparks). set -euo pipefail IMAGE="${IMAGE:-boardroom-grader:latest}" DIR="$(cd "$(dirname "$0")" && pwd)" echo ">> Building $IMAGE from $DIR" -docker build -t "$IMAGE" -f "$DIR/reviewer.Dockerfile" "$DIR" +docker build -t "$IMAGE" -f "$DIR/grader.Dockerfile" "$DIR" echo ">> Done. Image: $IMAGE" docker image inspect "$IMAGE" >/dev/null && echo ">> OK" diff --git a/sandbox/grader.Dockerfile b/sandbox/grader.Dockerfile index 3522874..3cbb6bc 100644 --- a/sandbox/grader.Dockerfile +++ b/sandbox/grader.Dockerfile @@ -1,15 +1,17 @@ -# Boardroom Map reviewer image — BUILT ON THE HEAD SPARK (aarch64), not packed into the +# Boardroom Map grader image — BUILT ON THE HEAD SPARK (aarch64), not packed into the # s9pk (the orchestrator ships this build context and builds it on the Spark; see -# reviewers.ensure_reviewer_image). +# graders.ensure_grader_image). # -# One-shot, read-only document reviewer (grader_agent.py) speaking the -# OpenAI-compatible API directly — a lean pure-Python image that builds fast. +# One-shot, read-only role agent (grader_agent.py; BM_ROLE = extractor | grader | +# adjudicator) speaking the OpenAI-compatible API directly — a lean pure-Python +# image that builds fast. # # At RUN time the orchestrator launches this HARDENED (non-root, --cap-drop ALL, # --security-opt no-new-privileges, read-only rootfs, no docker socket, only -# /docs (ro), /persona (ro), /RUBRIC.md (ro) and /out (rw) mounted, cpu/mem/pid -# caps) and attached to the per-job network. In air-gapped mode that network is -# --internal, so the container can reach ONLY the on-Spark model proxy. +# /docs (ro), /BDEF.md (ro), /schema.json (ro), /persona (ro) and /out (rw) +# mounted — the adjudicator instead gets /grades (ro) + /extraction.json (ro) — +# cpu/mem/pid caps) and attached to the per-job network. In air-gapped mode that +# network is --internal, so the container can reach ONLY the on-Spark model proxy. FROM python:3.11-slim RUN apt-get update && apt-get install -y --no-install-recommends \ diff --git a/sandbox/grader_agent.py b/sandbox/grader_agent.py index 5aab5bd..d7cf407 100644 --- a/sandbox/grader_agent.py +++ b/sandbox/grader_agent.py @@ -1,43 +1,56 @@ -"""Boardroom Map reviewer — a sandboxed, ONE-SHOT agent that reads confidential -documents through a LOCAL model and writes a single report, then exits. +"""Boardroom Map role agent — a sandboxed, ONE-SHOT, single-completion agent. -Unlike a swarm worker, this never loops forever and never writes to a shared -workspace. It mounts the documents read-only at /docs, runs one model (through the -on-Spark proxy) under its PERSONA + the shared RUBRIC, and writes exactly one -file to /out: - role=reviewer -> /out/.md - role=synthesizer -> /out/CONSOLIDATED_REPORT.md (also reads /reports) +One container = one role = one model call (plus a bounded reliability ladder / +repair round-trip). No tools, no loops, no shared writable workspace. The +orchestrator (orchestrator/graders.py + adjudicator.py) launches this hardened +(non-root, read-only rootfs, per-job network) with BM_ROLE set to: -It speaks the OpenAI-compatible /v1/chat/completions API directly (no Claude CLI, -no Anthropic translation — small local models handle this far better). The whole -document set is pre-loaded into the prompt up to a budget; for anything larger the -model can pull more with read_file. Native tool_calls are used when available, -with a JSON-action text fallback for models without a vLLM tool parser. + extractor reads /docs (ro) + the BDEF red-flag taxonomy, emits STRUCTURED + JSON per /schema.json (extraction schema) -> /out/extraction.json + grader reads /docs (ro) + the full /BDEF.md rubric, scores categories + A-H per /schema.json (grades schema) -> /out/.json + adjudicator reads /extraction.json (ro) + the panel's /grades/*.json (ro), + writes a MARKDOWN adjudication (no scores) -> /out/ADJUDICATION.md -In air-gapped mode the container is on an --internal Docker network: the only -thing reachable is the model proxy. web_search is offered ONLY when BM_SEARXNG_URL -is set (local-services mode). +Env (set by the orchestrator): + BM_ROLE, BM_GRADER_ID, BM_GRADER_NAME, BM_MODEL, + BM_LLM_BASE (http://boardroom-proxy:4000/v1), BM_LLM_KEY, + BM_TEMPERATURE (extractor forced to 0.0), BM_MAX_MODEL_LEN + +Mounts: /docs (ro), /BDEF.md (ro), /schema.json (ro; role-appropriate), +/persona/PERSONA.md (ro, optional), /out (rw); adjudicator additionally +/grades (ro) and /extraction.json (ro). + +JSON reliability ladder (extractor + grader): + 1. response_format = {"type":"json_schema", ..., "strict": true} + 2. on HTTP 4xx: retry with top-level {"guided_json": } (vLLM ext.) + 3. on another 4xx: retry plain +Parse whole-reply JSON, else the first brace-balanced {...} block. If invalid, +ONE repair round-trip; if still bad, write the raw text to the output path plus +a sibling .invalid marker containing the error. Full jsonschema +validation runs orchestrator-side; only lightweight structural checks here. + +Air-gapped: the per-job Docker network is --internal, so the only reachable +endpoint is the model proxy. Pure Python stdlib (urllib) — no pip deps. """ from __future__ import annotations import json import os -import re import ssl import time import traceback -import urllib.parse +import urllib.error import urllib.request -_SSL_CTX = ssl._create_unverified_context() # LAN self-signed (SearXNG) +_SSL_CTX = ssl._create_unverified_context() # LAN self-signed certs -RID = os.environ.get("BM_REVIEWER_ID", "reviewer") -NAME = os.environ.get("BM_REVIEWER_NAME", RID) -ROLE = os.environ.get("BM_ROLE", "reviewer") # reviewer | synthesizer -MODEL = os.environ.get("BM_MODEL", "reviewer-a") +ROLE = os.environ.get("BM_ROLE", "grader") # extractor | grader | adjudicator +GID = os.environ.get("BM_GRADER_ID", ROLE) +NAME = os.environ.get("BM_GRADER_NAME", GID) +MODEL = os.environ.get("BM_MODEL", "grader-a") LLM_BASE = os.environ.get("BM_LLM_BASE", "http://boardroom-proxy:4000/v1").rstrip("/") LLM_KEY = os.environ.get("BM_LLM_KEY", "sk-local") -SEARXNG_URL = os.environ.get("BM_SEARXNG_URL", "").rstrip("/") try: TEMPERATURE = float(os.environ.get("BM_TEMPERATURE", "") or "0.3") except ValueError: @@ -48,156 +61,44 @@ except ValueError: MAX_MODEL_LEN = 32768 DOCS = "/docs" -REPORTS = "/reports" -OUT_DIR = "/out" +BDEF_PATH = "/BDEF.md" +SCHEMA_PATH = "/schema.json" PERSONA_PATH = "/persona/PERSONA.md" -RUBRIC_PATH = "/RUBRIC.md" +GRADES_DIR = "/grades" +EXTRACTION_PATH = "/extraction.json" +OUT_DIR = "/out" -# Leave headroom for the system/rubric/persona + the model's output; spend the -# rest on document text (~3 chars/token is a safe rough estimate). -DOC_BUDGET = max(8000, (MAX_MODEL_LEN - 3500) * 3) -MAX_STEPS = 8 -MAX_OUTPUT_TOKENS = 2048 +MAX_OUTPUT_TOKENS = 3072 +TRANSPORT_RETRIES = 4 + + +def log(msg: str) -> None: + print(f"[{GID}] {msg}", flush=True) # ---------------------------------------------------------------- io helpers -def read_text(path: str, limit: int = 1_000_000) -> str: +def read_text(path: str, limit: int = 2_000_000) -> str: try: with open(path, errors="replace") as f: return f.read()[:limit] - except FileNotFoundError: + except (FileNotFoundError, IsADirectoryError): return "" -def _roots() -> list[str]: - return [DOCS, REPORTS] if ROLE == "synthesizer" else [DOCS] - - -def _safe(path: str) -> str: - """Resolve a path inside an allowed read root; refuse escapes.""" - cand = path or "." - for root in _roots(): - p = os.path.realpath(os.path.join(root, cand) if not os.path.isabs(cand) else cand) - if p == root or p.startswith(root + os.sep): - return p - raise ValueError(f"path outside allowed roots: {path}") - - -def list_dir(root: str) -> list[str]: - out = [] - if not os.path.isdir(root): - return out - for r, _dirs, files in os.walk(root): - for fn in files: - out.append(os.path.relpath(os.path.join(r, fn), root)) - return sorted(out) - - -# ---------------------------------------------------------------- tools -def tool_list_files(args: dict) -> str: - lines = [] - for root in _roots(): - names = list_dir(root) - if names: - lines.append(f"{root}:") - lines += [f" {n}" for n in names] - return "\n".join(lines) or "(no files)" - - -def tool_read_file(args: dict) -> str: - p = _safe(args["path"]) - try: - with open(p, errors="replace") as f: - return f.read()[:20000] - except FileNotFoundError: - return f"(no such file: {args['path']})" - except IsADirectoryError: - return f"(is a directory: {args['path']})" - - -def tool_web_search(args: dict) -> str: - if not SEARXNG_URL: - return "web search unavailable" - q = urllib.parse.urlencode({"q": args.get("query", ""), "format": "json"}) - try: - req = urllib.request.Request(f"{SEARXNG_URL}/search?{q}", headers={"User-Agent": "boardroom-grader"}) - with urllib.request.urlopen(req, timeout=20, context=_SSL_CTX) as resp: - data = json.loads(resp.read().decode()) - lines = [f"- {r.get('title','')}\n {r.get('url','')}\n {r.get('content','')[:300]}" - for r in (data.get("results") or [])[:8]] - return "\n".join(lines) or "(no results)" - except Exception as e: - return f"search error: {e}" - - -DISPATCH = {"list_files": tool_list_files, "read_file": tool_read_file, "web_search": tool_web_search} - -TOOLS = [ - {"type": "function", "function": { - "name": "list_files", "description": "List the available document (and report) files.", - "parameters": {"type": "object", "properties": {}}}}, - {"type": "function", "function": { - "name": "read_file", "description": "Read a document or report file by its path (from list_files).", - "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}}, -] -if SEARXNG_URL: - TOOLS.append({"type": "function", "function": { - "name": "web_search", "description": "Search the web via SearXNG; returns top results.", - "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}) - - -def run_tool(name: str, args: dict) -> str: - fn = DISPATCH.get(name) - if not fn: - return f"(unknown tool: {name})" - try: - return fn(args) - except Exception as e: - return f"(tool error: {e})" - - -# ---------------------------------------------------------------- LLM -def chat(messages: list) -> dict: - body = {"model": MODEL, "messages": messages, "tools": TOOLS, - "tool_choice": "auto", "temperature": TEMPERATURE, "max_tokens": MAX_OUTPUT_TOKENS} - req = urllib.request.Request( - f"{LLM_BASE}/chat/completions", - data=json.dumps(body).encode(), - headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"}, - method="POST") - with urllib.request.urlopen(req, timeout=300, context=_SSL_CTX) as resp: - data = json.loads(resp.read().decode()) - return data["choices"][0]["message"] - - -_JSON_ACTION = re.compile(r"```(?:json)?\s*(\{.*?\})\s*```", re.DOTALL) - - -def _text_fallback_calls(content: str) -> list: - if not content: +def _doc_names() -> list[str]: + if not os.path.isdir(DOCS): return [] - m = _JSON_ACTION.search(content) - if not m: - return [] - try: - obj = json.loads(m.group(1)) - except json.JSONDecodeError: - return [] - name = obj.get("tool") or obj.get("name") - if name in DISPATCH: - return [{"id": "fallback", "function": {"name": name, "arguments": json.dumps(obj.get("args", {}))}}] - return [] + return sorted(fn for fn in os.listdir(DOCS) + if fn.endswith(".txt") and os.path.isfile(os.path.join(DOCS, fn))) -# ---------------------------------------------------------------- prompts -def _preload_docs() -> tuple[str, bool]: - """Concatenate the document text up to DOC_BUDGET. Returns (text, truncated).""" - names = list_dir(DOCS) +def preload_docs(budget: int) -> tuple[str, bool]: + """Concatenate /docs/*.txt up to `budget` chars. Returns (text, truncated).""" chunks, used, truncated = [], 0, False - for n in names: + for n in _doc_names(): body = read_text(os.path.join(DOCS, n)) header = f"\n\n========== DOCUMENT: {n} ==========\n" - room = DOC_BUDGET - used + room = budget - used if room <= 0: truncated = True break @@ -209,129 +110,337 @@ def _preload_docs() -> tuple[str, bool]: return "".join(chunks), truncated -def _preload_reports() -> str: - names = list_dir(REPORTS) +def out_path() -> str: + if ROLE == "adjudicator": + return os.path.join(OUT_DIR, "ADJUDICATION.md") + if ROLE == "extractor": + return os.path.join(OUT_DIR, "extraction.json") + return os.path.join(OUT_DIR, f"{GID}.json") + + +def write_out(text: str) -> None: + os.makedirs(OUT_DIR, exist_ok=True) + with open(out_path(), "w") as f: + f.write(text.rstrip() + "\n") + + +def write_invalid_marker(err: str) -> None: + try: + os.makedirs(OUT_DIR, exist_ok=True) + with open(out_path() + ".invalid", "w") as f: + f.write(err.strip() + "\n") + except Exception: + pass + + +# ---------------------------------------------------------------- LLM client +def _post(payload: dict, timeout: int = 600) -> dict: + req = urllib.request.Request( + f"{LLM_BASE}/chat/completions", + data=json.dumps(payload).encode(), + headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"}, + method="POST") + with urllib.request.urlopen(req, timeout=timeout, context=_SSL_CTX) as resp: + return json.loads(resp.read().decode()) + + +def chat(messages: list, extra: dict | None = None) -> str: + """One completion. Retries with backoff on 5xx/connection errors; raises + HTTPError immediately on 4xx so the caller can walk the reliability ladder.""" + payload = {"model": MODEL, "messages": messages, + "temperature": TEMPERATURE, "max_tokens": MAX_OUTPUT_TOKENS} + if extra: + payload.update(extra) + last: Exception | None = None + for attempt in range(1, TRANSPORT_RETRIES + 1): + try: + data = _post(payload) + return (data["choices"][0]["message"].get("content") or "").strip() + except urllib.error.HTTPError as e: + if 400 <= e.code < 500: + raise + last = e + except Exception as e: # URLError, timeout, bad JSON envelope... + last = e + sleep = 5 * attempt + log(f"transport error ({last}); retry {attempt}/{TRANSPORT_RETRIES} in {sleep}s") + time.sleep(sleep) + raise RuntimeError(f"model endpoint unreachable after {TRANSPORT_RETRIES} attempts: {last}") + + +def chat_json(messages: list, schema: dict) -> str: + """The JSON reliability ladder: strict json_schema -> guided_json -> plain.""" + ladder = [ + ("json_schema", {"response_format": {"type": "json_schema", "json_schema": { + "name": schema.get("title") or "output", "schema": schema, "strict": True}}}), + ("guided_json", {"guided_json": schema}), + ("plain", None), + ] + last: Exception | None = None + for mode, extra in ladder: + try: + return chat(messages, extra=extra) + except urllib.error.HTTPError as e: + if 400 <= e.code < 500: + log(f"{mode} request rejected (HTTP {e.code}); trying next mode") + last = e + continue + raise + raise RuntimeError(f"all completion modes were rejected by the endpoint: {last}") + + +# ---------------------------------------------------------------- JSON parse +def parse_json(text: str): + """Whole-reply json.loads, else the first brace-balanced {...} block.""" + if not text: + return None + try: + return json.loads(text) + except json.JSONDecodeError: + pass + start = text.find("{") + while start != -1: + depth, in_str, esc = 0, False, False + for i in range(start, len(text)): + c = text[i] + if in_str: + if esc: + esc = False + elif c == "\\": + esc = True + elif c == '"': + in_str = False + elif c == '"': + in_str = True + elif c == "{": + depth += 1 + elif c == "}": + depth -= 1 + if depth == 0: + try: + return json.loads(text[start:i + 1]) + except json.JSONDecodeError: + break + start = text.find("{", start + 1) + return None + + +def structural_error(obj) -> str | None: + """Lightweight in-agent checks; the full jsonschema pass is orchestrator-side.""" + if not isinstance(obj, dict): + return "top-level value is not a JSON object" + if ROLE == "extractor": + for k in ("schema_version", "deck", "kpis", "forward_targets", + "red_flag_candidates", "narrative"): + if k not in obj: + return f"missing required key: {k}" + if not isinstance(obj.get("deck"), dict) or "period" not in obj["deck"]: + return "deck.period is missing" + else: # grader + for k in ("schema_version", "grader", "categories", "red_flags", "overall_comment"): + if k not in obj: + return f"missing required key: {k}" + cats = obj.get("categories") + if not isinstance(cats, list) or len(cats) != 8: + return "categories must contain exactly 8 entries (A-H)" + return None + + +# ---------------------------------------------------------------- prompts +def _persona() -> str: + return read_text(PERSONA_PATH).strip() + + +def _bdef() -> str: + return read_text(BDEF_PATH).strip() + + +def _red_flag_taxonomy() -> str: + """The '## Red-flag taxonomy' section of BDEF.md (falls back to the whole rubric).""" + bdef = _bdef() + low = bdef.lower() + i = low.find("## red-flag taxonomy") + return bdef[i:].strip() if i != -1 else bdef + + +def build_extractor_prompt(schema: dict) -> tuple[str, str, bool]: + schema_text = json.dumps(schema, indent=2) + system = ( + "You are the structured-data EXTRACTOR for a board-deck grading pipeline. " + "You turn the deck text into machine-readable JSON; you do not grade.\n\n" + "HARD RULES:\n" + "- Reply with ONLY one JSON object conforming exactly to the schema below. " + "No prose, no markdown fences, no comments.\n" + "- NEVER invent numbers. Every actual/target must appear in the deck text; " + "record where in \"source\".\n" + "- canonical_name is lower_snake_case, generic, and stable across quarters " + "(arr, ebitda_margin, churn_rate...).\n" + "- deck.period is the reporting period as printed on the deck " + "(e.g. 2026-Q2, 2026-H1, FY2026, 2026-05); null if truly absent.\n" + "- If the deck text you were given was truncated, set deck.truncated = true.\n\n" + "# OUTPUT SCHEMA (JSON Schema)\n" + schema_text + ) + persona = _persona() + if persona: + system = persona + "\n\n" + system + taxonomy = _red_flag_taxonomy() + prefix = ("# RED-FLAG TAXONOMY\nUse ONLY these codes when populating " + "red_flag_candidates:\n\n" + taxonomy + "\n\n# BOARD DECK TEXT\n") + suffix = ("\n\n# TASK\nExtract the deck metadata, every KPI actual, every " + "forward-looking target, red-flag candidates (taxonomy codes above), " + "and the narrative summary. Output the JSON object now.") + budget = max(8000, (MAX_MODEL_LEN - 3500) * 3 - len(system) - len(prefix) - len(suffix)) + docs, truncated = preload_docs(budget) + note = ("\n\n(NOTE: the deck text above was TRUNCATED to fit the context window — " + "set deck.truncated = true.)" if truncated else "") + return system, prefix + docs + note + suffix, truncated + + +def build_grader_prompt(schema: dict) -> tuple[str, str, bool]: + schema_text = json.dumps(schema, indent=2) + system = ( + f"You are '{NAME}', one grader on a panel scoring a portfolio-company board " + "deck against the BDEF rubric.\n\n" + "HARD RULES:\n" + "- Score every BDEF category A-H with an integer 1-5.\n" + "- Any score ABOVE or BELOW 3 REQUIRES verbatim evidence quotes from the deck, " + "each with a location (e.g. 'slide 6'). Unsupported non-3 scores will be " + "regressed to 3 by the pipeline.\n" + "- Do NOT compute totals or composite scores; numbers are computed elsewhere.\n" + f"- Set \"grader\" to exactly \"{GID}\".\n" + "- Use only the red-flag taxonomy codes defined in the rubric.\n" + "- Reply with ONLY one JSON object conforming exactly to the provided schema. " + "No prose, no markdown fences." + ) + persona = _persona() + if persona: + system += "\n\n# YOUR LENS — how YOU specifically read this deck\n" + persona + bdef = _bdef() + prefix = "# BDEF RUBRIC\n" + bdef + "\n\n# BOARD DECK TEXT\n" + suffix = ("\n\n# OUTPUT SCHEMA (JSON Schema)\n" + schema_text + + "\n\n# TASK\nGrade the deck per the rubric and your lens. " + "Output the JSON object now.") + budget = max(8000, (MAX_MODEL_LEN - 3500) * 3 - len(system) - len(prefix) - len(suffix)) + docs, truncated = preload_docs(budget) + note = ("\n\n(NOTE: the deck text above was TRUNCATED to fit the context window — " + "grade what is shown and mention the truncation in overall_comment.)" + if truncated else "") + return system, prefix + docs + note + suffix, truncated + + +def _load_panel_grades() -> str: + """All /grades/*.json panel reports, skipping extraction.json and anything + flagged invalid by the agent that produced it (sibling .invalid marker).""" parts = [] - budget = DOC_BUDGET - used = 0 - for n in names: - body = read_text(os.path.join(REPORTS, n)) - header = f"\n\n========== REVIEWER REPORT: {n} ==========\n" - seg = (header + body)[: max(0, budget - used)] - parts.append(seg) - used += len(seg) + if not os.path.isdir(GRADES_DIR): + return "" + for fn in sorted(os.listdir(GRADES_DIR)): + p = os.path.join(GRADES_DIR, fn) + if not fn.endswith(".json") or not os.path.isfile(p): + continue + if fn == "extraction.json" or os.path.exists(p + ".invalid"): + continue + parts.append(f"\n\n========== GRADER REPORT: {fn} ==========\n" + read_text(p)) return "".join(parts) -def system_prompt() -> str: - persona = read_text(PERSONA_PATH).strip() - if ROLE == "synthesizer": - base = (f"You are '{NAME}', the lead reviewer chairing a document-review panel. " - "You are given the panel members' individual reports (and the source " - "documents for reference). Produce ONE consolidated report in Markdown.") - else: - base = (f"You are '{NAME}', an expert confidential-document reviewer. Read the " - "document(s) provided and produce ONE written report in Markdown. Base every " - "statement on the documents; never invent facts. Be specific and cite the " - "document/section for each point.") - if persona: - base += "\n\n# YOUR LENS — how YOU specifically read this\n" + persona - tools_note = ( - "\n\nYou can call list_files and read_file to pull more content if what was " - "pre-loaded is truncated" - + (", and web_search for external context" if SEARXNG_URL else "") - + ". When done, reply with the FINAL report only — no tool call. If your client " - 'cannot emit tool calls, reply with a single fenced block: ' - '```json\\n{"tool":"read_file","args":{"path":"..."}}\\n``` and nothing else.' +def build_adjudicator_prompt() -> tuple[str, str]: + system = _persona() or ( + "You are the adjudicator chairing a panel of board-deck graders. You did not " + "read the deck first-hand for a fresh opinion — you weigh the panel's evidence." ) - return base + tools_note - - -def first_user_message() -> str: - rubric = read_text(RUBRIC_PATH).strip() or "Produce a thorough review report." - if ROLE == "synthesizer": - reports = _preload_reports() - docs, truncated = _preload_docs() - return (f"# REVIEW RUBRIC\n{rubric}\n\n# PANEL REPORTS\n{reports}\n\n" - f"# SOURCE DOCUMENTS (for reference){' (truncated)' if truncated else ''}\n{docs}\n\n" - "# YOUR TASK\nConsolidate the panel's reports into one authoritative report per the " - "rubric: shared findings, conflicts (and your adjudication), anything only one " - "reviewer caught, and a prioritized overall recommendation. Attribute points to " - "reviewers. Output the final consolidated report now.") - docs, truncated = _preload_docs() - note = ("\n\n(Note: the documents were truncated to fit context — use read_file to pull any " - "section you need in full.)" if truncated else "") - return (f"# REVIEW RUBRIC\n{rubric}\n\n# DOCUMENT(S)\n{docs}{note}\n\n" - "# YOUR TASK\nReview the document(s) above per the rubric and your lens. Output your " - "final report now.") + extraction_txt = read_text(EXTRACTION_PATH) + grades_txt = _load_panel_grades() + body = ("# STRUCTURED EXTRACTION (ground truth pulled from the deck)\n" + + extraction_txt + "\n\n# PANEL GRADE REPORTS\n" + grades_txt) + budget = max(8000, (MAX_MODEL_LEN - 3500) * 3 - len(system) - 1500) + if len(body) > budget: + body = body[:budget] + "\n\n(NOTE: input truncated to fit the context window.)" + task = ( + "\n\n# TASK\nWrite a MARKDOWN adjudication of the panel:\n" + "- Consensus per BDEF category A-H (one line each).\n" + "- Material disagreements: where graders diverge, what each cites, and whose " + "evidence is stronger (verbatim deck quotes beat assertions).\n" + "- Red flags: which are CONFIRMED and which are DISMISSED, and why.\n" + "- Exactly 3 questions the board should ask management next quarter.\n" + "Do NOT output numeric scores, totals, or JSON — narrative Markdown only." + ) + return system, body + task # ---------------------------------------------------------------- run -def out_path() -> str: - name = "CONSOLIDATED_REPORT.md" if ROLE == "synthesizer" else f"{RID}.md" - return os.path.join(OUT_DIR, name) +def finalize(obj: dict, truncated: bool) -> dict: + if ROLE == "grader": + obj["grader"] = GID + elif ROLE == "extractor" and truncated: + deck = obj.get("deck") + if isinstance(deck, dict): + deck["truncated"] = True + return obj -def write_report(text: str) -> None: - os.makedirs(OUT_DIR, exist_ok=True) - with open(out_path(), "w") as f: - f.write(text.strip() + "\n") +def run_json_role() -> None: + schema = json.loads(read_text(SCHEMA_PATH) or "{}") + if ROLE == "extractor": + system, user, truncated = build_extractor_prompt(schema) + else: + system, user, truncated = build_grader_prompt(schema) + if truncated: + log("document text truncated to fit the context window") + messages = [{"role": "system", "content": system}, {"role": "user", "content": user}] + + text = chat_json(messages, schema) + obj = parse_json(text) + err = structural_error(obj) if obj is not None else "reply was not parseable JSON" + if obj is not None and err is None: + write_out(json.dumps(finalize(obj, truncated), indent=2)) + log(f"wrote {out_path()}") + return + + # ONE repair round-trip. + log(f"invalid reply ({err}); attempting one repair round-trip") + repair = messages + [ + {"role": "assistant", "content": text or "(empty reply)"}, + {"role": "user", "content": ( + f"Your previous reply was not valid JSON or failed validation: {err}. " + "Reply with ONLY the corrected JSON.")}, + ] + text2 = chat_json(repair, schema) + obj2 = parse_json(text2) + err2 = structural_error(obj2) if obj2 is not None else "reply was not parseable JSON" + if obj2 is not None and err2 is None: + write_out(json.dumps(finalize(obj2, truncated), indent=2)) + log(f"wrote {out_path()} (after repair)") + return + + # Still bad: leave the raw text + an .invalid marker for the orchestrator. + write_out(text2 or text or "") + write_invalid_marker(f"invalid after repair round-trip: {err2}") + log(f"FAILED to produce valid JSON: {err2} (raw text + .invalid marker written)") -def run() -> str: - messages = [{"role": "system", "content": system_prompt()}, - {"role": "user", "content": first_user_message()}] - last_text = "" - for _ in range(MAX_STEPS): - msg = chat(messages) - content = msg.get("content") or "" - calls = msg.get("tool_calls") or [] - if content.strip(): - last_text = content.strip() - if not calls: - calls = _text_fallback_calls(content) - if not calls: - break # final report - messages.append({"role": "assistant", "content": content}) - for c in calls: - args = json.loads(c["function"]["arguments"] or "{}") - res = run_tool(c["function"]["name"], args) - messages.append({"role": "user", "content": f"[tool {c['function']['name']} result]\n{res[:20000]}"}) - continue - messages.append({"role": "assistant", "content": content or None, "tool_calls": calls}) - for c in calls: - try: - args = json.loads(c["function"]["arguments"] or "{}") - except json.JSONDecodeError: - args = {} - res = run_tool(c["function"]["name"], args) - messages.append({"role": "tool", "tool_call_id": c.get("id", ""), "content": res[:20000]}) - - # If the model ended on a tool turn with no report text, ask once more plainly. - if not last_text.strip(): - messages.append({"role": "user", "content": "Now output your final report in Markdown."}) - try: - last_text = (chat(messages).get("content") or "").strip() - except Exception: - pass - return last_text +def run_adjudicator() -> None: + system, user = build_adjudicator_prompt() + messages = [{"role": "system", "content": system}, {"role": "user", "content": user}] + text = chat(messages) + if not text.strip(): + raise RuntimeError("model returned an empty adjudication") + write_out(text) + log(f"wrote {out_path()} ({len(text)} chars)") def main() -> None: - print(f"[{RID}] reviewer starting (role={ROLE} model={MODEL} base={LLM_BASE})", flush=True) + log(f"starting (role={ROLE} model={MODEL} base={LLM_BASE} temp={TEMPERATURE})") try: - report = run() - if not report.strip(): - report = f"# {NAME}\n\n(The model returned no report text.)" - write_report(report) - print(f"[{RID}] report written to {out_path()} ({len(report)} chars)", flush=True) + if ROLE == "adjudicator": + run_adjudicator() + else: + run_json_role() except Exception as e: print(traceback.format_exc(), flush=True) - # Always leave a file so the orchestrator can see this reviewer ran. - try: - write_report(f"# {NAME} — ERROR\n\nThis reviewer failed: {e}\n") - except Exception: - pass + if ROLE != "adjudicator": + # Leave a marker so the orchestrator sees this agent ran and failed. + write_invalid_marker(f"agent error: {e}") raise diff --git a/startos/actions/configure-companies.ts b/startos/actions/configure-companies.ts new file mode 100644 index 0000000..8912ebd --- /dev/null +++ b/startos/actions/configure-companies.ts @@ -0,0 +1,185 @@ +import { sdk } from '../sdk' +import { configFile } from '../file-models/config' + +const { InputSpec, Value, List } = sdk + +const inputSpec = InputSpec.of({ + companies: Value.list( + List.obj( + { + name: 'Portfolio Companies', + description: + 'One entry per portfolio company. The SLUG is the folder you drop its ' + + 'decks into (inbox//2026-Q2-deck.pdf) and the key its running ' + + 'scorecard ledger lives under. Pinned targets are the KPIs the scorer ' + + 'holds the company to even when a deck goes quiet about them — pin the ' + + 'profitability thresholds especially, since they carry the heaviest ' + + 'weight (30 of the quant 60).', + default: [], + minLength: 0, + maxLength: 64, + }, + { + uniqueBy: 'slug', + displayAs: '{{slug}}', + spec: InputSpec.of({ + slug: Value.text({ + name: 'Slug', + description: + 'Folder name under the inbox and the ledger key. Stable — do not ' + + 'rename once decks have been graded.', + required: true, + default: null, + placeholder: 'acme-widgets', + patterns: [ + { regex: '^[a-z0-9][a-z0-9-]{0,40}$', + description: 'Lowercase letters, numbers, dashes (max 41 chars).' }, + ], + }), + name: Value.text({ + name: 'Display Name (optional)', + description: 'Shown on the dashboard and scorecards. Empty = the slug.', + required: false, + default: null, + placeholder: 'Acme Widgets, Inc.', + }), + kpiAliases: Value.textarea({ + name: 'KPI Aliases (optional)', + description: + 'One line per KPI: canonical=alias1;alias2 — maps the names a deck ' + + 'uses onto the canonical KPI name, so "Adj. EBITDA" and "EBITDA ' + + '(adj)" both land on the same pinned target.', + required: false, + default: null, + minRows: 2, + maxRows: 12, + placeholder: 'ebitda=Adj. EBITDA;EBITDA (adj)\nmrr=Monthly Recurring Revenue;MRR', + }), + pinnedTargets: Value.list( + List.obj( + { + name: 'Pinned KPI Targets', + description: + 'Targets graded every quarter regardless of what the deck ' + + 'chooses to show. Mark profitability KPIs (EBITDA, net margin, ' + + 'FCF...) so they score in the heavier profitability bucket.', + default: [], + minLength: 0, + maxLength: 32, + }, + { + uniqueBy: 'kpi', + displayAs: '{{kpi}} → {{target}}', + spec: InputSpec.of({ + kpi: Value.text({ + name: 'KPI Name', + description: 'Canonical KPI name (see the aliases field above).', + required: true, + default: null, + placeholder: 'ebitda', + }), + target: Value.number({ + name: 'Target', + description: 'The numeric target for the period.', + required: true, + default: null, + integer: false, + }), + unit: Value.text({ + name: 'Unit (optional)', + description: 'For display only, e.g. "USD", "%", "customers".', + required: false, + default: null, + placeholder: 'USD', + }), + direction: Value.select({ + name: 'Direction', + description: 'Whether hitting the target means being at-or-above it (revenue) or at-or-below it (churn, burn).', + default: 'gte', + values: { + gte: 'At or above target', + lte: 'At or below target', + }, + }), + profitability: Value.toggle({ + name: 'Profitability KPI', + description: 'Score this KPI in the profitability bucket (heaviest weight) instead of the general KPI bucket.', + default: false, + }), + }), + }, + ), + ), + }), + }, + ), + ), +}) + +export const configureCompanies = sdk.Action.withInput( + 'configure-companies', + + async ({ effects }) => ({ + name: 'Configure Companies', + description: 'Define the portfolio companies, their inbox slugs, KPI aliases, and pinned KPI targets.', + warning: null, + allowedStatuses: 'any', + group: 'Grading', + visibility: 'enabled', + }), + + inputSpec, + + async ({ effects }) => { + const cfg = await configFile.read().const(effects) + if (!cfg) return {} + return { + companies: cfg.companies.map((c) => ({ + slug: c.slug, + name: c.name || undefined, + kpiAliases: c.kpiAliases || undefined, + pinnedTargets: c.pinnedTargets.map((t) => ({ + kpi: t.kpi, + target: t.target, + unit: t.unit || undefined, + direction: t.direction, + profitability: t.profitability, + })), + })), + } + }, + + async ({ effects, input }) => { + await configFile.merge(effects, { + companies: input.companies.map((c) => ({ + slug: c.slug, + name: c.name ?? '', + kpiAliases: c.kpiAliases ?? '', + pinnedTargets: c.pinnedTargets.map((t) => ({ + kpi: t.kpi, + target: t.target, + unit: t.unit ?? '', + direction: t.direction, + profitability: t.profitability, + })), + })), + }) + + return { + version: '1', + title: 'Companies Configured', + message: + 'Saved ' + input.companies.length + ' company(ies). Drop each company\'s ' + + 'decks into inbox// (e.g. inbox/' + + (input.companies[0]?.slug || 'acme-widgets') + + '/2026-Q2-deck.pdf), then run "Grade Decks".', + result: { + type: 'single', + value: input.companies.map((c) => c.slug).join(', ') || '(none)', + copyable: false, + qr: false, + masked: false, + }, + } + }, +) diff --git a/startos/actions/configure-graders.ts b/startos/actions/configure-graders.ts index a7d097c..01c8ae4 100644 --- a/startos/actions/configure-graders.ts +++ b/startos/actions/configure-graders.ts @@ -4,14 +4,15 @@ import { configFile } from '../file-models/config' const { InputSpec, Value, List } = sdk const inputSpec = InputSpec.of({ - reviewers: Value.list( + graders: Value.list( List.obj( { - name: 'Review Panel', + name: 'Grading Panel', description: - 'Who sits on the panel — one entry per review. Add as many as you like. ' + - 'Each reviewer is a model from your catalog plus a PERSONA: the lens it ' + - 'reads through, so the same document gets examined from different angles.', + 'Who grades the decks — one entry per grader. Add as many as you like. ' + + 'Each grader is a model from your catalog plus a PERSONA: the lens it ' + + 'grades through, so the same deck gets scored from different angles ' + + 'before the deterministic composite is computed.', default: [], minLength: 1, maxLength: 32, @@ -22,10 +23,10 @@ const inputSpec = InputSpec.of({ spec: InputSpec.of({ name: Value.text({ name: 'Name', - description: 'Unique reviewer name. Becomes its container and report filename.', + description: 'Unique grader name. Becomes its container and grade-sheet filename.', required: true, default: null, - placeholder: 'risk-counsel', + placeholder: 'munger-lens', patterns: [ { regex: '^[A-Za-z0-9][A-Za-z0-9 _-]{0,40}$', description: 'Letters, numbers, spaces, dashes, underscores (max 41 chars).' }, @@ -33,31 +34,32 @@ const inputSpec = InputSpec.of({ }), model: Value.text({ name: 'Model Alias', - description: 'Which catalog model this reviewer uses (must match an alias from "Configure Models").', + description: 'Which catalog model this grader uses (must match an alias from "Configure Models").', required: true, default: null, - placeholder: 'reviewer-a', + placeholder: 'grader-a', }), persona: Value.textarea({ name: 'Persona / Lens', description: - 'How THIS reviewer should read the documents — its priorities and ' + - 'weighting. Injected into its system prompt. e.g. "You are skeptical ' + - 'legal counsel: weight liability, ambiguous obligations, and missing ' + - 'clauses above everything." Leave empty for a neutral reviewer.', + 'How THIS grader should read the deck — its priorities and ' + + 'weighting within the BDEF rubric. Injected into its system prompt. ' + + 'e.g. "You are a Munger-style inversion skeptic: ask what would have ' + + 'to be true for this deck to be hiding a deteriorating business." ' + + 'Leave empty for a neutral grader.', required: false, default: null, minRows: 3, maxRows: 16, placeholder: - 'You are a financial-controls reviewer. Focus on numbers that do not ' + - 'reconcile, unstated assumptions behind projections, and anything that ' + - 'would concern an auditor. Organize findings by severity.', + 'You are a Girdley-style operator. Weight unit economics, owner ' + + 'accountability, and whether the KPIs the board was promised last ' + + 'quarter are still being reported. Flag every silently dropped metric.', }), temperature: Value.number({ name: 'Sampling Temperature (optional)', description: - 'Best-effort per-reviewer sampling temperature for extra diversity. ' + + 'Best-effort per-grader sampling temperature for extra diversity. ' + 'Persona is the primary lever. Leave empty to use the model default.', required: false, default: null, @@ -71,15 +73,15 @@ const inputSpec = InputSpec.of({ ), }) -export const configureReviewers = sdk.Action.withInput( - 'configure-reviewers', +export const configureGraders = sdk.Action.withInput( + 'configure-graders', async ({ effects }) => ({ - name: 'Configure Reviewers', - description: 'Define the review panel: which models and which personas, and how many reviews.', + name: 'Configure Graders', + description: 'Define the grading panel: which models and which personas grade each deck.', warning: null, allowedStatuses: 'any', - group: null, + group: 'Grading', visibility: 'enabled', }), @@ -88,22 +90,23 @@ export const configureReviewers = sdk.Action.withInput( async ({ effects }) => { const cfg = await configFile.read().const(effects) if (!cfg) return {} - return { reviewers: cfg.reviewers } + return { graders: cfg.graders } }, async ({ effects, input }) => { - await configFile.merge(effects, { reviewers: input.reviewers }) + await configFile.merge(effects, { graders: input.graders }) return { version: '1', title: 'Panel Configured', message: - 'Saved a panel of ' + input.reviewers.length + ' reviewer(s). Each model ' + - 'alias must exist in "Configure Models". Set the rubric in "Configure ' + - 'Review", then drop documents and run a review.', + 'Saved a panel of ' + input.graders.length + ' grader(s). Each model ' + + 'alias must exist in "Configure Models". Set the scoring knobs in ' + + '"Configure Grading", add companies in "Configure Companies", then drop ' + + 'decks into inbox// and run "Grade Decks".', result: { type: 'single', - value: input.reviewers.map((r) => r.name).join(', '), + value: input.graders.map((g) => g.name).join(', '), copyable: false, qr: false, masked: false, diff --git a/startos/actions/configure-grading.ts b/startos/actions/configure-grading.ts index 1e78b26..1aa0c69 100644 --- a/startos/actions/configure-grading.ts +++ b/startos/actions/configure-grading.ts @@ -4,26 +4,33 @@ import { configFile } from '../file-models/config' const { InputSpec, Value } = sdk const inputSpec = InputSpec.of({ - reviewInstructions: Value.textarea({ - name: 'Review Rubric', + bdefOverride: Value.textarea({ + name: 'BDEF Rubric Override (optional)', description: - 'What every reviewer should look for and produce. Layered above each ' + - 'reviewer\'s persona. Be concrete about the structure you want back.', - required: true, + 'Leave EMPTY to grade against the baked-in BDEF v1.1 framework ' + + '(Girdley + Munger/Buffett). Non-empty text replaces the rubric wholesale, ' + + 'so include the qualitative categories A-H if you customize it.', + required: false, default: null, minRows: 5, - maxRows: 20, - placeholder: - 'Review the attached document(s). Produce: a short summary, key findings, ' + - 'risks/red flags, open questions, and recommendations. Cite the document and ' + - 'section for each point. Never invent facts not present in the documents.', + maxRows: 24, + placeholder: '(empty = the built-in BDEF v1.1 rubric)', + }), + extractorModel: Value.text({ + name: 'Extractor Model (optional)', + description: + 'Catalog alias of the model that runs the stage-1 structured KPI ' + + 'extraction over each deck. Empty = use the first model.', + required: false, + default: null, + placeholder: 'grader-a', }), networkMode: Value.select({ name: 'Network Mode', description: - 'Air-gapped: reviewers reach ONLY the on-Spark model proxy — zero internet, ' + - 'documents never leave your hardware (models must be pre-pulled into the ' + - 'Spark HF cache, all on the head Spark). Local services: reviewers may also ' + + 'Air-gapped: graders reach ONLY the on-Spark model proxy — zero internet, ' + + 'board decks never leave your hardware (models must be pre-pulled into the ' + + 'Spark HF cache, all on the head Spark). Local services: graders may also ' + 'reach LAN services like SearXNG and the second Spark (this network has ' + 'egress unless you firewall it).', default: 'airgapped', @@ -34,59 +41,151 @@ const inputSpec = InputSpec.of({ }), searxngUrl: Value.text({ name: 'SearXNG URL (local-services only)', - description: 'JSON-search endpoint to give reviewers a web_search tool. Ignored in air-gapped mode. Empty = no web search.', + description: 'JSON-search endpoint to give graders a web_search tool. Ignored in air-gapped mode. Empty = no web search.', required: false, default: null, placeholder: 'https://searxng.local', }), - synthesisEnabled: Value.toggle({ - name: 'Synthesize a Consolidated Report', + adjudicatorEnabled: Value.toggle({ + name: 'Run an Adjudicator', description: - 'After the panel finishes, run a local "lead reviewer" that reads all the ' + - 'individual reports and writes one consolidated report (themes, conflicts, ' + - 'consensus, recommendation). No frontier model — stays on the Sparks.', + 'After the panel finishes, run a local "lead grader" that reads every ' + + 'grade sheet, reconciles disagreements, and settles the qualitative scores ' + + 'the deterministic composite uses. No frontier model — stays on the Sparks.', default: true, }), - synthesisModel: Value.text({ - name: 'Lead Reviewer Model (optional)', - description: 'Catalog alias of the model that writes the consolidated report. Empty = use the first model.', + adjudicatorModel: Value.text({ + name: 'Adjudicator Model (optional)', + description: 'Catalog alias of the model that adjudicates the panel. Empty = use the first model.', required: false, default: null, - placeholder: 'reviewer-a', + placeholder: 'grader-a', }), - synthesisPersona: Value.textarea({ - name: 'Lead Reviewer Instructions (optional)', - description: 'Override how the consolidated report is written. Empty = a sensible built-in default.', + adjudicatorPersona: Value.textarea({ + name: 'Adjudicator Instructions (optional)', + description: 'Override how the adjudicator reconciles the panel. Empty = a sensible built-in default.', required: false, default: null, minRows: 3, maxRows: 14, }), wipeRemoteDocs: Value.toggle({ - name: 'Wipe Documents From Sparks After Review', + name: 'Wipe Decks From Sparks After Grading', description: - 'Delete the extracted document text from the Sparks when a job finishes. ' + - 'Reports are always kept on this StartOS box. Recommended for confidential material.', + 'Delete the extracted deck text from the Sparks when a job finishes. ' + + 'Scorecards and ledgers are always kept on this StartOS box. Recommended ' + + 'for confidential board material.', default: true, }), autoRunOnDrop: Value.toggle({ - name: 'Auto-run When Documents Are Dropped', + name: 'Auto-grade When Decks Are Dropped', description: - 'Start a review automatically (after a short debounce) whenever new files ' + - 'land in the inbox. Off by default so you trigger reviews explicitly.', + 'Start grading automatically (after a short debounce) whenever new decks ' + + 'land in the inbox. Off by default so you trigger grading explicitly.', default: false, }), + // --- Deterministic-scorer weights (composite = quant 60 + qual 40 - flags) --- + profitabilityKpi: Value.number({ + name: 'Weight: Profitability KPIs', + description: 'Points for the profitability KPI attainment bucket — the heaviest slice of the quant 60. Default 30.', + required: true, + default: 30, + integer: false, + min: 0, + max: 100, + }), + otherKpi: Value.number({ + name: 'Weight: Other KPIs', + description: 'Points for the non-profitability measurable-KPI bucket. Default 20.', + required: true, + default: 20, + integer: false, + min: 0, + max: 100, + }), + forecastIntegrity: Value.number({ + name: 'Weight: Forecast Integrity', + description: 'Points for deck N actuals hitting what deck N-1 promised. Default 10.', + required: true, + default: 10, + integer: false, + min: 0, + max: 100, + }), + qualCategoryMax: Value.number({ + name: 'Weight: Max Per Qualitative Category', + description: 'Max points per BDEF category A-H (8 categories x 5 = the qualitative 40). Default 5.', + required: true, + default: 5, + integer: false, + min: 0, + max: 100, + }), + redFlagCap: Value.number({ + name: 'Red-flag Penalty Cap', + description: 'Maximum total points red flags can subtract from the composite. Default 15.', + required: true, + default: 15, + integer: false, + min: 0, + max: 100, + }), + kpiCreditFloor: Value.number({ + name: 'KPI Credit Floor', + description: 'actual/target ratio below which a KPI earns zero credit (linear credit above it). Default 0.5.', + required: true, + default: 0.5, + integer: false, + min: 0, + max: 1, + }), + droppedKpiPenalty: Value.number({ + name: 'Dropped-KPI Penalty', + description: 'Penalty per KPI that silently disappeared from the deck. Default 2.', + required: true, + default: 2, + integer: false, + min: 0, + max: 100, + }), + droppedKpiMax: Value.number({ + name: 'Dropped-KPI Flag Limit', + description: 'Count at most this many dropped-KPI flags per deck. Default 3.', + required: true, + default: 3, + integer: true, + min: 0, + max: 50, + }), + evidenceFullCredit: Value.number({ + name: 'Evidence Full-credit Threshold', + description: 'Quote characters a qualitative finding needs for full weight (thinner evidence scales down). Default 400.', + required: true, + default: 400, + integer: true, + min: 0, + max: 100000, + }), + singleSourceFlagFactor: Value.number({ + name: 'Single-source Flag Damping', + description: 'Multiplier applied to red flags raised by only one grader. Default 0.5.', + required: true, + default: 0.5, + integer: false, + min: 0, + max: 1, + }), }) -export const configureReview = sdk.Action.withInput( - 'configure-review', +export const configureGrading = sdk.Action.withInput( + 'configure-grading', async ({ effects }) => ({ - name: 'Configure Review', - description: 'Set the rubric, air-gap mode, synthesis, and document retention.', + name: 'Configure Grading', + description: 'Set the BDEF rubric, air-gap mode, adjudication, retention, and the scoring weights.', warning: null, allowedStatuses: 'any', - group: null, + group: 'Grading', visibility: 'enabled', }), @@ -96,36 +195,60 @@ export const configureReview = sdk.Action.withInput( const cfg = await configFile.read().const(effects) if (!cfg) return {} return { - reviewInstructions: cfg.reviewInstructions, + bdefOverride: cfg.bdefOverride || undefined, + extractorModel: cfg.extractorModel || undefined, networkMode: cfg.networkMode, searxngUrl: cfg.searxngUrl || undefined, - synthesisEnabled: cfg.synthesisEnabled, - synthesisModel: cfg.synthesisModel || undefined, - synthesisPersona: cfg.synthesisPersona || undefined, + adjudicatorEnabled: cfg.adjudicatorEnabled, + adjudicatorModel: cfg.adjudicatorModel || undefined, + adjudicatorPersona: cfg.adjudicatorPersona || undefined, wipeRemoteDocs: cfg.wipeRemoteDocs, autoRunOnDrop: cfg.autoRunOnDrop, + profitabilityKpi: cfg.weights.profitabilityKpi, + otherKpi: cfg.weights.otherKpi, + forecastIntegrity: cfg.weights.forecastIntegrity, + qualCategoryMax: cfg.weights.qualCategoryMax, + redFlagCap: cfg.weights.redFlagCap, + kpiCreditFloor: cfg.weights.kpiCreditFloor, + droppedKpiPenalty: cfg.weights.droppedKpiPenalty, + droppedKpiMax: cfg.weights.droppedKpiMax, + evidenceFullCredit: cfg.weights.evidenceFullCredit, + singleSourceFlagFactor: cfg.weights.singleSourceFlagFactor, } }, async ({ effects, input }) => { await configFile.merge(effects, { - reviewInstructions: input.reviewInstructions, + bdefOverride: input.bdefOverride ?? '', + extractorModel: input.extractorModel ?? '', networkMode: input.networkMode, searxngUrl: input.searxngUrl ?? '', - synthesisEnabled: input.synthesisEnabled, - synthesisModel: input.synthesisModel ?? '', - synthesisPersona: input.synthesisPersona ?? '', + adjudicatorEnabled: input.adjudicatorEnabled, + adjudicatorModel: input.adjudicatorModel ?? '', + adjudicatorPersona: input.adjudicatorPersona ?? '', wipeRemoteDocs: input.wipeRemoteDocs, autoRunOnDrop: input.autoRunOnDrop, + weights: { + profitabilityKpi: input.profitabilityKpi, + otherKpi: input.otherKpi, + forecastIntegrity: input.forecastIntegrity, + qualCategoryMax: input.qualCategoryMax, + redFlagCap: input.redFlagCap, + kpiCreditFloor: input.kpiCreditFloor, + droppedKpiPenalty: input.droppedKpiPenalty, + droppedKpiMax: input.droppedKpiMax, + evidenceFullCredit: input.evidenceFullCredit, + singleSourceFlagFactor: input.singleSourceFlagFactor, + }, }) return { version: '1', - title: 'Review Settings Saved', + title: 'Grading Settings Saved', message: input.networkMode === 'airgapped' - ? 'Saved. Reviewers will run air-gapped (no internet). Ensure all models are on the head Spark and pre-pulled into its HF cache.' - : 'Saved. Reviewers run in local-services mode and may reach the network — make sure that is acceptable for these documents.', + ? 'Saved. Graders will run air-gapped (no internet). Ensure all models are on the head Spark and pre-pulled into its HF cache.' + : 'Saved. Graders run in local-services mode and may reach the network — make sure that is acceptable for these board decks.', result: { type: 'single', value: input.networkMode, copyable: false, qr: false, masked: false }, } }, diff --git a/startos/actions/configure-models.ts b/startos/actions/configure-models.ts index f80cfb4..9e934bf 100644 --- a/startos/actions/configure-models.ts +++ b/startos/actions/configure-models.ts @@ -9,7 +9,7 @@ const inputSpec = InputSpec.of({ { name: 'Model Catalog', description: - 'The local models this service can serve on your Sparks. Each reviewer ' + + 'The local models this service can serve on your Sparks. Each grader ' + 'references one of these by its alias. The job runner loads models in ' + 'waves so you can run a panel across more models than fit in GPU memory ' + 'at once.', @@ -23,10 +23,10 @@ const inputSpec = InputSpec.of({ spec: InputSpec.of({ alias: Value.text({ name: 'Alias', - description: 'Short name reviewers use to pick this model (e.g. "qwen-32b").', + description: 'Short name graders use to pick this model (e.g. "qwen-32b").', required: true, default: null, - placeholder: 'reviewer-a', + placeholder: 'grader-a', patterns: [ { regex: '^[a-z0-9][a-z0-9-]{0,30}$', description: 'Lowercase letters, numbers, dashes (max 31 chars).' }, @@ -42,7 +42,7 @@ const inputSpec = InputSpec.of({ spark: Value.select({ name: 'Served On', description: - 'Which Spark serves this model. Air-gapped review mode requires the ' + + 'Which Spark serves this model. Air-gapped grading mode requires the ' + 'head (primary) Spark; the secondary is used only in local-services mode.', default: 'primary', values: { primary: 'Primary (head) Spark', secondary: 'Secondary Spark' }, @@ -68,7 +68,7 @@ const inputSpec = InputSpec.of({ }), maxModelLen: Value.number({ name: 'Max Model Length', - description: 'vLLM --max-model-len (context window). Documents are chunked to fit.', + description: 'vLLM --max-model-len (context window). Deck text is chunked to fit.', required: true, default: 32768, integer: true, @@ -77,7 +77,7 @@ const inputSpec = InputSpec.of({ toolCallParser: Value.text({ name: 'Tool-Call Parser', description: - 'vLLM tool-call parser for the reviewer\'s read-file tool loop. Match the ' + + 'vLLM tool-call parser for the grader\'s read-file tool loop. Match the ' + 'served model family (Qwen3 → "hermes"). Empty disables native tool-calling.', required: false, default: 'hermes', @@ -112,7 +112,7 @@ export const configureModels = sdk.Action.withInput( description: 'Define the local model catalog served on your Sparks and the serving knobs.', warning: null, allowedStatuses: 'any', - group: null, + group: 'Setup', visibility: 'enabled', }), @@ -146,7 +146,7 @@ export const configureModels = sdk.Action.withInput( title: 'Models Configured', message: 'Saved ' + input.models.length + ' model(s). Make sure each is present in ' + - 'the Spark HF cache for air-gapped runs, then set "Configure Reviewers".', + 'the Spark HF cache for air-gapped runs, then set "Configure Graders".', result: { type: 'single', value: input.models.map((m) => m.alias).join(', '), diff --git a/startos/actions/configure-sparks.ts b/startos/actions/configure-sparks.ts index 4808f1d..c1a8079 100644 --- a/startos/actions/configure-sparks.ts +++ b/startos/actions/configure-sparks.ts @@ -7,7 +7,7 @@ const { InputSpec, Value } = sdk const inputSpec = InputSpec.of({ primarySparkHost: Value.text({ name: 'Primary Spark Host', - description: 'Hostname or IP of the head DGX Spark (reachable over SSH). Serves models, hosts the model proxy, and runs the reviewer panel.', + description: 'Hostname or IP of the head DGX Spark (reachable over SSH). Serves models, hosts the model proxy, and runs the grading panel.', required: true, default: null, placeholder: 'spark-01.local', @@ -46,8 +46,8 @@ const inputSpec = InputSpec.of({ name: 'Use Both Sparks', description: 'Allow models to be served on a second Spark (over ConnectX/200GbE) for ' + - 'extra capacity. NOTE: in air-gapped review mode all models must run on the ' + - 'head Spark; the second Spark is used only in local-services mode.', + 'extra capacity. NOTE: in air-gapped grading mode all models must run on ' + + 'the head Spark; the second Spark is used only in local-services mode.', default: false, }), secondarySparkHost: Value.text({ @@ -67,7 +67,7 @@ const inputSpec = InputSpec.of({ }), remoteWorkDir: Value.text({ name: 'Remote Work Directory', - description: 'Absolute path on the head Spark for staged document text, the HF cache, and logs.', + description: 'Absolute path on the head Spark for staged deck text, the HF cache, and logs.', required: true, default: '/home/nvidia/boardroom-map', }), @@ -78,8 +78,8 @@ const inputSpec = InputSpec.of({ default: 'boardroom-vllm:latest', }), graderImage: Value.text({ - name: 'Reviewer Image Tag', - description: 'The sandboxed reviewer image built on the head Spark from sandbox/build.sh.', + name: 'Grader Image Tag', + description: 'The sandboxed grader image built on the head Spark from sandbox/build.sh.', required: true, default: 'boardroom-grader:latest', }), @@ -103,7 +103,7 @@ export const configureSparks = sdk.Action.withInput( description: 'Set the DGX Spark connection details, SSH credentials, and image tags.', warning: null, allowedStatuses: 'any', - group: null, + group: 'Setup', visibility: 'enabled', }), @@ -154,7 +154,7 @@ export const configureSparks = sdk.Action.withInput( title: 'Sparks Configured', message: 'Saved. Use "Test Spark Connection" to verify SSH + GPU access, then set ' + - '"Configure Models" and "Configure Reviewers".', + '"Configure Models" and "Configure Graders".', result: { type: 'single', value: input.primarySparkHost, copyable: false, qr: false, masked: false }, } }, diff --git a/startos/actions/grade-decks.ts b/startos/actions/grade-decks.ts index de89e55..72a5882 100644 --- a/startos/actions/grade-decks.ts +++ b/startos/actions/grade-decks.ts @@ -2,17 +2,17 @@ import { startSdk } from '@start9labs/start-sdk' import { sdk } from '../sdk' /** - * Trigger a review of whatever is currently in the inbox. The running job-runner + * Trigger grading of whatever is currently in the inbox. The running job-runner * thread polls for /data/state/run_request and starts a job when it appears, so * this action just drops that request file (decoupled from the daemon — no need * to reach its HTTP port from the action's one-shot container). */ -export const runReview = sdk.Action.withoutInput( - 'run-review', +export const gradeDecks = sdk.Action.withoutInput( + 'grade-decks', async ({ effects }) => ({ - name: 'Run Review', - description: 'Convene the panel now over the documents currently in the inbox.', + name: 'Grade Decks', + description: 'Grade all decks currently in the inbox (inbox//...).', warning: null, allowedStatuses: 'only-running', group: null, @@ -36,23 +36,23 @@ export const runReview = sdk.Action.withoutInput( 'sh', '-c', 'mkdir -p /data/state && date +%s > /data/state/run_request && ' + - 'n=$(ls -1 /data/inbox 2>/dev/null | wc -l | tr -d " "); ' + - 'echo "Review requested. $n file(s) in the inbox."', + 'n=$(find /data/inbox -type f 2>/dev/null | wc -l | tr -d " "); ' + + 'echo "Grading requested. $n deck file(s) in the inbox."', ], { mounts, env: { BM_DATA_DIR: '/data' } }, - 'run-review', + 'grade-decks', ) - output = (stdout?.toString() || '').trim() || 'Review requested.' + output = (stdout?.toString() || '').trim() || 'Grading requested.' } catch (e: any) { - output = 'Could not request a review: ' + (e?.message || String(e)) + output = 'Could not request grading: ' + (e?.message || String(e)) } return { version: '1', - title: 'Review Requested', + title: 'Grading Requested', message: output + - ' Watch the Web UI for progress; reports appear there and via "View Latest Report".', + ' Watch the Web UI for progress; scorecards appear there and via "View Latest Scorecard".', result: { type: 'single', value: output, copyable: false, qr: false, masked: false }, } }, diff --git a/startos/actions/index.ts b/startos/actions/index.ts index ef0ac59..6540781 100644 --- a/startos/actions/index.ts +++ b/startos/actions/index.ts @@ -1,17 +1,22 @@ import { sdk } from '../sdk' import { configureSparks } from './configure-sparks' -import { configureModels } from './configure-models' -import { configureReviewers } from './configure-reviewers' -import { configureReview } from './configure-review' -import { runReview } from './run-review' import { testConnection } from './test-connection' -import { latestReport } from './latest-report' +import { configureModels } from './configure-models' +import { configureGraders } from './configure-graders' +import { configureGrading } from './configure-grading' +import { configureCompanies } from './configure-companies' +import { gradeDecks } from './grade-decks' +import { latestScorecard } from './latest-scorecard' +// Setup group: Sparks -> connection test -> model catalog. +// Grading group: panel -> scoring knobs -> portfolio companies. +// Ungrouped (day-to-day): grade the inbox, read the latest scorecard. export const actions = sdk.Actions.of() .addAction(configureSparks) - .addAction(configureModels) - .addAction(configureReviewers) - .addAction(configureReview) - .addAction(runReview) .addAction(testConnection) - .addAction(latestReport) + .addAction(configureModels) + .addAction(configureGraders) + .addAction(configureGrading) + .addAction(configureCompanies) + .addAction(gradeDecks) + .addAction(latestScorecard) diff --git a/startos/actions/latest-scorecard.ts b/startos/actions/latest-scorecard.ts index 3958bd8..59f35ca 100644 --- a/startos/actions/latest-scorecard.ts +++ b/startos/actions/latest-scorecard.ts @@ -2,17 +2,18 @@ import { startSdk } from '@start9labs/start-sdk' import { sdk } from '../sdk' /** - * Returns the latest report as a copyable result, so you can read it straight - * from the StartOS service page without opening the Web UI. The job runner saves - * the most recent report (consolidated if synthesis is on, else the panel - * digest) to /data/reports/latest.md on the StartOS host — no Spark round-trip. + * Returns the latest scorecard as a copyable result, so you can read it straight + * from the StartOS service page without opening the dashboard. The job runner + * saves the most recent scorecard to /data/reports/latest-scorecard.md (with + * /data/reports/latest.md as the legacy fallback) on the StartOS host — no + * Spark round-trip. */ -export const latestReport = sdk.Action.withoutInput( - 'latest-report', +export const latestScorecard = sdk.Action.withoutInput( + 'latest-scorecard', async ({ effects }) => ({ - name: 'View Latest Report', - description: 'Show the most recent review report produced by the panel.', + name: 'View Latest Scorecard', + description: 'Show the most recent deck scorecard produced by the grading panel.', warning: null, allowedStatuses: 'any', group: null, @@ -32,19 +33,25 @@ export const latestReport = sdk.Action.withoutInput( const { stdout } = await startSdk.runCommand( effects, { imageId: 'main' }, - ['sh', '-c', 'cat /data/reports/latest.md 2>/dev/null || echo "(no report yet — drop documents in the inbox and run a review)"'], + [ + 'sh', + '-c', + 'cat /data/reports/latest-scorecard.md 2>/dev/null || ' + + 'cat /data/reports/latest.md 2>/dev/null || ' + + 'echo "(no scorecard yet — drop decks into inbox// and run Grade Decks)"', + ], { mounts, env: { BM_DATA_DIR: '/data' } }, - 'latest-report', + 'latest-scorecard', ) - report = (stdout?.toString() || '').trim() || '(no report yet)' + report = (stdout?.toString() || '').trim() || '(no scorecard yet)' } catch (e: any) { - report = 'Could not read report: ' + (e?.message || String(e)) + report = 'Could not read scorecard: ' + (e?.message || String(e)) } return { version: '1', - title: 'Latest Boardroom Map Report', - message: 'The panel\'s most recent review.', + title: 'Latest Scorecard', + message: 'The panel\'s most recent deck scorecard.', result: { type: 'single', value: report, copyable: true, qr: false, masked: false }, } }, diff --git a/startos/actions/test-connection.ts b/startos/actions/test-connection.ts index 213d962..38fd9ee 100644 --- a/startos/actions/test-connection.ts +++ b/startos/actions/test-connection.ts @@ -3,18 +3,19 @@ import { sdk } from '../sdk' /** * Runs a one-shot in the orchestrator image that SSHes into the configured - * Spark(s) and reports `nvidia-smi` plus whether the vLLM + reviewer images are - * built. Reuses orchestrator/spark_client.py so SSH logic lives in one place. + * Spark(s) and reports `nvidia-smi` plus whether the vLLM (boardroom-vllm) and + * grader (boardroom-grader) images are built. Reuses orchestrator/spark_client.py + * so SSH logic lives in one place. */ export const testConnection = sdk.Action.withoutInput( 'test-connection', async ({ effects }) => ({ name: 'Test Spark Connection', - description: 'SSH into the configured Spark(s) and verify GPU + serving/reviewer image access.', + description: 'SSH into the configured Spark(s) and verify GPU + serving/grader image access.', warning: null, allowedStatuses: 'any', - group: null, + group: 'Setup', visibility: 'enabled', }), diff --git a/startos/file-models/config.ts b/startos/file-models/config.ts index 263dd09..4ecec67 100644 --- a/startos/file-models/config.ts +++ b/startos/file-models/config.ts @@ -3,16 +3,21 @@ import { FileHelper, z } from '@start9labs/start-sdk' /** * Boardroom Map configuration, persisted to the `main` volume as config.json. * - * Written by the StartOS actions (Configure Sparks / Models / Reviewers / - * Review) and read by the Python orchestrator inside the container, which mounts - * the same volume at /data and reads /data/config.json. Keep field names in sync - * with orchestrator/bm_config.py (CONFIG_DEFAULTS). + * Written by the StartOS actions (Configure Sparks / Models / Graders / + * Grading / Companies) and read by the Python orchestrator inside the + * container, which mounts the same volume at /data and reads /data/config.json. + * Keep field names in sync with orchestrator/bm_config.py (CONFIG_DEFAULTS). * - * Boardroom Map is a CONTROL PLANE: a GPU-free orchestrator on StartOS that SSHes into - * one or two DGX Sparks to serve local models and run a panel of sandboxed - * "reviewer" containers over confidential documents you drop in. There is NO - * frontier model and NO cloud key — everything stays on your hardware. The only - * secrets are the Spark SSH key and an optional Hugging Face token (secrets.ts). + * Boardroom Map is a CONTROL PLANE: a GPU-free orchestrator on StartOS that + * SSHes into one or two DGX Sparks to serve local models and grade the board + * decks you drop into /data/inbox//. A panel of sandboxed + * "grader" containers scores each deck against the BDEF v1.1 framework + * (Girdley + Munger/Buffett); Python then computes a deterministic composite + * (quant KPI attainment 60 incl. profitability 30, qualitative categories 40, + * red-flag penalties up to -15) and appends it to the company's running + * ledger. There is NO frontier model and NO cloud key — everything stays on + * your hardware. The only secrets are the Spark SSH key and an optional + * Hugging Face token (secrets.ts). */ export const configShape = z.object({ // --- Spark connection (mirrors LLaMA-Factory / Nightshift) --- @@ -35,20 +40,21 @@ export const configShape = z.object({ // --- Serving (vLLM on the Sparks) --- gpuMemoryUtilization: z.string().default('0.85'), maxModelLen: z.number().int().positive().default(32768), - // vLLM tool-call parser for native function-calling (the reviewer's read-file + // vLLM tool-call parser for native function-calling (the grader's read-file // tool loop relies on it). Must match the served model family — Qwen3 → - // 'hermes'. Empty disables native tool-calling (reviewers fall back to a + // 'hermes'. Empty disables native tool-calling (graders fall back to a // JSON-action text protocol). toolCallParser: z.string().default('hermes'), // LiteLLM router exposing every model alias on one OpenAI-compatible endpoint. proxyPort: z.number().int().positive().default(4000), // How many distinct models may be co-resident on the HEAD Spark at once. The - // job runner loads models in WAVES so it never exceeds this — letting you run a - // panel across more models than fit in GPU memory simultaneously. 1 is safest. + // job runner loads models in WAVES so it never exceeds this — letting you run + // a panel across more models than fit in GPU memory simultaneously. 1 is + // safest. maxConcurrentModels: z.number().int().positive().default(1), // The MODEL CATALOG: the set of local models the service can serve. Each - // reviewer (below) references one of these by `alias`. Mirrors + // grader (below) references one of these by `alias`. Mirrors // bm_config.py CONFIG_DEFAULTS["models"]. models: z .array( @@ -62,13 +68,14 @@ export const configShape = z.object({ }), ) .default([ - { alias: 'reviewer-a', hfModel: 'Qwen/Qwen3-32B-FP8', spark: 'primary', port: 8001 }, + { alias: 'grader-a', hfModel: 'Qwen/Qwen3-32B-FP8', spark: 'primary', port: 8001 }, ]), - // --- The review panel: one entry per reviewer ("number of reviews") --- - // Each reviewer is a model + a persona (the lens it reads through) + an - // optional temperature. Mirrors bm_config.py CONFIG_DEFAULTS["reviewers"]. - reviewers: z + // --- The grading panel: one entry per grader --- + // Each grader is a model + a persona (the lens it grades through — e.g. a + // Munger-style inversion skeptic or a Girdley-style operator) + an optional + // temperature. Mirrors bm_config.py CONFIG_DEFAULTS["graders"]. + graders: z .array( z.object({ name: z.string(), @@ -79,50 +86,124 @@ export const configShape = z.object({ }), ) .default([ - { name: 'reviewer-1', model: 'reviewer-a', persona: '', temperature: null }, + { name: 'munger-lens', model: 'grader-a', persona: '', temperature: null }, ]), - // --- Review job settings --- - // The rubric: what every reviewer should look for / produce. Layered above - // each reviewer's persona. - reviewInstructions: z.string().default( - 'Review the attached document(s). Produce a structured report: a 3-5 sentence ' + - 'summary, the key findings and insights, risks or red flags, open questions, ' + - 'and concrete recommendations. Cite the document and section for each point. ' + - 'Be honest about uncertainty; never invent facts not present in the documents.', - ), - // Confidentiality posture for the reviewer containers: - // 'airgapped' — reviewers join an --internal Docker network: they can + // Which catalog model runs the stage-1 structured KPI extractor over each + // deck. Empty = first model in the catalog. + extractorModel: z.string().default(''), + + // --- Grading job settings --- + // The rubric override. Empty = the baked-in BDEF v1.1 framework + // (orchestrator/bdef.md — Girdley + Munger/Buffett). Non-empty text replaces + // it wholesale, so include scoring categories A-H if you customize. + bdefOverride: z.string().default(''), + + // Deterministic-scorer knobs. The composite is 0-100 = quant 60 (profitability + // 30 + other KPIs 20 + forecast integrity 10) + qualitative 40 (8 BDEF + // categories x 5) - red-flag penalties (capped at 15). Mirrors + // bm_config.py WEIGHTS_DEFAULTS; keep both in sync. + weights: z + .object({ + // Points for the profitability KPI attainment bucket (heaviest weight). + profitabilityKpi: z.number().default(30), + // Points for the non-profitability measurable-KPI bucket. + otherKpi: z.number().default(20), + // Points for forecast integrity: deck N actuals vs deck N-1 stated targets. + forecastIntegrity: z.number().default(10), + // Max points per qualitative BDEF category A-H (8 x 5 = 40). + qualCategoryMax: z.number().default(5), + // Cap on total red-flag penalty. + redFlagCap: z.number().default(15), + // actual/target ratio below which a KPI earns zero credit. + kpiCreditFloor: z.number().default(0.5), + // Penalty per KPI that silently disappeared from the deck. + droppedKpiPenalty: z.number().default(2), + // Count at most this many dropped-KPI flags. + droppedKpiMax: z.number().default(3), + // Quote characters required for full qualitative-evidence weight. + evidenceFullCredit: z.number().default(400), + // Damping factor for red flags raised by a single grader only. + singleSourceFlagFactor: z.number().default(0.5), + }) + .default({ + profitabilityKpi: 30, + otherKpi: 20, + forecastIntegrity: 10, + qualCategoryMax: 5, + redFlagCap: 15, + kpiCreditFloor: 0.5, + droppedKpiPenalty: 2, + droppedKpiMax: 3, + evidenceFullCredit: 400, + singleSourceFlagFactor: 0.5, + }), + + // Confidentiality posture for the grader containers: + // 'airgapped' — graders join an --internal Docker network: they can // reach ONLY the on-Spark model proxy, with zero internet // egress. Models must be pre-pulled into the Spark's HF // cache (no live download). All models must be on the head // Spark. Strongest confidentiality. - // 'local_services' — reviewers may also reach configured LAN services + // 'local_services' — graders may also reach configured LAN services // (e.g. SearXNG) and the second Spark. NOTE: this network // has egress unless you firewall it — use only when you - // accept that reviewers can reach the network. + // accept that graders can reach the network. networkMode: z.enum(['airgapped', 'local_services']).default('airgapped'), - // SearXNG JSON endpoint, used ONLY in local_services mode to give reviewers a + // SearXNG JSON endpoint, used ONLY in local_services mode to give graders a // web_search tool. Empty = no web search. searxngUrl: z.string().default(''), - // --- Synthesis (a local lead reviewer; no frontier model) --- - synthesisEnabled: z.boolean().default(true), - // Alias of the model that writes the consolidated report. Empty = first model. - synthesisModel: z.string().default(''), - // Optional persona/instructions for the lead reviewer. Empty = built-in default. - synthesisPersona: z.string().default(''), + // --- Adjudication (a local lead grader; no frontier model) --- + adjudicatorEnabled: z.boolean().default(true), + // Alias of the model that reconciles the panel's grades. Empty = first model. + adjudicatorModel: z.string().default(''), + // Optional persona/instructions for the adjudicator. Empty = built-in default. + adjudicatorPersona: z.string().default(''), // --- Document handling --- - // After a job, wipe the extracted document text from the Sparks. Reports are + // After a job, wipe the extracted deck text from the Sparks. Scorecards are // kept on the StartOS box regardless. Default true for confidentiality. wipeRemoteDocs: z.boolean().default(true), - // Watch /data/inbox and auto-start a review when files land (debounced). - // Default false: you trigger reviews explicitly with "Run Review". + // Watch /data/inbox and auto-start grading when decks land (debounced). + // Default false: you trigger grading explicitly with "Grade Decks". autoRunOnDrop: z.boolean().default(false), // Name of the per-job Docker network created on the head Spark. networkName: z.string().default('boardroom-net'), + // --- Portfolio companies --- + // The authoritative source of pinned KPI targets and KPI-name aliases. Decks + // are dropped into /data/inbox// and each company keeps its own running + // scorecard ledger. Mirrors bm_config.py CONFIG_DEFAULTS["companies"]. + companies: z + .array( + z.object({ + // Directory name under /data/inbox and the ledger key. Stable — do not + // rename once decks have been graded. + slug: z.string(), + // Display name for the dashboard. Empty = the slug. + name: z.string().default(''), + // Newline-separated "canonical=alias1;alias2" lines mapping the names a + // deck uses for a KPI onto its canonical name. + kpiAliases: z.string().default(''), + // Targets the scorer holds the company to even when a deck goes quiet + // about them. `profitability: true` marks the KPI as part of the + // heavier profitability bucket. + pinnedTargets: z + .array( + z.object({ + kpi: z.string(), + target: z.number(), + unit: z.string().default(''), + direction: z.enum(['gte', 'lte']).default('gte'), + profitability: z.boolean().default(false), + }), + ) + .default([]), + }), + ) + .default([]), + // --- Auth flags (the secret itself lives in secrets.ts) --- hfTokenSet: z.boolean().default(false), }) diff --git a/startos/interfaces.ts b/startos/interfaces.ts index 674cba5..bc98db6 100644 --- a/startos/interfaces.ts +++ b/startos/interfaces.ts @@ -13,8 +13,9 @@ export const setInterfaces = sdk.setupInterfaces(async ({ effects }) => { name: 'Web UI', id: 'webui', description: - 'The Boardroom Map control panel: drop documents in, convene the reviewer ' + - 'panel, watch the job run on your Sparks, and read the reports.', + 'The Boardroom Map control panel: drop board decks into each company\'s ' + + 'inbox, run the grading panel on your Sparks, and watch per-company ' + + 'scorecard trends.', type: 'ui', username: null, path: '', diff --git a/startos/main.ts b/startos/main.ts index b78cc26..9307e80 100644 --- a/startos/main.ts +++ b/startos/main.ts @@ -3,8 +3,8 @@ import { WEB_UI_PORT } from './interfaces' export const main = sdk.setupMain(async ({ effects }) => { // Mount the persistent volume at /data: config.json, ssh key, optional HF - // token, the dropped-document inbox, job run state, and saved reports all live - // here. + // token, the per-company deck inbox, job run state, and saved scorecards / + // ledgers all live here. const mounts = sdk.Mounts.of().mountVolume({ volumeId: 'main', mountpoint: '/data', @@ -19,9 +19,10 @@ export const main = sdk.setupMain(async ({ effects }) => { 'boardroom-webui', ) - // The web UI runs the FastAPI app AND, in a background thread, the Boardroom Map job - // runner (which extracts dropped documents, serves the chosen models on the - // Sparks in waves, runs the reviewer panel, and synthesizes a report). + // The web UI runs the FastAPI app AND, in a background thread, the Boardroom + // Map job runner (which extracts dropped decks, serves the chosen models on + // the Sparks in waves, runs the grading panel + adjudicator, computes the + // deterministic composite, and updates each company's scorecard ledger). return sdk.Daemons.of(effects).addDaemon('webui', { subcontainer: sub, exec: { diff --git a/startos/manifest/index.ts b/startos/manifest/index.ts index 5d9f9c6..c571176 100644 --- a/startos/manifest/index.ts +++ b/startos/manifest/index.ts @@ -7,16 +7,19 @@ import { setupManifest } from '@start9labs/start-sdk' * does not run any GPU workload itself. It is a small web UI + job runner that * SSHes into one or two NVIDIA DGX Sparks to: * 1. serve a panel of local LLMs with vLLM (loaded in waves to fit GPU memory), - * 2. extract text from documents you drop in (PDF/DOCX/TXT/MD — done on the - * StartOS box), ship it to the Sparks, and launch a panel of sandboxed - * "reviewer" containers (each a model + a persona) that read the documents - * and write a report, - * 3. optionally run a local "lead reviewer" that synthesizes the panel's - * reports into one consolidated report. + * 2. extract text from the board decks you drop into inbox// + * (PDF/DOCX/TXT/MD — done on the StartOS box), ship it to the Sparks, and + * launch a panel of sandboxed "grader" containers (each a model + a + * persona) that grade each deck against the BDEF v1.1 framework + * (Girdley + Munger/Buffett), + * 3. optionally run a local "adjudicator" that reconciles the panel, after + * which Python computes a deterministic composite (quant KPI attainment 60 + * incl. profitability 30, qualitative categories 40, red-flag penalties + * up to -15) and appends it to the company's running scorecard ledger. * * There is NO frontier model and NO cloud API key. In the default `airgapped` - * network mode the reviewer containers can reach ONLY the on-Spark model proxy — - * the documents and their reviews never touch the internet. + * network mode the grader containers can reach ONLY the on-Spark model proxy — + * the decks and their grades never touch the internet. * * NOTE: s9pk.mk extracts the package identifier from the single-quoted value on * the line below, so keep that field on one line and avoid stray quotes above it. @@ -30,17 +33,21 @@ export const manifest = setupManifest({ marketingUrl: 'https://github.com/ten31/boardroom-map', donationUrl: null, description: { - short: 'A private panel of local LLMs that reviews your confidential documents on your DGX Sparks', + short: 'Grade portfolio-company board decks with local LLMs on your DGX Sparks — BDEF scoring, per-company running scorecards', long: - 'Boardroom Map lets you drop confidential documents in and convene a panel of ' + - 'local LLMs running on your NVIDIA DGX Sparks to review them. You choose ' + - 'which models and which personas (lenses) sit on the panel and how many ' + - 'reviews to run. Each reviewer reads the documents and writes a report; an ' + - 'optional local lead reviewer synthesizes them into one consolidated ' + - 'report. There is no frontier model and no cloud key: in the default ' + - 'air-gapped mode the reviewers reach only the on-Spark model endpoint, so ' + - 'your documents and their reviews never leave your hardware. No GPU is ' + - 'needed on the StartOS host.', + 'Boardroom Map turns your DGX Sparks into a private board-deck grading ' + + 'panel. Drop each portfolio company\'s deck into its inbox folder and a ' + + 'panel of local LLMs (each a model + a persona) grades it against the ' + + 'BDEF v1.1 framework (Girdley + Munger/Buffett); an optional local ' + + 'adjudicator reconciles the panel, then a deterministic scorer computes a ' + + '0-100 composite — quantitative KPI attainment worth 60 (profitability ' + + 'alone 30, plus forecast integrity: deck N actuals vs deck N-1 promises), ' + + 'qualitative categories worth 40, and red-flag penalties up to -15. Each ' + + 'company keeps a running scorecard ledger, and a web dashboard shows the ' + + 'trends. There is no frontier model and no cloud key: in the default ' + + 'air-gapped mode the graders reach only the on-Spark model endpoint, so ' + + 'your confidential decks never leave your hardware. No GPU is needed on ' + + 'the StartOS host.', }, // Arch-agnostic orchestrator. Docker build paths are relative to the PROJECT // ROOT (where the Makefile runs), matching the Start9 convention. @@ -53,7 +60,7 @@ export const manifest = setupManifest({ }, }, arch: ['x86_64', 'aarch64'], - // The orchestrator only SSHes out + extracts document text on CPU; it never + // The orchestrator only SSHes out + extracts deck text on CPU; it never // touches a local GPU. nvidiaContainer: false, }, @@ -66,10 +73,12 @@ export const manifest = setupManifest({ alerts: { install: 'Boardroom Map drives work on REMOTE machines (your DGX Sparks) over SSH; ' + - 'nothing serves or runs on your StartOS server. After install: ' + - '(1) "Configure Sparks" for SSH access, (2) "Configure Models" for the ' + - 'local models to serve, (3) "Configure Reviewers" for the panel + personas, ' + - '(4) "Configure Review" for the rubric and air-gap mode. Then drop ' + - 'documents in the inbox and run "Run Review".', + 'nothing serves or runs on your StartOS server, and your confidential ' + + 'board decks stay on your LAN. After install: (1) "Configure Sparks" for ' + + 'SSH access, (2) "Configure Models" for the local models to serve, ' + + '(3) "Configure Graders" for the panel + personas, (4) "Configure Grading" ' + + 'for the BDEF rubric, air-gap mode, and scoring weights, (5) "Configure ' + + 'Companies" for slugs and pinned KPI targets. Then drop decks into ' + + 'inbox// and run "Grade Decks".', }, }) diff --git a/startos/versions/v_0_1_0.ts b/startos/versions/v_0_1_0.ts index 602767e..eff579e 100644 --- a/startos/versions/v_0_1_0.ts +++ b/startos/versions/v_0_1_0.ts @@ -7,10 +7,12 @@ import { VersionInfo } from '@start9labs/start-sdk' export const v_0_1_0 = VersionInfo.of({ version: '0.1.0:0', releaseNotes: - 'Initial release: drop confidential documents in and convene a panel of ' + - 'local LLMs on your DGX Sparks to review them. Choose the models, the ' + - 'personas, and how many reviews; an optional local lead reviewer ' + - 'synthesizes a consolidated report. Default air-gapped mode keeps documents ' + - 'and reviews entirely on your hardware.', + 'Initial version — BDEF v1.1 deck grading with per-company scorecards. ' + + 'Drop board decks into inbox// and a panel of local LLMs on ' + + 'your DGX Sparks grades them against the BDEF framework (Girdley + ' + + 'Munger/Buffett); a deterministic scorer computes the 0-100 composite ' + + '(quant 60 incl. profitability 30, qualitative 40, red flags to -15) and ' + + 'appends it to each company\'s running ledger. Default air-gapped mode ' + + 'keeps decks and grades entirely on your hardware.', migrations: {}, })