jonathan/boardroom-map
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement BDEF v1.1 grading: scoring core, per-deck pipeline, ledger, dashboard, StartOS layer

Browse Source 
- Deterministic scoring.py (quant 60 / qual 40 / flags -15, profitability heaviest)
- Per-company JSON ledger with forecast-target chaining deck N-1 -> N
- Single-shot sandbox agent with guided-JSON fallback ladder (no tool loop)
- Portfolio dashboard with sparklines, KPI hit rates, BDEF category bars
- 48 unit tests green; endpoints smoke-tested; npm check+build green

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jonathan Kirkwood
2026-07-06 14:15:12 -05:00
co-authored by Claude Fable 5
parent 1dde915540
commit b1d7aed9f4
48 changed files with 4907 additions and 971 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -4,3 +4,4 @@ javascript/
.DS_Store
**/__pycache__/
*.pyc
.claude/
README.md
+61 -30
View File
@@ -1,44 +1,75 @@
# Boardroom Map — a private document-review panel for your DGX Sparks
# Boardroom Map — private board-deck grading on your DGX Sparks
Boardroom Map is a StartOS service (`.s9pk`) that lets you **drop confidential
documents in and have a panel of local LLMs review them** on your NVIDIA DGX
Sparks. You pick the models and the personas (lenses), and how many reviews to
run; each reviewer writes a report, and an optional **local lead reviewer**
synthesizes them into one consolidated report. There is **no frontier model and
no cloud API key** — in the default air-gapped mode the documents and reviews
Boardroom Map is a StartOS service (`.s9pk`) that **grades portfolio-company
board decks with a panel of local LLMs** on your NVIDIA DGX Sparks. Drop each
company's deck into `inbox/<company-slug>/`; the panel grades it against the
**BDEF v1.1 framework** (Girdley + Munger/Buffett), an optional local
**adjudicator** reconciles the panel, and a deterministic Python scorer computes
a 0–100 composite that lands on the company's **running scorecard ledger**. A
web dashboard shows per-company trends. There is **no frontier model and no
cloud API key** — in the default air-gapped mode the decks and their grades
never leave your hardware.
It is a sibling of [Nightshift](../nightshift) and reuses the same control-plane
It is a sibling of [Chambers](../chambers) and reuses the same control-plane
pattern (a GPU-free orchestrator on StartOS driving the Sparks over SSH), but
with the swarm, the git blackboard, and the Claude overseer removed and replaced
by an on-demand **document-review pipeline**.
swaps the free-form document-review panel for a **deterministic deck-grading
pipeline** with pinned KPI targets and per-company ledgers.
## The scoring model
`composite (0–100) = quant 60 + qual 40 − red flags (capped at 15)`
- **Quantitative 60:** profitability KPI attainment **30** (heaviest slice),
other measurable KPIs **20**, **forecast integrity 10** — deck N's actuals are
chained against deck N−1's stated targets, so moved goalposts cost points.
KPI credit is linear above a floor ratio (default 0.5 → zero credit below).
- **Qualitative 40:** eight BDEF categories (A–H) × 5 points, scored by the
panel with evidence quotes; thin evidence scales down.
- **Red flags:** up to **−15**; silently dropped KPIs are auto-flagged (capped),
and flags raised by a single grader are damped by 0.5.
Every knob lives in config (`weights`, per-company `pinnedTargets` and
`kpiAliases`) so the model can be retuned without a rebuild.
## Architecture
```
StartOS box (control plane, no GPU) DGX Spark(s)
┌────────────────────────────────────┐ ┌───────────────────────────────┐
│ FastAPI web UI + job runner │ SSH │ per-job Docker network │
│ • inbox (drop documents) │ ───────▶│ (──internal in airgapped) │
│ FastAPI dashboard + job runner │ SSH │ per-job Docker network │
│ • inbox/<company-slug>/ (decks) │ ───────▶│ (──internal in airgapped) │
│ • extract text (PDF/DOCX/TXT/MD) │ rsync │ ┌─────────┐ ┌────────────┐ │
│ • plan model "waves" │ ───────▶│ │ vLLM(s) │◀─│ LiteLLM │ │
│ • launch reviewer containers │ │ └─────────┘ │ router │ │
│ • pull reports, synthesize, wipe │◀─────── │ ┌──────────────┐ ▲ │ │
│ • reports saved here only │ rsync │ │ reviewer ×N │──┘ │ │
│ • extractor → graders → adjudicator│ │ └─────────┘ │ router │ │
│ • deterministic composite scorer │◀─────── │ ┌──────────────┐ ▲ │ │
│ • per-company ledgers + scorecards │ rsync │ │ grader ×N │──┘ │ │
└────────────────────────────────────┘ │ │ (read-only, │ │ │
│ │ sandboxed) │ │ │
│ └──────────────┘ │ │
└───────────────────────────────┘
```
- **Reviewers** are one-shot, read-only, hardened containers (non-root,
- **Graders** are one-shot, read-only, hardened containers (non-root,
`--cap-drop ALL`, read-only rootfs, no docker socket). In air-gapped mode they
sit on an `--internal` network and can reach only the model proxy.
- **Waves:** the job runner serves models in waves bounded by
`maxConcurrentModels`, so a panel can span more models than fit in GPU memory
at once.
- **Confidentiality:** documents are extracted to text on the StartOS box; only
- **Air-gap modes:** `airgapped` (default — graders reach only the on-Spark
model proxy, zero egress, models pre-pulled) or `local_services` (graders may
reach LAN services like SearXNG and the second Spark — has egress unless
firewalled).
- **Confidentiality:** decks are extracted to text on the StartOS box; only
text crosses to the Sparks, and it is wiped from the Sparks after the job.
Scorecards and ledgers live only on the StartOS box.
## Setup order
Configure Sparks → Test Spark Connection → Configure Models → Configure Graders
→ Configure Grading (rubric, air-gap, weights) → Configure Companies (slugs,
KPI aliases, pinned targets — especially profitability thresholds) → drop decks
into `inbox/<company-slug>/2026-Q2-deck.pdf` → **Grade Decks** → watch the
dashboard.
## Repo layout
@@ -46,25 +77,25 @@ StartOS box (control plane, no GPU) DGX Spark(s)
startos/ StartOS package definition (TypeScript / start-sdk)
manifest/ main.ts interfaces.ts versions/ file-models/ actions/
orchestrator/ The control-plane app (Python)
app.py FastAPI UI + JSON API
jobs.py the job runner (extract → serve waves → review → synthesize)
app.py FastAPI dashboard + JSON API
jobs.py the job runner (extract → serve waves → grade → adjudicate → score)
serving.py vLLM + LiteLLM router on the Sparks, in waves
reviewers.py launch the reviewer panel
synthesis.py the local lead reviewer
graders.py launch the grading panel
adjudicator.py the local lead grader
extraction.py PDF/DOCX/TXT/MD → text (on the StartOS box)
preflight.py probe models before launching reviewers
preflight.py probe models before launching graders
spark_client.py SSH/rsync helpers
bdef.md the baked-in BDEF v1.1 rubric
bm_config.py config defaults (mirrors startos/file-models/config.ts)
sandbox/ reviewer image (built ON the Spark, not packed in the s9pk)
grader_agent.py reviewer.Dockerfile build.sh
openclaw/ what each Spark needs provisioned (OpenClaw's job)
sandbox/ grader image (built ON the Spark, not packed in the s9pk)
grader_agent.py grader.Dockerfile build.sh
```
## Build
Same path as Nightshift — GitHub CI (`.github/workflows/build.yml`) or a local
build with `start-cli` (see the s9pk-build-on-mac recipe). The vLLM and reviewer
images are built **on the Sparks**, not packed into the `.s9pk`.
GitHub CI (`.github/workflows/build.yml`) or a local build with `start-cli`
(see the s9pk-build-on-mac recipe). The vLLM and grader images are built **on
the Sparks**, not packed into the `.s9pk`.
```
npm ci && npm run check && npm run build # type-check + bundle
@@ -74,5 +105,5 @@ make # pack the .s9pk (needs start-cli)
## Status
v0.1 — source complete, `tsc`-clean and Python-syntax-clean. Not yet validated
against live Sparks. See `openclaw/OPENCLAW_SPEC.md` for the Spark-side
provisioning (HF model pre-pull is required for air-gapped runs).
against live Sparks. HF model pre-pull on the head Spark is required for
air-gapped runs.
assets/instructions.md
+46 -23
View File
@@ -1,13 +1,14 @@
# Boardroom Map
Drop confidential documents in and convene a **panel of local LLMs** running on
your DGX Sparks to review them. You choose the models, the personas (lenses), and
how many reviews. An optional **local lead reviewer** synthesizes the panel into
one consolidated report. There is **no frontier model and no cloud key** — in the
default air-gapped mode the documents and their reviews never leave your hardware.
Drop portfolio-company board decks in and have a **panel of local LLMs** running
on your DGX Sparks grade them against the **BDEF v1.1 framework** (Girdley +
Munger/Buffett). A deterministic scorer turns the panel's grades into a 0–100
composite and appends it to each company's **running scorecard ledger**; the web
dashboard shows the trends. There is **no frontier model and no cloud key** — in
the default air-gapped mode your confidential decks never leave your hardware.
Boardroom Map is a *control plane*: nothing serves or runs on your StartOS box (it only
SSHes to the Sparks and extracts document text on CPU).
Boardroom Map is a *control plane*: nothing serves or runs on your StartOS box
(it only SSHes to the Sparks and extracts deck text on CPU).
## Setup (run the Actions in order)
@@ -16,32 +17,54 @@ SSHes to the Sparks and extracts document text on CPU).
2. **Configure Models** — the catalog of local models to serve (alias → HF id →
which Spark → port), and serving knobs. For air-gapped runs every model must be
on the **head Spark** and present in its HF cache.
3. **Configure Reviewers** — the panel: one entry per review, each a model + a
persona (the lens it reads through) + an optional temperature.
4. **Configure Review** — the rubric, the **Network Mode** (air-gapped vs
local-services), synthesis on/off + lead model, and whether to wipe documents
from the Sparks afterward.
3. **Configure Graders** — the panel: one entry per grader, each a model + a
persona (the lens it grades through — e.g. a Munger inversion skeptic, a
Girdley operator, a skeptical CFO) + an optional temperature.
4. **Configure Grading** — the BDEF rubric override (empty = the built-in
BDEF v1.1), the **Network Mode** (air-gapped vs local-services), the
extractor + adjudicator models, deck retention, and the scoring weights.
5. **Configure Companies** — one entry per portfolio company: its inbox **slug**,
display name, KPI aliases, and **pinned KPI targets**. Pin the profitability
thresholds especially — profitability carries the heaviest weight.
## Running a review
## Grading decks
1. Open the **Web UI** and drag your documents (PDF / DOCX / TXT / MD) onto the
inbox (or drop them in the service's `inbox` folder).
2. Click **Run Review** (or enable *auto-run on drop*).
1. Drop each company's deck into its inbox folder, e.g.
`inbox/acme-widgets/2026-Q2-deck.pdf` (PDF / DOCX / TXT / MD), via the
**Web UI** or the service's `inbox` directory.
2. Run **Grade Decks** (or enable *auto-grade on drop*).
3. Watch the activity log. The service extracts text locally, serves the needed
models on the Sparks **in waves** (so a panel can span more models than fit in
GPU memory at once), runs each reviewer, then the lead reviewer, and saves the
reports. Read them in the Web UI or via **View Latest Report**.
GPU memory at once), runs the structured KPI extractor, then each grader, then
the adjudicator, and finally computes the composite and updates the company's
ledger. Read the results on the dashboard or via **View Latest Scorecard**.
## How the score works
The composite is **0–100 = quantitative 60 + qualitative 40 − red flags (max 15)**:
- **Quant 60** — profitability KPI attainment **30** (the heaviest single slice),
other measurable KPIs **20**, and **forecast integrity 10**: deck N's actuals
are chained against what deck N−1 promised, so sandbagging and quietly moved
goalposts cost points.
- **Qual 40** — eight BDEF categories (A–H), up to 5 points each, scored by the
panel with evidence quotes (thin evidence scales the score down).
- **Red flags** — up to **−15**; silently dropped KPIs are flagged automatically,
and flags raised by only one grader are damped.
Pinned targets from **Configure Companies** are graded every quarter whether or
not the deck mentions them — a deck cannot improve its score by going quiet.
## Network modes
- **Air-gapped (default):** reviewer containers join an `--internal` Docker
- **Air-gapped (default):** grader containers join an `--internal` Docker
network — they can reach only the on-Spark model proxy, with zero internet
egress. Models are served from a pre-pulled HF cache. All models must be on the
head Spark. Strongest confidentiality.
- **Local services:** reviewers may also reach LAN services (e.g. SearXNG) and the
- **Local services:** graders may also reach LAN services (e.g. SearXNG) and the
second Spark. This network has egress unless you firewall it — use only when you
accept that reviewers can reach the network.
accept that graders can reach the network.
The original documents are extracted to plain text on the StartOS box; only that
The original decks are extracted to plain text on the StartOS box; only that
text is shipped to the Sparks, and it is wiped from the Sparks after the job (the
reports are kept on your StartOS box).
scorecards and ledgers are kept on your StartOS box).
icon.svg
+8 -9
View File
@@ -6,13 +6,12 @@
</linearGradient>
</defs>
<rect width="256" height="256" rx="48" fill="url(#bg)"/>
<!-- document -->
<rect x="74" y="56" width="92" height="120" rx="8" fill="#e9edf5"/>
<rect x="88" y="78" width="64" height="7" rx="3.5" fill="#9aa3b5"/>
<rect x="88" y="96" width="64" height="7" rx="3.5" fill="#9aa3b5"/>
<rect x="88" y="114" width="44" height="7" rx="3.5" fill="#9aa3b5"/>
<!-- magnifying glass (the review) -->
<circle cx="158" cy="150" r="34" fill="none" stroke="#c9a24b" stroke-width="10"/>
<line x1="182" y1="174" x2="206" y2="198" stroke="#c9a24b" stroke-width="12" stroke-linecap="round"/>
<circle cx="158" cy="150" r="20" fill="#c9a24b" opacity="0.18"/>
<!-- scorecard bars: an ascending quarterly trend -->
<rect x="56" y="148" width="28" height="52" rx="6" fill="#3d4666"/>
<rect x="100" y="124" width="28" height="76" rx="6" fill="#5a6a94"/>
<rect x="144" y="96" width="28" height="104" rx="6" fill="#c9a24b"/>
<!-- spark above the top bar: the latest grade -->
<path d="M186 44 l7 18 18 7 -18 7 -7 18 -7 -18 -18 -7 18 -7 z" fill="#c9a24b"/>
<!-- baseline -->
<rect x="48" y="204" width="160" height="6" rx="3" fill="#2a3040"/>
</svg>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 941 B

After

Width:  |  Height:  |  Size: 837 B

instructions.md
+46 -23
View File
@@ -1,13 +1,14 @@
# Boardroom Map
Drop confidential documents in and convene a **panel of local LLMs** running on
your DGX Sparks to review them. You choose the models, the personas (lenses), and
how many reviews. An optional **local lead reviewer** synthesizes the panel into
one consolidated report. There is **no frontier model and no cloud key** — in the
default air-gapped mode the documents and their reviews never leave your hardware.
Drop portfolio-company board decks in and have a **panel of local LLMs** running
on your DGX Sparks grade them against the **BDEF v1.1 framework** (Girdley +
Munger/Buffett). A deterministic scorer turns the panel's grades into a 0–100
composite and appends it to each company's **running scorecard ledger**; the web
dashboard shows the trends. There is **no frontier model and no cloud key** — in
the default air-gapped mode your confidential decks never leave your hardware.
Boardroom Map is a *control plane*: nothing serves or runs on your StartOS box (it only
SSHes to the Sparks and extracts document text on CPU).
Boardroom Map is a *control plane*: nothing serves or runs on your StartOS box
(it only SSHes to the Sparks and extracts deck text on CPU).
## Setup (run the Actions in order)
@@ -16,32 +17,54 @@ SSHes to the Sparks and extracts document text on CPU).
2. **Configure Models** — the catalog of local models to serve (alias → HF id →
which Spark → port), and serving knobs. For air-gapped runs every model must be
on the **head Spark** and present in its HF cache.
3. **Configure Reviewers** — the panel: one entry per review, each a model + a
persona (the lens it reads through) + an optional temperature.
4. **Configure Review** — the rubric, the **Network Mode** (air-gapped vs
local-services), synthesis on/off + lead model, and whether to wipe documents
from the Sparks afterward.
3. **Configure Graders** — the panel: one entry per grader, each a model + a
persona (the lens it grades through — e.g. a Munger inversion skeptic, a
Girdley operator, a skeptical CFO) + an optional temperature.
4. **Configure Grading** — the BDEF rubric override (empty = the built-in
BDEF v1.1), the **Network Mode** (air-gapped vs local-services), the
extractor + adjudicator models, deck retention, and the scoring weights.
5. **Configure Companies** — one entry per portfolio company: its inbox **slug**,
display name, KPI aliases, and **pinned KPI targets**. Pin the profitability
thresholds especially — profitability carries the heaviest weight.
## Running a review
## Grading decks
1. Open the **Web UI** and drag your documents (PDF / DOCX / TXT / MD) onto the
inbox (or drop them in the service's `inbox` folder).
2. Click **Run Review** (or enable *auto-run on drop*).
1. Drop each company's deck into its inbox folder, e.g.
`inbox/acme-widgets/2026-Q2-deck.pdf` (PDF / DOCX / TXT / MD), via the
**Web UI** or the service's `inbox` directory.
2. Run **Grade Decks** (or enable *auto-grade on drop*).
3. Watch the activity log. The service extracts text locally, serves the needed
models on the Sparks **in waves** (so a panel can span more models than fit in
GPU memory at once), runs each reviewer, then the lead reviewer, and saves the
reports. Read them in the Web UI or via **View Latest Report**.
GPU memory at once), runs the structured KPI extractor, then each grader, then
the adjudicator, and finally computes the composite and updates the company's
ledger. Read the results on the dashboard or via **View Latest Scorecard**.
## How the score works
The composite is **0–100 = quantitative 60 + qualitative 40 − red flags (max 15)**:
- **Quant 60** — profitability KPI attainment **30** (the heaviest single slice),
other measurable KPIs **20**, and **forecast integrity 10**: deck N's actuals
are chained against what deck N−1 promised, so sandbagging and quietly moved
goalposts cost points.
- **Qual 40** — eight BDEF categories (A–H), up to 5 points each, scored by the
panel with evidence quotes (thin evidence scales the score down).
- **Red flags** — up to **−15**; silently dropped KPIs are flagged automatically,
and flags raised by only one grader are damped.
Pinned targets from **Configure Companies** are graded every quarter whether or
not the deck mentions them — a deck cannot improve its score by going quiet.
## Network modes
- **Air-gapped (default):** reviewer containers join an `--internal` Docker
- **Air-gapped (default):** grader containers join an `--internal` Docker
network — they can reach only the on-Spark model proxy, with zero internet
egress. Models are served from a pre-pulled HF cache. All models must be on the
head Spark. Strongest confidentiality.
- **Local services:** reviewers may also reach LAN services (e.g. SearXNG) and the
- **Local services:** graders may also reach LAN services (e.g. SearXNG) and the
second Spark. This network has egress unless you firewall it — use only when you
accept that reviewers can reach the network.
accept that graders can reach the network.
The original documents are extracted to plain text on the StartOS box; only that
The original decks are extracted to plain text on the StartOS box; only that
text is shipped to the Sparks, and it is wiped from the Sparks after the job (the
reports are kept on your StartOS box).
scorecards and ledgers are kept on your StartOS box).
orchestrator/adjudicator.py
+37 -52
View File
@@ -1,88 +1,73 @@
"""Local lead-reviewer synthesis — no frontier model.
"""Panel adjudication — a local model weighs the graders' evidence. No scores.
After the panel finishes, one more hardened container (the "lead reviewer") reads
all the individual reports (mounted read-only at /reports) plus the documents
(/docs), runs a configured local model, and writes a single consolidated report
to /out/CONSOLIDATED_REPORT.md: shared themes, where reviewers disagree, the
consensus, and an overall recommendation.
After the panel finishes grading one deck (and the outputs have been validated),
one more hardened one-shot container reads the structured extraction
(/extraction.json, ro) plus every panel grade report (/grades, ro) and writes a
MARKDOWN adjudication to /out/ADJUDICATION.md: consensus per BDEF category,
material disagreements and whose evidence is stronger, red flags confirmed or
dismissed, and three questions for next quarter. It never computes numbers —
scoring is deterministic Python (scoring.py).
It reuses the same one-shot reviewer image, switched to BM_ROLE=synthesizer.
It reuses the grader image, switched to BM_ROLE=adjudicator. Mount contract
(remote paths under the per-deck dir):
out/ -> /grades:ro (panel *.json; the agent skips
extraction.json and *.invalid)
out/extraction.json -> /extraction.json:ro
personas/adjudicator.md -> /persona/PERSONA.md:ro
adjudicator-out/ -> /out:rw
No /docs — the adjudicator judges the panel's evidence, not the deck first-hand.
"""
from __future__ import annotations
import shlex
import spark_client as sc
import graders as gr_mod
import serving
DEFAULT_LEAD_PERSONA = (
"You are the lead reviewer chairing the panel. You did not read the documents "
"first-hand for a fresh opinion — your job is to CONSOLIDATE the panel's "
"individual reports into one authoritative report. Identify the findings the "
"reviewers agree on, surface and adjudicate where they conflict, note anything "
"only one reviewer caught, and end with a prioritized recommendation. Attribute "
"points to the reviewer(s) who raised them. Do not invent findings."
)
import spark_client as sc
def pick_model(cfg: dict) -> str:
alias = (cfg.get("synthesisModel") or "").strip()
alias = (cfg.get("adjudicatorModel") or "").strip()
if alias:
return alias
models = cfg.get("models") or []
return models[0]["alias"] if models else ""
def run_synthesis(cfg: dict, jobdir: str, rubric: str, log, wait_timeout: int = 1800) -> dict:
"""Launch the lead-reviewer container and wait for the consolidated report."""
def run_adjudication(cfg: dict, remote_deck_dir: str, log, wait_timeout: int = 1800) -> dict:
"""Launch the adjudicator container for one deck and wait for ADJUDICATION.md."""
head = sc.head(cfg)
q = shlex.quote
model = pick_model(cfg)
if not model:
raise RuntimeError("no model available for synthesis (configure a model catalog)")
raise RuntimeError("no model available for adjudication (configure a model catalog)")
persona = (cfg.get("synthesisPersona") or "").strip() or DEFAULT_LEAD_PERSONA
rid = "adjudicator"
net = serving.net_name(cfg)
base = serving.reviewer_proxy_base(cfg)
rid = "lead-reviewer"
sc.run(head,
f"mkdir -p {q(jobdir)}/personas {q(jobdir)}/synth-out && "
f"printf '%s' {q(persona)} > {q(jobdir)}/personas/{rid}.md && "
f"printf '%s' {q(rubric)} > {q(jobdir)}/RUBRIC.md",
timeout=30)
env = (
f"-e BM_REVIEWER_ID={q(rid)} -e BM_REVIEWER_NAME=lead-reviewer -e BM_ROLE=synthesizer "
f"-e BM_MODEL={q(model)} -e BM_LLM_BASE={q(base)} -e BM_LLM_KEY=sk-local "
f"-e BM_MAX_MODEL_LEN={q(str(int(cfg.get('maxModelLen', 32768))))} -e HOME=/home/rev "
)
harden = (
"--user 1000:1000 --security-opt no-new-privileges --cap-drop ALL "
"--read-only --tmpfs /tmp:size=256m --tmpfs /home/rev:size=128m "
"--pids-limit 256 --memory 6g --cpus 4"
)
env = gr_mod.base_env(cfg, rid, "adjudicator", "adjudicator", model, None)
mounts = (
f"-v {q(jobdir)}/docs:/docs:ro "
f"-v {q(jobdir)}/out:/reports:ro "
f"-v {q(jobdir)}/synth-out:/out "
f"-v {q(jobdir)}/personas/{rid}.md:/persona/PERSONA.md:ro "
f"-v {q(jobdir)}/RUBRIC.md:/RUBRIC.md:ro "
f"-v {q(remote_deck_dir)}/out:/grades:ro "
f"-v {q(remote_deck_dir)}/out/extraction.json:/extraction.json:ro "
f"-v {q(remote_deck_dir)}/personas/{rid}.md:/persona/PERSONA.md:ro "
f"-v {q(remote_deck_dir)}/adjudicator-out:/out "
)
cname = f"bm-grader-{rid}"
cname = f"bm-grader-{rid}-{gr_mod.container_suffix(remote_deck_dir)}"
cmd = (
f"docker rm -f {cname} >/dev/null 2>&1; "
f"docker run -d --name {cname} --network {q(net)} {harden} {env} {mounts} {q(cfg['graderImage'])}"
f"docker run -d --name {cname} --network {q(net)} {gr_mod.HARDEN} {env} {mounts} "
f"{q(cfg['graderImage'])}"
)
log(f"[synthesis] lead reviewer up -> {model}")
log(f"[adjudicator] up -> {model}")
r = sc.run(head, cmd, timeout=120)
if r.returncode != 0:
raise RuntimeError(f"synthesis launch failed: {r.stderr or r.stdout}")
raise RuntimeError(f"adjudicator launch failed: {r.stderr or r.stdout}")
w = sc.run(head, f"docker wait {cname}", timeout=wait_timeout)
code = (w.stdout or "").strip()
chk = sc.run(head, f"test -s {q(jobdir)}/synth-out/CONSOLIDATED_REPORT.md && echo OK || echo MISSING", timeout=30)
chk = sc.run(head, f"test -s {q(remote_deck_dir)}/adjudicator-out/ADJUDICATION.md "
"&& echo OK || echo MISSING", timeout=30)
wrote = "OK" in (chk.stdout or "")
sc.run(head, f"docker rm -f {cname} 2>/dev/null; true", timeout=30)
log(f"[synthesis] lead reviewer exited (code={code or '?'}), report={'written' if wrote else 'MISSING'}")
log(f"[adjudicator] exited (code={code or '?'}), "
f"adjudication={'written' if wrote else 'MISSING'}")
return {"model": model, "exit": code, "report": wrote}
orchestrator/app.py
+264 -49
View File
@@ -1,28 +1,34 @@
"""Boardroom Map orchestrator web app.
Serves the control-panel UI and a small JSON API, and starts the background job
runner (see jobs.py) that actually convenes the panel. Most configuration happens
through the StartOS *actions* (Configure Sparks / Models / Reviewers / Review);
this UI is for dropping documents, triggering a review, watching it run, and
reading reports.
Serves the portfolio dashboard and a small JSON API, and starts the background
job runner (see jobs.py) that grades the dropped board decks. Most configuration
happens through the StartOS *actions* (Configure Sparks / Models / Graders /
Grading); this UI is for dropping decks per company, triggering a grading run,
watching it, and reading the per-company scorecard ledger.
"""
from __future__ import annotations
import os
import threading
from fastapi import FastAPI, HTTPException, UploadFile, File
from fastapi.responses import HTMLResponse, PlainTextResponse
from fastapi import FastAPI, File, Form, HTTPException, UploadFile
from fastapi.responses import HTMLResponse, JSONResponse, PlainTextResponse
from fastapi.templating import Jinja2Templates
from starlette.requests import Request
import bm_config
import extraction
import reviewers as rev_mod
import decks
import graders as grader_mod
import jobs
import ledger as ledger_mod
import serving
from jobs import runner, INBOX, REPORTS_DIR
DATA_DIR = os.environ.get("BM_DATA_DIR", "/data")
LEDGER_DIR = os.path.join(DATA_DIR, "ledger")
runner = jobs.runner
INBOX = getattr(jobs, "INBOX", os.path.join(DATA_DIR, "inbox"))
REPORTS_DIR = getattr(jobs, "REPORTS_DIR", os.path.join(DATA_DIR, "reports"))
app = FastAPI(title="Boardroom Map Orchestrator")
templates = Jinja2Templates(directory=os.path.join(os.path.dirname(__file__), "templates"))
@@ -53,19 +59,19 @@ def status():
"configured": {
"sparks": bool(cfg.get("primarySparkHost")),
"models": len(cfg.get("models") or []),
"reviewers": len(cfg.get("reviewers") or []),
"graders": len(cfg.get("graders") or []),
},
"networkMode": cfg.get("networkMode"),
"synthesis": bool(cfg.get("synthesisEnabled")),
"adjudicator": bool(cfg.get("adjudicatorEnabled")),
"wipeRemoteDocs": bool(cfg.get("wipeRemoteDocs")),
"autoRunOnDrop": bool(cfg.get("autoRunOnDrop")),
"models": [{"alias": m["alias"], "hfModel": m["hfModel"], "spark": m.get("spark", "primary")}
for m in (cfg.get("models") or [])],
"panel": [{"name": r.get("name"), "model": r.get("model"),
"persona": bool((r.get("persona") or "").strip()),
"known": (r.get("model") in catalog)}
for r in (cfg.get("reviewers") or [])],
"inbox": _inbox_list(),
"panel": [{"name": g.get("name"), "model": g.get("model"),
"persona": bool((g.get("persona") or "").strip()),
"known": (g.get("model") in catalog)}
for g in (cfg.get("graders") or [])],
"inbox": _inbox_grouped(),
"runtime": runner.snapshot(),
}
@@ -75,46 +81,89 @@ def events():
return {"events": runner.events()}
def _inbox_list() -> list[dict]:
if not os.path.isdir(INBOX):
return []
out = []
for fn in sorted(os.listdir(INBOX)):
p = os.path.join(INBOX, fn)
if os.path.isfile(p):
ext = os.path.splitext(fn)[1].lower()
out.append({"name": fn, "bytes": os.path.getsize(p),
"supported": ext in extraction.SUPPORTED})
return out
# ----------------------------------------------------------------------- inbox
def _inbox_grouped() -> dict:
"""Company-grouped inbox view: {companies: {slug: [file dicts]}, skipped: [...]}."""
try:
d = decks.discover(INBOX)
except Exception:
return {"companies": {}, "skipped": []}
companies: dict[str, list] = {}
for u in d.get("units") or []:
lst = companies.setdefault(u["company_slug"], [])
for f in u.get("files") or []:
try:
size = os.path.getsize(f)
except OSError:
size = 0
lst.append({"name": os.path.basename(f), "bytes": size,
"period": u.get("period"), "supported": True})
for fn in u.get("ignored") or []:
lst.append({"name": fn, "bytes": 0, "period": None, "supported": False})
# discover() drops units with no supported files, so sweep the company dirs
# for anything it didn't list (unsupported strays) and flag them.
try:
for entry in sorted(os.listdir(INBOX)):
cdir = os.path.join(INBOX, entry)
if entry.startswith(".") or not os.path.isdir(cdir):
continue
slug = decks.slugify(entry)
seen = {f["name"] for f in companies.get(slug, [])}
for fn in sorted(os.listdir(cdir)):
if fn.startswith(".") or fn in seen or not os.path.isfile(os.path.join(cdir, fn)):
continue
try:
size = os.path.getsize(os.path.join(cdir, fn))
except OSError:
size = 0
companies.setdefault(slug, []).append(
{"name": fn, "bytes": size, "period": decks.parse_period_from_name(fn),
"supported": os.path.splitext(fn)[1].lower() in decks.SUPPORTED_EXTS})
except OSError:
pass
return {"companies": companies, "skipped": d.get("skipped") or []}
@app.get("/api/inbox")
def inbox():
return {"inbox": _inbox_list()}
return _inbox_grouped()
# ----------------------------------------------------------------------- documents
@app.post("/api/upload")
async def upload(files: list[UploadFile] = File(...)):
os.makedirs(INBOX, exist_ok=True)
async def upload(request: Request,
files: list[UploadFile] = File(...),
company: str | None = Form(None)):
name = (company or request.query_params.get("company") or "").strip()
if not name:
raise HTTPException(400, "company is required — root-level files are not graded")
slug = decks.slugify(name)
dest_dir = os.path.join(INBOX, slug)
os.makedirs(dest_dir, exist_ok=True)
saved = []
for f in files:
name = os.path.basename(f.filename or "document")
dest = os.path.join(INBOX, name)
fn = os.path.basename(f.filename or "deck")
dest = os.path.join(dest_dir, fn)
with open(dest, "wb") as out:
while chunk := await f.read(1 << 20):
out.write(chunk)
saved.append(name)
return {"ok": True, "saved": saved}
saved.append(fn)
return {"ok": True, "company": slug, "saved": saved}
@app.post("/api/inbox/clear")
def inbox_clear():
import shutil
if os.path.isdir(INBOX):
for fn in os.listdir(INBOX):
p = os.path.join(INBOX, fn)
try:
if os.path.isfile(p):
os.remove(p)
elif os.path.isdir(p):
shutil.rmtree(p)
except OSError:
pass
return {"ok": True}
@@ -124,12 +173,16 @@ def run_now():
cfg = bm_config.load()
if not cfg.get("primarySparkHost"):
raise HTTPException(400, "No Spark configured (Configure Sparks).")
if not (cfg.get("models") and cfg.get("reviewers")):
raise HTTPException(400, "Configure at least one model and one reviewer first.")
if not _inbox_list():
raise HTTPException(400, "Inbox is empty — upload documents first.")
if not (cfg.get("models") and cfg.get("graders")):
raise HTTPException(400, "Configure at least one model and one grader first.")
try:
units = decks.discover(INBOX).get("units") or []
except Exception:
units = []
if not units:
raise HTTPException(400, "Inbox is empty — upload decks into a company folder first.")
runner.request_run()
return {"ok": True, "message": "Review requested — watch the activity log."}
return {"ok": True, "message": "Grading requested — watch the activity log."}
@app.get("/api/serving")
@@ -140,20 +193,21 @@ def serving_status():
return {"serving": serving.health(cfg)}
@app.post("/api/reviewer/build-image")
def build_reviewer_image():
@app.post("/api/grader/build-image")
def build_grader_image():
cfg = bm_config.load()
if not cfg.get("primarySparkHost"):
raise HTTPException(400, "No Spark configured.")
threading.Thread(target=lambda: _safe_build(cfg), daemon=True).start()
return {"ok": True, "message": "Building reviewer image on the head Spark — watch the activity log."}
return {"ok": True, "message": "Building grader image on the head Spark — watch the activity log."}
def _safe_build(cfg: dict):
try:
rev_mod.ensure_reviewer_image(cfg, runner.log)
fn = getattr(grader_mod, "ensure_grader_image", None) or grader_mod.ensure_reviewer_image
fn(cfg, runner.log)
except Exception as e:
runner.log(f"[reviewers] image build failed: {e}")
runner.log(f"[graders] image build failed: {e}")
@app.post("/api/stop")
@@ -167,21 +221,182 @@ def stop():
return {"ok": True}
# ----------------------------------------------------------------------- reports
# ----------------------------------------------------------------------- companies / ledger
def _config_companies(cfg: dict) -> list[dict]:
"""Companies registered in the StartOS config (may have zero graded decks)."""
out = []
for c in (cfg.get("companies") or []):
if isinstance(c, dict):
name = (c.get("name") or c.get("company") or c.get("slug") or "").strip()
else:
name = str(c).strip()
if name:
out.append({"slug": decks.slugify(name), "name": name})
return out
def _ledger_companies() -> list[dict]:
try:
led = ledger_mod.Ledger(LEDGER_DIR)
return led.all_companies() or []
except Exception:
return []
@app.get("/api/companies")
def companies():
cfg = bm_config.load()
rows, seen = [], set()
for c in _ledger_companies():
try:
hist = [{"period": h.get("period"), "composite": h.get("composite")}
for h in (c.get("history") or [])]
latest = None
if hist:
delta = None
cur, prev = hist[-1]["composite"], (hist[-2]["composite"] if len(hist) >= 2 else None)
if isinstance(cur, (int, float)) and isinstance(prev, (int, float)):
delta = round(cur - prev, 2)
latest = {"period": hist[-1]["period"], "composite": cur, "delta": delta}
slug = c.get("slug") or decks.slugify(c.get("name") or "")
seen.add(slug)
rows.append({"slug": slug, "name": c.get("name") or slug,
"auto_created": bool(c.get("auto_created")),
"deck_count": len(hist), "latest": latest, "history": hist})
except Exception:
continue
for c in _config_companies(cfg):
if c["slug"] not in seen:
seen.add(c["slug"])
rows.append({"slug": c["slug"], "name": c["name"], "auto_created": False,
"deck_count": 0, "latest": None, "history": []})
rows.sort(key=lambda r: (r["name"] or "").lower())
return {"companies": rows}
def _kpi_hit_rate(records: list[dict]) -> dict:
"""Per canonical KPI across records (oldest→newest):
attempts, hits (credit>=0.999), streak of hits ending at the latest attempt,
last_credit and profitability tag."""
stats: dict[str, dict] = {}
for rec in records:
for k in (rec.get("kpi_results") or []):
cn = k.get("canonical_name") or k.get("name")
if not cn:
continue
s = stats.setdefault(cn, {"name": k.get("name") or cn, "attempts": 0, "hits": 0,
"streak": 0, "last_credit": None, "profitability": False})
if k.get("name"):
s["name"] = k["name"]
s["profitability"] = bool(k.get("profitability", s["profitability"]))
credit = k.get("credit")
if not isinstance(credit, (int, float)):
continue # no target matched — not an attempt
s["attempts"] += 1
s["last_credit"] = credit
if credit >= 0.999:
s["hits"] += 1
s["streak"] += 1
else:
s["streak"] = 0
return stats
def _categories_latest(records: list[dict]) -> dict:
if not records:
return {}
def cats(rec):
return ((rec.get("qual") or {}).get("categories")) or {}
latest = cats(records[-1])
prev = cats(records[-2]) if len(records) >= 2 else {}
out = {}
for cid in "ABCDEFGH":
cur = latest.get(cid)
if cur is None and prev.get(cid) is None:
continue
out[cid] = {"latest_adjusted": (cur or {}).get("adjusted"),
"previous_adjusted": (prev.get(cid) or {}).get("adjusted")}
return out
@app.get("/api/companies/{slug}")
def company_detail(slug: str):
slug = os.path.basename(slug)
company, records = None, []
try:
led = ledger_mod.Ledger(LEDGER_DIR)
company = led.get_company(slug)
if company is not None:
records = led.deck_records(slug) or []
except Exception:
company, records = None, []
if company is None:
cfg = bm_config.load()
match = next((c for c in _config_companies(cfg) if c["slug"] == slug), None)
if not match:
raise HTTPException(404, "no such company")
company = {"slug": slug, "name": match["name"], "auto_created": False,
"kpi_aliases": {}, "pinned_targets": [], "extracted_targets": {},
"history": []}
open_flags = (((records[-1].get("penalties") or {}).get("flags")) or []) if records else []
return {
"company": company,
"records": records,
"kpi_hit_rate": _kpi_hit_rate(records),
"categories_latest": _categories_latest(records),
"open_flags": open_flags,
}
@app.get("/api/companies/{slug}/scorecard", response_class=PlainTextResponse)
def company_scorecard(slug: str):
slug = os.path.basename(slug)
path = os.path.join(LEDGER_DIR, slug, "SCORECARD.md")
if not os.path.exists(path):
raise HTTPException(404, "no scorecard yet for this company")
return open(path, errors="replace").read()
@app.get("/api/companies/{slug}/decks/{deck_id}")
def deck_record(slug: str, deck_id: str):
slug, deck_id = os.path.basename(slug), os.path.basename(deck_id)
rec = None
try:
led = ledger_mod.Ledger(LEDGER_DIR)
rec = led.deck_record(slug, deck_id)
except Exception:
rec = None
if rec is None:
raise HTTPException(404, "no such deck record")
return JSONResponse(rec)
@app.get("/api/companies/{slug}/decks/{deck_id}/report", response_class=PlainTextResponse)
def deck_report(slug: str, deck_id: str):
slug, deck_id = os.path.basename(slug), os.path.basename(deck_id)
if deck_id.endswith(".md"):
deck_id = deck_id[:-3]
path = os.path.join(LEDGER_DIR, slug, "decks", f"{deck_id}.md")
if not os.path.exists(path):
raise HTTPException(404, "no report for this deck")
return open(path, errors="replace").read()
# ----------------------------------------------------------------------- reports (legacy job reports)
@app.get("/api/reports")
def list_reports():
if not os.path.isdir(REPORTS_DIR):
return {"reports": []}
jobs = sorted((d for d in os.listdir(REPORTS_DIR)
jobs_ = sorted((d for d in os.listdir(REPORTS_DIR)
if os.path.isdir(os.path.join(REPORTS_DIR, d))), reverse=True)
return {"reports": jobs}
return {"reports": jobs_}
@app.get("/api/report", response_class=PlainTextResponse)
def latest_report():
path = os.path.join(REPORTS_DIR, "latest.md")
if not os.path.exists(path):
return "(no report yet — drop documents in the inbox and run a review)"
return "(no report yet — drop decks in a company folder and run a grading job)"
return open(path, errors="replace").read().strip() or "(empty report)"
orchestrator/bdef.md
+103
View File
@@ -0,0 +1,103 @@
# Board Deck Evaluation Framework (BDEF v1.1)
Inch Wide, Mile Deep — Girdley traits integrated with Munger & Buffett principles.
You are grading a portfolio-company board deck. The deck should let an owner's
representative answer, with high confidence: Are incentives aligned with long-term
owners? Has management inverted the problem and built in margin of safety? Are they
inside (and rationally expanding) their circle of competence? Is capital allocated
with owner-like patience, or is activity masquerading as progress? Would this
company survive a Lollapalooza of bad incentives, biases, and external shocks?
Score each category 1–5. A score above or below 3 REQUIRES verbatim evidence
quotes from the deck. Judge what the deck actually shows — absence of evidence on
a category is itself information (score 2–3 with the absence noted, not a guess).
## A. Incentive Alignment & Skin in the Game
Probes: Does compensation/promotion demonstrably reward rational long-term capital
allocation and owner-like behavior? Visible misalignments (short-term bonus
weighting, option reloads, metrics that invite channel stuffing or earnings
management)? Does management have skin in the game that survives a multi-year
downturn? Munger test: if I changed the incentives, would behavior change predictably?
- 1: Incentives invisible or visibly perverse. 3: Headcount/engagement shown but no
comp structure or ownership data. 5: Comp, promotion criteria, and ownership shown
and clearly aligned with long-term owners.
## B. Inversion Discipline & Margin of Safety
Probes: Are plausible failure modes explicitly modeled for major initiatives and
forecasts? Visible conservatism in assumptions, capital buffers, competitive-response
planning? Does the deck show what the company would NOT do even if it looked attractive?
- 1: Only upside shown; hockey-stick forecasts with no falsifiers. 3: Generic risk
slide, no quantified margin of safety. 5: Explicit inversion — what breaks the
thesis, how much buffer exists, and pre-committed "we won't do X" boundaries.
## C. Circle of Competence & Rational Learning
Probes: Does management accurately describe the boundaries of what they know well?
Disciplined expansion of the circle rather than overreach into new areas? Is learning
from mistakes visible and systematic?
- 1: Confident claims in adjacencies with no demonstrated competence. 3: Competent in
core but boundaries unstated. 5: Explicit "we know / we don't know", postmortems,
and disciplined expansion criteria.
## D. Capital Allocation Quality
Probes: Is every significant capital decision framed as opportunity cost vs long-term
owner return (including returning capital)? Patience ("sit on your ass") vs activity
bias? Are buybacks, dividends, M&A, and reinvestment held to the same owner rigor?
- 1: Growth for its own sake; projects listed without expected returns. 3: Budgets
shown but no alternatives comparison. 5: Every major incremental dollar shown with
expected return vs alternatives, including the do-nothing/return-it option.
## E. Moat Durability & Competitive Reality
Probes: Is the moat described in specific, testable terms (cost, switching costs,
network effects, brand) rather than generic "great team" language? What is management
actively doing to widen/defend it, and which threats are acknowledged? Buffett test:
would an intelligent owner buy this business at a fair price today based on the
durability shown?
- 1: "Great team / huge TAM" hand-waving. 3: Moat named but not evidenced or
threatened realistically. 5: Specific, testable moat with widening actions and
honestly acknowledged threats.
## F. Psychological & Cultural Health
Probes: Does the deck's framing reward early surfacing of problems, or filter
information upward? Evidence of Lollapalooza effects (multiple biases/misaligned
incentives compounding)? Does "no drama" reflect genuine psychological safety or
suppressed dissent? Do problem employees move on quickly; do values drive hiring/firing?
- 1: Only good news; problems appear late and pre-spun. 3: Engagement scores without
bad-news examples. 5: Bad news travels fast and visibly; the deck itself surfaces
problems early with owners' candor.
## G. Simplicity, Clarity & Decision Velocity
Probes: Does the deck avoid unnecessary complexity ("simple stays simple")? Are
repeatable processes and decision frameworks visible, or is the company reliant on
heroic individual effort? Is the board asked to judge the few things that matter
enormously rather than many that matter little?
- 1: Impressively complex deck obscuring weak economics. 3: Clear but unfocused.
5: A model of clarity an intelligent owner could absorb in one sitting, focused on
the 2–3 decisions that matter.
## H. Board Value-Add & Governance Quality
Probes: Does the deck position the board to pull (high-leverage questions on
incentives, inversion, capital allocation, moat) rather than rubber-stamp? Evidence
the board functions as owners' representatives rather than management's advisors?
Clear asks with recommendations and the inversion of those decisions?
- 1: No asks, or trivia; board presides rather than governs. 3: Asks listed without
recommendation or inversion. 5: The few decisions that matter, each with a clear
recommendation and what would make it wrong.
## Red-flag taxonomy
Use these codes (severity 1–5; suggest severity per guidance):
- `adjusted_metrics` (2–4): heavy reliance on adjusted/non-GAAP numbers without bridges.
- `metric_redefinition` (3–5): a KPI's definition changed between periods.
- `kpi_dropped` (2–3): a previously reported KPI silently disappeared.
- `hockey_stick_forecast` (2–4): forecast with no inversion or margin of safety.
- `channel_stuffing_risk` (3–5): incentives/metrics that invite pull-forward behavior.
- `short_term_comp` (2–4): compensation heavily weighted to short-term outcomes.
- `related_party` (3–5): related-party transactions or conflicts.
- `governance_gap` (2–4): big questions (succession, major bets, incentive redesign) get superficial treatment while minutiae fill the deck.
- `cash_runway_silence` (3–5): cash/runway/burn not clearly disclosed.
- `no_profitability_visibility` (3): no profit/margin/cash KPI reported at all.
- `overreach_adjacency` (2–4): confident expansion outside demonstrated competence.
- `activity_bias` (2–3): busy project lists without linkage to moat or owner returns.
- `complexity_smokescreen` (2–4): complexity that appears designed to obscure economics.
- `suppressed_dissent` (3–5): signs bad news is filtered before reaching the board.
Do NOT compute totals or a composite score. Numbers are computed elsewhere.
orchestrator/bm_config.py
+45 -15
View File
@@ -12,13 +12,23 @@ import spark_client as sc
DATA_DIR = os.environ.get("BM_DATA_DIR", "/data")
HF_TOKEN_PATH = os.path.join(DATA_DIR, "secrets", "hf_token")
BDEF_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "bdef.md")
DEFAULT_RUBRIC = (
"Review the attached document(s). Produce a structured report: a 3-5 sentence "
"summary, the key findings and insights, risks or red flags, open questions, "
"and concrete recommendations. Cite the document and section for each point. "
"Be honest about uncertainty; never invent facts not present in the documents."
)
# Scoring weights (composite 0-100 = quant 60 + qual 40 - penalties).
# Every knob the deterministic scorer uses lives here so the user can retune
# without a rebuild. Keep flat: StartOS action inputs are flat number fields.
WEIGHTS_DEFAULTS = {
"profitabilityKpi": 30, # profitability KPI attainment bucket
"otherKpi": 20, # non-profitability measurable KPI bucket
"forecastIntegrity": 10, # deck N actuals vs deck N-1 stated targets
"qualCategoryMax": 5, # each BDEF category A-H maxes at this (8x5=40)
"redFlagCap": 15, # max total penalty
"kpiCreditFloor": 0.5, # actual/target ratio below which credit = 0
"droppedKpiPenalty": 2, # severity of a KPI that silently disappeared
"droppedKpiMax": 3, # count at most this many dropped-KPI flags
"evidenceFullCredit": 400, # quote chars for full qualitative weight
"singleSourceFlagFactor": 0.5, # damping for flags raised by one source only
}
CONFIG_DEFAULTS = {
# Spark connection
@@ -39,22 +49,29 @@ CONFIG_DEFAULTS = {
"proxyPort": 4000,
"maxConcurrentModels": 1,
"models": [
{"alias": "reviewer-a", "hfModel": "Qwen/Qwen3-32B-FP8", "spark": "primary", "port": 8001},
{"alias": "grader-a", "hfModel": "Qwen/Qwen3-32B-FP8", "spark": "primary", "port": 8001},
],
# Review panel
"reviewers": [
{"name": "reviewer-1", "model": "reviewer-a", "persona": "", "temperature": None},
# Grading panel
"graders": [
{"name": "munger-lens", "model": "grader-a", "persona": "", "temperature": None},
],
# Review job settings
"reviewInstructions": DEFAULT_RUBRIC,
# Which catalog model runs the stage-1 structured extractor ("" = first model)
"extractorModel": "",
# Grading job settings
"bdefOverride": "", # non-empty replaces the baked-in bdef.md rubric
"weights": dict(WEIGHTS_DEFAULTS),
"networkMode": "airgapped",
"searxngUrl": "",
"synthesisEnabled": True,
"synthesisModel": "",
"synthesisPersona": "",
"adjudicatorEnabled": True,
"adjudicatorModel": "",
"adjudicatorPersona": "",
"wipeRemoteDocs": True,
"autoRunOnDrop": False,
"networkName": "boardroom-net",
# Portfolio companies (authoritative source of pinned targets / aliases).
# pinnedTargets: [{kpi, target, unit, direction: gte|lte, profitability}]
# kpiAliases: newline-separated "canonical=alias1;alias2" lines.
"companies": [],
# Flags
"hfTokenSet": False,
}
@@ -68,9 +85,22 @@ def load() -> dict:
except FileNotFoundError:
return merged
merged.update({k: v for k, v in saved.items() if v is not None})
# weights merge key-by-key so a partially-saved weights object keeps defaults
w = dict(WEIGHTS_DEFAULTS)
w.update({k: v for k, v in (merged.get("weights") or {}).items() if v is not None})
merged["weights"] = w
return merged
def bdef_text(cfg: dict) -> str:
"""The grading rubric: config override if set, else the baked-in spec."""
override = (cfg.get("bdefOverride") or "").strip()
if override:
return override
with open(BDEF_PATH, encoding="utf-8") as f:
return f.read()
def hf_token() -> str | None:
if os.path.exists(HF_TOKEN_PATH):
t = open(HF_TOKEN_PATH).read().strip()
orchestrator/decks.py
+122
View File
@@ -0,0 +1,122 @@
"""Deck discovery — map the inbox onto (company, period) grading units.
The inbox is organized by company: /data/inbox/<company>/<deck files>. The
subfolder name is the company slug; the reporting period is parsed from each
filename (2026-Q2, Q2 2026, 2026-H1, 2026-05, FY2026 ...). Files whose period
cannot be parsed form a period-less unit that the extractor's own deck.period
can later fill in. Files at the inbox root are skipped (we would not know the
company) and reported so the UI can nag the operator.
"""
from __future__ import annotations
import os
import re
SUPPORTED_EXTS = {".pdf", ".pptx", ".docx", ".txt", ".md", ".text"}
# Canonical period forms: "2026-Q2", "2026-H1", "2026-05", "FY2026".
_PERIOD_PATTERNS = [
# 2026-Q2 / 2026Q2 / 2026_Q2 / 2026 Q2
(re.compile(r"(?<!\d)((?:19|20)\d{2})[-_ ]?[Qq]([1-4])(?!\d)"),
lambda m: f"{m.group(1)}-Q{m.group(2)}"),
# Q2-2026 / Q2_2026 / Q2 2026
(re.compile(r"(?<![A-Za-z0-9])[Qq]([1-4])[-_ ]((?:19|20)\d{2})(?!\d)"),
lambda m: f"{m.group(2)}-Q{m.group(1)}"),
# 2026-H1 / 2026H2
(re.compile(r"(?<!\d)((?:19|20)\d{2})[-_ ]?[Hh]([12])(?!\d)"),
lambda m: f"{m.group(1)}-H{m.group(2)}"),
# FY2026 / FY-2026 / FY 2026
(re.compile(r"(?<![A-Za-z0-9])[Ff][Yy][-_ ]?((?:19|20)\d{2})(?!\d)"),
lambda m: f"FY{m.group(1)}"),
# 2026-05 (month 01-12; checked last so Q/H/FY forms win)
(re.compile(r"(?<!\d)((?:19|20)\d{2})[-_](0[1-9]|1[0-2])(?!\d)"),
lambda m: f"{m.group(1)}-{m.group(2)}"),
]
# Granularity ranks break ties between periods starting the same month
# (coarser first: FY2026 < 2026-H1 < 2026-Q1 < 2026-01).
_GRAN_FY, _GRAN_H, _GRAN_Q, _GRAN_M = 0, 1, 2, 3
_UNKNOWN_KEY = (9999, 99, 9)
def slugify(name: str) -> str:
"""Lowercase [a-z0-9-] slug for a company folder name."""
s = re.sub(r"[^a-z0-9]+", "-", (name or "").strip().lower()).strip("-")
return s or "company"
def parse_period_from_name(filename: str) -> str | None:
"""Canonical period parsed from a filename, or None."""
base = os.path.basename(filename)
for pat, canon in _PERIOD_PATTERNS:
m = pat.search(base)
if m:
return canon(m)
return None
def period_sort_key(period: str | None) -> tuple:
"""(year, start_month, granularity_rank); unknown/None sorts last."""
if not period:
return _UNKNOWN_KEY
m = re.fullmatch(r"((?:19|20)\d{2})-Q([1-4])", period)
if m:
return (int(m.group(1)), (int(m.group(2)) - 1) * 3 + 1, _GRAN_Q)
m = re.fullmatch(r"((?:19|20)\d{2})-H([12])", period)
if m:
return (int(m.group(1)), (int(m.group(2)) - 1) * 6 + 1, _GRAN_H)
m = re.fullmatch(r"((?:19|20)\d{2})-(0[1-9]|1[0-2])", period)
if m:
return (int(m.group(1)), int(m.group(2)), _GRAN_M)
m = re.fullmatch(r"FY((?:19|20)\d{2})", period)
if m:
return (int(m.group(1)), 1, _GRAN_FY)
return _UNKNOWN_KEY
def discover(inbox_dir: str) -> dict:
"""Scan the inbox into grading units.
Returns {"units": [{company_slug, period, period_source, files, ignored}],
"skipped": [root-level file names]}. Units are grouped by (company, parsed
period), sorted by company then oldest period first (period-less last)."""
units: dict[tuple, dict] = {}
skipped: list[str] = []
if not os.path.isdir(inbox_dir):
return {"units": [], "skipped": []}
for entry in sorted(os.listdir(inbox_dir)):
if entry.startswith("."):
continue
path = os.path.join(inbox_dir, entry)
if os.path.isfile(path):
skipped.append(entry)
continue
if not os.path.isdir(path):
continue
company = slugify(entry)
for fn in sorted(os.listdir(path)):
if fn.startswith("."):
continue
fpath = os.path.join(path, fn)
if not os.path.isfile(fpath):
continue
period = parse_period_from_name(fn)
key = (company, period_sort_key(period), period)
unit = units.setdefault(key, {
"company_slug": company,
"period": period,
"period_source": "filename" if period else "unknown",
"files": [],
"ignored": [],
})
ext = os.path.splitext(fn)[1].lower()
if ext in SUPPORTED_EXTS:
unit["files"].append(os.path.abspath(fpath))
else:
unit["ignored"].append(fn)
out = [u for _, u in sorted(units.items(), key=lambda kv: (kv[0][0], kv[0][1]))
if u["files"]]
for u in out:
u["files"].sort()
u["ignored"].sort()
return {"units": out, "skipped": skipped}
orchestrator/extraction.py
+88 -9
View File
@@ -5,6 +5,7 @@ to the Sparks, we extract plain text here so that only normalized text (never th
original binaries) crosses to the review containers. Supported formats:
.pdf -> pypdf
.pptx -> python-pptx (text frames, tables, chart data, notes)
.docx -> python-docx
.txt .md .text -> read as UTF-8
@@ -17,7 +18,7 @@ from __future__ import annotations
import os
TEXT_EXTS = {".txt", ".md", ".text", ".markdown"}
SUPPORTED = TEXT_EXTS | {".pdf", ".docx"}
SUPPORTED = TEXT_EXTS | {".pdf", ".docx", ".pptx"}
def _extract_pdf(path: str) -> str:
@@ -48,6 +49,73 @@ def _extract_docx(path: str) -> str:
return "\n".join(lines).strip()
def _pptx_chart_text(shape) -> list[str]:
"""Best-effort chart title + series names/values (chart XML varies wildly)."""
lines: list[str] = []
try:
chart = shape.chart
try:
if chart.has_title and chart.chart_title.has_text_frame:
lines.append(f"[chart] {chart.chart_title.text_frame.text}")
except Exception:
pass
for plot in chart.plots:
try:
cats = [str(c) for c in (plot.categories or [])]
except Exception:
cats = []
for series in plot.series:
try:
name = str(series.name)
except Exception:
name = "(series)"
try:
vals = ["" if v is None else f"{v:g}" for v in series.values]
except Exception:
vals = []
if cats and len(cats) == len(vals):
pairs = ", ".join(f"{c}={v}" for c, v in zip(cats, vals))
else:
pairs = ", ".join(vals)
lines.append(f"[chart series] {name}: {pairs}")
except Exception:
pass
return lines
def _extract_pptx(path: str) -> str:
from pptx import Presentation
prs = Presentation(path)
parts: list[str] = []
for n, slide in enumerate(prs.slides, 1):
body: list[str] = []
for shape in slide.shapes:
if getattr(shape, "has_text_frame", False):
txt = shape.text_frame.text.strip()
if txt:
body.append(txt)
if getattr(shape, "has_table", False):
for row in shape.table.rows:
cells = [c.text.strip() for c in row.cells]
if any(cells):
body.append(" | ".join(cells))
if getattr(shape, "has_chart", False):
body.extend(_pptx_chart_text(shape))
notes = ""
try:
if slide.has_notes_slide:
notes = (slide.notes_slide.notes_text_frame.text or "").strip()
except Exception:
pass
if notes:
body.append(f"--- notes ---\n{notes}")
if len("".join(body)) < 20:
body.append(f"[low_text_slide: slide {n}]")
parts.append(f"\n\n===== slide {n} =====\n" + "\n".join(body))
return "".join(parts).strip()
def _extract_text(path: str) -> str:
with open(path, errors="replace") as f:
return f.read().strip()
@@ -57,6 +125,8 @@ def extract_file(path: str) -> str:
ext = os.path.splitext(path)[1].lower()
if ext == ".pdf":
return _extract_pdf(path)
if ext == ".pptx":
return _extract_pptx(path)
if ext == ".docx":
return _extract_docx(path)
if ext in TEXT_EXTS:
@@ -70,20 +140,16 @@ def _safe_name(name: str) -> str:
return (keep or "document").replace(" ", "_")
def extract_inbox(inbox: str, out_dir: str, log=print) -> list[dict]:
"""Extract every supported file in `inbox` to a .txt in `out_dir`.
def extract_files(files: list[str], out_dir: str, log=print) -> list[dict]:
"""Extract an explicit list of files (a deck unit) to .txt files in `out_dir`.
Returns a manifest: [{source, out, chars, ok, error}]. Skips unsupported
files (recorded with ok=False) rather than failing the whole job."""
os.makedirs(out_dir, exist_ok=True)
manifest: list[dict] = []
if not os.path.isdir(inbox):
return manifest
seen: dict[str, int] = {}
for fn in sorted(os.listdir(inbox)):
src = os.path.join(inbox, fn)
if not os.path.isfile(src):
continue
for src in files:
fn = os.path.basename(src)
ext = os.path.splitext(fn)[1].lower()
rec = {"source": fn, "out": None, "chars": 0, "ok": False, "error": ""}
if ext not in SUPPORTED:
@@ -113,3 +179,16 @@ def extract_inbox(inbox: str, out_dir: str, log=print) -> list[dict]:
log(f"[extract] {fn} -> {out_name} ({len(text)} chars)")
manifest.append(rec)
return manifest
def extract_inbox(inbox: str, out_dir: str, log=print) -> list[dict]:
"""Extract every supported file in `inbox` to a .txt in `out_dir`.
Returns a manifest: [{source, out, chars, ok, error}]. Skips unsupported
files (recorded with ok=False) rather than failing the whole job."""
if not os.path.isdir(inbox):
os.makedirs(out_dir, exist_ok=True)
return []
files = [os.path.join(inbox, fn) for fn in sorted(os.listdir(inbox))
if os.path.isfile(os.path.join(inbox, fn))]
return extract_files(files, out_dir, log)
orchestrator/graders.py
+158 -82
View File
@@ -1,67 +1,87 @@
"""Launch the reviewer panel on the head Spark over SSH.
"""Launch the grading panel (and the stage-1 extractor) on the head Spark.
Each reviewer is a ONE-SHOT, hardened, read-only container: it reads the document
text mounted at /docs, runs its model (through the on-Spark proxy) under its
persona + the shared rubric, writes a single report to /out/<id>.md, and exits.
There is no shared writable workspace and no git — reviewers cannot alter the
documents or each other's reports.
Each role agent is a ONE-SHOT, hardened, read-only container running
sandbox/grader_agent.py: it reads the deck text mounted at /docs, runs its model
(through the on-Spark proxy) under its persona + the BDEF rubric, writes exactly
one output file to /out, and exits. There is no tool loop and no shared writable
workspace — agents cannot alter the deck text or each other's reports.
Per-deck mount contract (remote paths under {remoteWorkDir}/jobs/<job>/<company>/<deck>):
docs/ -> /docs:ro
BDEF.md -> /BDEF.md:ro
schemas/<role>.schema.json -> /schema.json:ro (extractor|grades)
personas/<rid>.md -> /persona/PERSONA.md:ro
out/ -> /out:rw
Sandbox (per the operator's confidentiality requirement):
* non-root, --cap-drop ALL, --security-opt no-new-privileges
* read-only rootfs + small writable tmpfs; only /docs (ro), /persona (ro), and
/out (rw, this reviewer's report dir) are mounted
* read-only rootfs + small writable tmpfs (/tmp, /home/rev)
* NO docker socket, cpu/mem/pid caps
* attached to the per-job network — in airgapped mode that network is
--internal, so the reviewer can reach ONLY the model proxy, never the internet
--internal, so the agent can reach ONLY the model proxy, never the internet
Reviewers hold no credentials beyond a dummy proxy key.
Agents hold no credentials beyond a dummy proxy key.
"""
from __future__ import annotations
import os
import re
import shlex
import shutil
import spark_client as sc
import bm_config
import prompts
import serving
import spark_client as sc
SANDBOX_SRC = os.path.join(os.path.dirname(os.path.abspath(__file__)), "sandbox")
ORCH_DIR = os.path.dirname(os.path.abspath(__file__))
SANDBOX_SRC = os.path.join(ORCH_DIR, "sandbox")
SCHEMAS_SRC = os.path.join(ORCH_DIR, "schemas")
SCHEMA_FILES = ("extraction.schema.json", "grades.schema.json")
HARDEN = (
"--user 1000:1000 --security-opt no-new-privileges --cap-drop ALL "
"--read-only --tmpfs /tmp:size=256m --tmpfs /home/rev:size=128m "
"--pids-limit 256 --memory 6g --cpus 4"
)
def ensure_reviewer_image(cfg: dict, log) -> None:
"""Build the reviewer image on the head Spark if missing (aarch64, native)."""
# ------------------------------------------------------------------ image
def ensure_grader_image(cfg: dict, log) -> None:
"""Build the grader image on the head Spark if missing (aarch64, native)."""
head = sc.head(cfg)
image = cfg["graderImage"]
q = shlex.quote
r = sc.run(head, f"docker image inspect {q(image)} >/dev/null 2>&1 && echo PRESENT || echo MISSING",
timeout=30)
if "PRESENT" in (r.stdout or ""):
log(f"[reviewers] image {image} already present on {head.host}")
log(f"[graders] image {image} already present on {head.host}")
return
if not os.path.isdir(SANDBOX_SRC):
raise RuntimeError(f"reviewer build context missing at {SANDBOX_SRC} (image not baked in?)")
raise RuntimeError(f"grader build context missing at {SANDBOX_SRC} (image not baked in?)")
remote_dir = f"{cfg['remoteWorkDir']}/sandbox-build"
log(f"[reviewers] building reviewer image {image} on {head.host} (first run; a few minutes)…")
log(f"[graders] building grader image {image} on {head.host} (first run; a few minutes)…")
push = sc.push_dir(head, SANDBOX_SRC, remote_dir)
if push.returncode != 0:
raise RuntimeError(f"rsync reviewer build context to {head.host} failed: {push.stderr}")
raise RuntimeError(f"rsync grader build context to {head.host} failed: {push.stderr}")
b = sc.run(head, f"cd {q(remote_dir)} && IMAGE={q(image)} bash build.sh", timeout=1800)
if b.returncode != 0:
raise RuntimeError(f"reviewer image build failed on {head.host}: {b.stderr or b.stdout}")
log(f"[reviewers] reviewer image built: {image}")
raise RuntimeError(f"grader image build failed on {head.host}: {b.stderr or b.stdout}")
log(f"[graders] grader image built: {image}")
# ------------------------------------------------------------------ roster
def slug(name: str) -> str:
s = re.sub(r"[^A-Za-z0-9]+", "-", (name or "").strip().lower()).strip("-")
return s or "reviewer"
return s or "grader"
def roster(cfg: dict) -> list[dict]:
"""Reviewer roster from config. Each: {rid, name, model alias, persona, temperature}."""
"""Grader roster from config. Each: {rid, name, model alias, persona, temperature}."""
out: list[dict] = []
seen: dict[str, int] = {}
for w in (cfg.get("reviewers") or []):
name = (w.get("name") or "reviewer").strip()
for w in (cfg.get("graders") or []):
name = (w.get("name") or "grader").strip()
rid = slug(name)
if rid in seen:
seen[rid] += 1
@@ -76,87 +96,143 @@ def roster(cfg: dict) -> list[dict]:
return out
def _container(cfg: dict, jobdir: str, rid: str, name: str, model: str, persona: str,
temperature, role: str, extra_mounts: str = "") -> str:
"""docker run command for one reviewer/synthesizer container (detached, one-shot)."""
q = shlex.quote
net = serving.net_name(cfg)
base = serving.reviewer_proxy_base(cfg)
searxng = (cfg.get("searxngUrl") or "").strip()
persona_path = f"{jobdir}/personas/{rid}.md"
# ------------------------------------------------------------------ staging
def stage_deck_files(cfg: dict, local_deck_dir: str, panel: list[dict]) -> None:
"""Write everything the role containers need into the LOCAL per-deck staging
dir (jobs.py rsyncs the whole dir to the Spark afterwards): the BDEF rubric,
both schemas, and one persona file per role agent (extractor + each grader +
adjudicator). Also pre-creates out/ and adjudicator-out/ so the rsync creates
them remotely with the SSH user's ownership (uid 1000 = the container user)."""
for sub in ("personas", "schemas", "out", "adjudicator-out"):
os.makedirs(os.path.join(local_deck_dir, sub), exist_ok=True)
with open(os.path.join(local_deck_dir, "BDEF.md"), "w") as f:
f.write(bm_config.bdef_text(cfg))
for fn in SCHEMA_FILES:
shutil.copyfile(os.path.join(SCHEMAS_SRC, fn),
os.path.join(local_deck_dir, "schemas", fn))
def persona(rid: str, text: str) -> None:
with open(os.path.join(local_deck_dir, "personas", f"{rid}.md"), "w") as f:
f.write((text or "").strip() + "\n")
persona("extractor", prompts.extractor_persona())
for r in panel:
persona(r["rid"], r["persona"] or prompts.default_grader_persona(r["name"]))
persona("adjudicator",
(cfg.get("adjudicatorPersona") or "").strip() or prompts.adjudicator_persona())
# ------------------------------------------------------------------ containers
def container_suffix(deckdir: str) -> str:
"""Short, docker-safe name suffix from the deck dir (…/<company>/<deck_id>)."""
parts = deckdir.rstrip("/").split("/")
tail = "-".join(parts[-2:]) if len(parts) >= 2 else parts[-1]
return slug(tail)[:48] or "deck"
def base_env(cfg: dict, rid: str, name: str, role: str, model: str, temperature) -> str:
"""The env-var block shared by every role container (also used by adjudicator.py)."""
q = shlex.quote
base = serving.reviewer_proxy_base(cfg)
env = (
f"-e BM_REVIEWER_ID={q(rid)} -e BM_REVIEWER_NAME={q(name)} -e BM_ROLE={q(role)} "
f"-e BM_ROLE={q(role)} -e BM_GRADER_ID={q(rid)} -e BM_GRADER_NAME={q(name)} "
f"-e BM_MODEL={q(model)} -e BM_LLM_BASE={q(base)} -e BM_LLM_KEY=sk-local "
f"-e BM_MAX_MODEL_LEN={q(str(int(cfg.get('maxModelLen', 32768))))} "
f"-e HOME=/home/rev "
)
if temperature is not None:
if role == "extractor":
env += "-e BM_TEMPERATURE=0.0 "
elif temperature is not None:
env += f"-e BM_TEMPERATURE={q(str(temperature))} "
# web_search is offered ONLY in local_services mode with a SearXNG URL.
if cfg.get("networkMode") == "local_services" and searxng:
env += f"-e BM_SEARXNG_URL={q(searxng)} "
harden = (
"--user 1000:1000 --security-opt no-new-privileges --cap-drop ALL "
"--read-only --tmpfs /tmp:size=256m --tmpfs /home/rev:size=128m "
"--pids-limit 256 --memory 6g --cpus 4"
)
mounts = (
f"-v {q(jobdir)}/docs:/docs:ro "
f"-v {q(jobdir)}/out:/out "
f"-v {q(persona_path)}:/persona/PERSONA.md:ro "
+ extra_mounts
)
cname = f"bm-grader-{rid}"
return (
f"docker rm -f {cname} >/dev/null 2>&1; "
f"docker run -d --name {cname} --network {q(net)} {harden} {env} {mounts} {q(cfg['graderImage'])}"
)
return env
def _write_persona(cfg: dict, jobdir: str, rid: str, persona: str) -> None:
def _container(cfg: dict, deckdir: str, rid: str, name: str, model: str,
temperature, role: str) -> tuple[str, str]:
"""(container name, docker run command) for one extractor/grader container."""
q = shlex.quote
sc.run(sc.head(cfg),
f"mkdir -p {q(jobdir)}/personas && printf '%s' {q(persona)} > {q(jobdir)}/personas/{rid}.md",
timeout=30)
net = serving.net_name(cfg)
schema_file = "extraction.schema.json" if role == "extractor" else "grades.schema.json"
env = base_env(cfg, rid, name, role, model, temperature)
mounts = (
f"-v {q(deckdir)}/docs:/docs:ro "
f"-v {q(deckdir)}/BDEF.md:/BDEF.md:ro "
f"-v {q(deckdir)}/schemas/{schema_file}:/schema.json:ro "
f"-v {q(deckdir)}/personas/{rid}.md:/persona/PERSONA.md:ro "
f"-v {q(deckdir)}/out:/out "
)
cname = f"bm-grader-{rid}-{container_suffix(deckdir)}"
cmd = (
f"docker rm -f {cname} >/dev/null 2>&1; "
f"docker run -d --name {cname} --network {q(net)} {HARDEN} {env} {mounts} "
f"{q(cfg['graderImage'])}"
)
return cname, cmd
def run_wave_reviewers(cfg: dict, jobdir: str, panel: list[dict], rubric: str, log,
wait_timeout: int = 1800) -> list[dict]:
"""Launch every reviewer in `panel` (already filtered to this wave's models),
wait for them to finish, and report status. Reports land in <jobdir>/out."""
def _wait_for(cfg: dict, cname: str, out_file: str, log, wait_timeout: int) -> tuple[str, bool]:
"""docker-wait a launched container, check its output file, remove it."""
head = sc.head(cfg)
q = shlex.quote
# Rubric is shared; write it once into the job dir, mounted into every container.
sc.run(head, f"mkdir -p {q(jobdir)}/out && printf '%s' {q(rubric)} > {q(jobdir)}/RUBRIC.md", timeout=30)
w = sc.run(head, f"docker wait {cname}", timeout=wait_timeout)
code = (w.stdout or "").strip()
chk = sc.run(head, f"test -s {q(out_file)} && echo OK || echo MISSING", timeout=30)
wrote = "OK" in (chk.stdout or "")
sc.run(head, f"docker rm -f {cname} 2>/dev/null; true", timeout=30)
return code, wrote
# ------------------------------------------------------------------ runs
def run_extractor(cfg: dict, remote_deck_dir: str, model_alias: str, log,
wait_timeout: int = 1800) -> dict:
"""Launch the stage-1 extractor for one deck and wait for out/extraction.json."""
head = sc.head(cfg)
cname, cmd = _container(cfg, remote_deck_dir, "extractor", "extractor",
model_alias, None, role="extractor")
r = sc.run(head, cmd, timeout=120)
if r.returncode != 0:
log(f"[graders] extractor launch FAILED: {r.stderr or r.stdout}")
return {"rid": "extractor", "model": model_alias, "ok": False,
"error": (r.stderr or r.stdout)[:300], "report": False}
log(f"[graders] up: extractor -> {model_alias}")
code, wrote = _wait_for(cfg, cname, f"{remote_deck_dir}/out/extraction.json",
log, wait_timeout)
log(f"[graders] extractor exited (code={code or '?'}), "
f"extraction.json={'written' if wrote else 'MISSING'}")
return {"rid": "extractor", "model": model_alias, "ok": True, "error": "",
"exit": code, "report": wrote}
def run_wave_graders(cfg: dict, remote_deck_dir: str, wave_panel: list[dict], log,
wait_timeout: int = 1800) -> list[dict]:
"""Launch every grader in `wave_panel` (already filtered to this wave's
models) against one deck, wait for them, and report status. Grade JSONs land
in <remote_deck_dir>/out/<rid>.json."""
head = sc.head(cfg)
launched = []
for r in panel:
_write_persona(cfg, jobdir, r["rid"], r["persona"])
cmd = _container(cfg, jobdir, r["rid"], r["name"], r["model"], r["persona"],
r.get("temperature"), role="reviewer",
extra_mounts=f"-v {q(jobdir)}/RUBRIC.md:/RUBRIC.md:ro ")
for r in wave_panel:
cname, cmd = _container(cfg, remote_deck_dir, r["rid"], r["name"], r["model"],
r.get("temperature"), role="grader")
res = sc.run(head, cmd, timeout=120)
if res.returncode != 0:
log(f"[reviewers] launch {r['rid']} FAILED: {res.stderr or res.stdout}")
launched.append({**r, "ok": False, "error": (res.stderr or res.stdout)[:300]})
log(f"[graders] launch {r['rid']} FAILED: {res.stderr or res.stdout}")
launched.append({**r, "ok": False, "error": (res.stderr or res.stdout)[:300],
"cname": cname})
continue
log(f"[reviewers] up: {r['rid']} -> {r['model']}")
launched.append({**r, "ok": True, "error": ""})
log(f"[graders] up: {r['rid']} -> {r['model']}")
launched.append({**r, "ok": True, "error": "", "cname": cname})
# Wait for each launched container to exit (they run in parallel; waiting
# sequentially still finishes when the slowest does).
results = []
for r in launched:
if not r["ok"]:
results.append(r)
results.append({k: v for k, v in r.items() if k != "cname"})
continue
cname = f"bm-grader-{r['rid']}"
w = sc.run(head, f"docker wait {cname}", timeout=wait_timeout)
code = (w.stdout or "").strip()
out_check = sc.run(head, f"test -s {q(jobdir)}/out/{q(r['rid'])}.md && echo OK || echo MISSING", timeout=30)
wrote = "OK" in (out_check.stdout or "")
log(f"[reviewers] {r['rid']} exited (code={code or '?'}), report={'written' if wrote else 'MISSING'}")
sc.run(head, f"docker rm -f {cname} 2>/dev/null; true", timeout=30)
results.append({**r, "exit": code, "report": wrote})
code, wrote = _wait_for(cfg, r["cname"], f"{remote_deck_dir}/out/{r['rid']}.json",
log, wait_timeout)
log(f"[graders] {r['rid']} exited (code={code or '?'}), "
f"grades={'written' if wrote else 'MISSING'}")
results.append({k: v for k, v in r.items() if k != "cname"}
| {"exit": code, "report": wrote})
return results
orchestrator/jobs.py
+419 -147
View File
@@ -1,40 +1,61 @@
"""The Boardroom Map job runner — convenes the review panel over dropped documents.
"""The Boardroom Map job runner — grades dropped board decks against the BDEF.
Runs as a background thread inside the FastAPI app. It does NOT run on a clock
like Nightshift; it reacts to triggers:
Runs as a background thread inside the FastAPI app. It does NOT run on a clock;
it reacts to triggers:
* an explicit "Run Review" (drops /data/state/run_request), or
* autoRunOnDrop: files landing in /data/inbox, once the inbox is stable.
* an explicit "Grade Decks" run request (drops /data/state/run_request), or
* autoRunOnDrop: files landing under /data/inbox/<company-slug>/, once the
inbox is stable across two ticks.
One job at a time. A job:
1. extract text from the inbox locally (CPU) — only text crosses to the Sparks
2. rsync the text to a per-job dir on the head Spark
3. serve the needed models in WAVES; run the reviewers for each wave
4. optionally run the local lead-reviewer synthesis
5. pull the reports back to /data/reports/<job>, assemble latest.md
6. wipe the documents from the Sparks (unless disabled) and tear serving down
One job at a time. A job iterates the discovered deck units OLDEST FIRST per
company, and for each deck:
All state (phase, current job, per-reviewer status, last report) is mirrored to
1. extract text locally (CPU) — only text crosses to the Sparks
2. rsync the per-deck bundle (docs/, BDEF.md, personas/, schemas/, out/,
adjudicator-out/) to {remoteWorkDir}/jobs/<job>/<company>/<deck>/
3. serve the needed models in WAVES (graders' models ∪ the extractor model);
the extractor runs when its model's wave is up, graders in their waves
4. pull out/, validate: extraction.json invalid => the DECK fails (the job
continues); grader JSONs are validated individually, invalid ones dropped;
fewer than 2 valid grade reports => the deck fails
5. adjudicate (optional, non-fatal): a local model weighs the panel's evidence
6. score deterministically (scoring.score_deck) against the company's pinned
targets + the prior deck's forward targets, and record it in the ledger
7. render DECK_REPORT.md + refresh the company SCORECARD.md and the
/data/reports copies
Then it wipes the remote job dir (unless disabled), tears serving down, and
moves the graded originals to /data/processed/<job>/<slug>/ (the company folder
stays in the inbox for reuse). One deck's failure never kills the job: the job
ends "done" if at least one deck was graded.
All state (phase, per-deck status, panel status, last report) is mirrored to
/data/state/runtime.json so the Web UI can render it.
"""
from __future__ import annotations
import json
import os
import re
import shutil
import threading
import time
import traceback
from collections import deque
from datetime import datetime
from datetime import datetime, timezone
import adjudicator as adj_mod
import bm_config
import decks
import extraction
import graders as gr_mod
import ledger as ledger_mod
import preflight
import reviewers as rev_mod
import scorecard
import scoring
import serving
import spark_client as sc
import synthesis as synth_mod
import validate
DATA_DIR = os.environ.get("BM_DATA_DIR", "/data")
INBOX = os.path.join(DATA_DIR, "inbox")
@@ -42,40 +63,77 @@ PROCESSED = os.path.join(DATA_DIR, "processed")
STATE_DIR = os.path.join(DATA_DIR, "state")
JOBS_DIR = os.path.join(STATE_DIR, "jobs")
REPORTS_DIR = os.path.join(DATA_DIR, "reports")
LEDGER_DIR = os.path.join(DATA_DIR, "ledger")
RUNTIME_PATH = os.path.join(STATE_DIR, "runtime.json")
REQUEST_PATH = os.path.join(STATE_DIR, "run_request")
TICK_SECONDS = 10
RUNNING_PHASES = ("extracting", "grading", "adjudicating", "scoring", "collecting")
# Canonical-ish reporting periods: 2026-Q2, 2026-H1, FY2026, 2026-05, 2026.
_PERIOD_RE = re.compile(
r"^(?:FY\s?-?\d{4}|\d{4}(?:[-/ ]?(?:Q[1-4]|H[12]|0[1-9]|1[0-2]))?)$", re.IGNORECASE)
def _inbox_signature() -> tuple[int, str]:
"""(count, signature) of supported files in the inbox, for stability checks."""
"""(count, signature) of supported files anywhere in the inbox tree, for
autoRunOnDrop stability checks (decks live in per-company subfolders)."""
if not os.path.isdir(INBOX):
return (0, "")
items = []
for fn in sorted(os.listdir(INBOX)):
p = os.path.join(INBOX, fn)
if os.path.isfile(p) and os.path.splitext(fn)[1].lower() in extraction.SUPPORTED:
items.append(f"{fn}:{os.path.getsize(p)}:{int(os.path.getmtime(p))}")
for root, _dirs, files in os.walk(INBOX):
for fn in files:
p = os.path.join(root, fn)
if os.path.splitext(fn)[1].lower() in extraction.SUPPORTED:
try:
items.append(f"{os.path.relpath(p, INBOX)}:{os.path.getsize(p)}:"
f"{int(os.path.getmtime(p))}")
except OSError:
pass
items.sort()
return (len(items), "|".join(items))
def _token(s: str) -> str:
t = re.sub(r"[^a-z0-9]+", "-", (s or "").lower()).strip("-")
return t or "deck"
def _composite(record) -> float | None:
"""Best-effort composite lookup on the scoring record (shape owned by scoring.py)."""
if not isinstance(record, dict):
return None
for k in ("composite", "composite_score"):
v = record.get(k)
if isinstance(v, (int, float)):
return v
for parent in ("scores", "totals", "score"):
d = record.get(parent)
if isinstance(d, dict) and isinstance(d.get("composite"), (int, float)):
return d["composite"]
return None
class JobRunner:
def __init__(self):
self._events = deque(maxlen=500)
self._lock = threading.Lock()
self.phase = "idle" # idle | extracting | reviewing | synthesizing | collecting | done | error
self.phase = "idle" # idle | extracting | grading | adjudicating | scoring | collecting | done | error
self.job_id = None
self.message = ""
self.panel: list[dict] = []
self.waves_total = 0
self.wave_index = 0
self.decks_total = 0
self.deck_index = 0
self.company = None
self.period = None
self.decks: list[dict] = []
self.last_report_path = None
self._thread = None
self._last_sig = None
self._stable_sig = None
self._last_done_sig = None
for d in (STATE_DIR, JOBS_DIR, REPORTS_DIR, INBOX, PROCESSED):
for d in (STATE_DIR, JOBS_DIR, REPORTS_DIR, LEDGER_DIR, INBOX, PROCESSED):
os.makedirs(d, exist_ok=True)
self._restore()
@@ -108,11 +166,12 @@ class JobRunner:
self.job_id = d.get("job_id")
self.message = d.get("message", "")
self.panel = d.get("panel", [])
self.decks = d.get("decks", [])
self.last_report_path = d.get("last_report_path")
for e in d.get("events", []):
self._events.append(e)
# A job can't survive a restart; reset a stuck running phase.
if self.phase in ("extracting", "reviewing", "synthesizing", "collecting"):
if self.phase in RUNNING_PHASES:
self.phase = "idle"
except Exception:
pass
@@ -125,6 +184,11 @@ class JobRunner:
"panel": self.panel,
"waves_total": self.waves_total,
"wave_index": self.wave_index,
"decks_total": self.decks_total,
"deck_index": self.deck_index,
"company": self.company,
"period": self.period,
"decks": self.decks,
"last_report_path": self.last_report_path,
}
@@ -136,7 +200,7 @@ class JobRunner:
self._thread.start()
def request_run(self):
"""Public hook (used by the API) to request a review immediately."""
"""Public hook (used by the API) to request a grading run immediately."""
try:
with open(REQUEST_PATH, "w") as f:
f.write(str(time.time()))
@@ -160,23 +224,17 @@ class JobRunner:
if os.path.exists(REQUEST_PATH):
os.remove(REQUEST_PATH)
triggered = True
self.log("[runner] review requested")
self.log("[runner] grading run requested")
elif cfg.get("autoRunOnDrop"):
count, sig = _inbox_signature()
if count and sig == self._last_sig and sig != self._last_done_sig:
# stable across two ticks and not the batch we last processed
triggered = True
self.log("[runner] inbox stable — auto-running review")
self.log("[runner] inbox stable — auto-running grading")
self._last_sig = sig
if not triggered:
return
count, _ = _inbox_signature()
if not count:
self.log("[runner] nothing to review (inbox empty of supported files)")
self.phase = "idle"
self._persist()
return
self._run_job(cfg)
# ------------------------------------------------------------- the job
@@ -186,114 +244,124 @@ class JobRunner:
self.message = ""
self.waves_total = 0
self.wave_index = 0
self.decks_total = 0
self.deck_index = 0
self.company = None
self.period = None
self.decks = []
self.panel = []
local_job = os.path.join(JOBS_DIR, job_id)
local_docs = os.path.join(local_job, "docs")
remote_job = f"{cfg['remoteWorkDir'].rstrip('/')}/jobs/{job_id}"
rubric = cfg.get("reviewInstructions") or bm_config.DEFAULT_RUBRIC
self.log(f"=== Review job {job_id} begins ===")
remote_root = f"{cfg['remoteWorkDir'].rstrip('/')}/jobs/{job_id}"
self.log(f"=== Grading job {job_id} begins ===")
try:
# 1. Extract locally (only text crosses to the Sparks).
self.phase = "extracting"; self._persist()
manifest = extraction.extract_inbox(INBOX, local_docs, self.log)
ok_docs = [m for m in manifest if m["ok"]]
if not ok_docs:
raise RuntimeError("no documents could be extracted (unsupported or empty inbox)")
self.log(f"[runner] extracted {len(ok_docs)} document(s)")
# 1. Discover deck units (per-company folders; oldest first).
disc = decks.discover(INBOX)
units = disc.get("units") or []
for fn in disc.get("skipped") or []:
self.log(f"[runner] WARNING: skipping root-level inbox file '{fn}' — "
"decks belong in /data/inbox/<company-slug>/")
if not units:
raise RuntimeError("nothing to grade — drop decks into /data/inbox/<company-slug>/")
# 2. Resolve the panel against the model catalog.
catalog = {m["alias"] for m in (cfg.get("models") or [])}
panel = rev_mod.roster(cfg)
# 2. Ledger, merged with the configured portfolio companies.
led = ledger_mod.Ledger(LEDGER_DIR)
led.merge_config_companies(cfg.get("companies") or [])
# Resolve the grader panel + extractor model against the catalog.
models = cfg.get("models") or []
catalog = {m["alias"] for m in models}
panel = gr_mod.roster(cfg)
valid = [r for r in panel if r["model"] in catalog]
invalid = [r for r in panel if r["model"] not in catalog]
for r in invalid:
self.log(f"[runner] WARNING: reviewer '{r['name']}' uses unknown model '{r['model']}' — skipped")
if not valid:
raise RuntimeError("no reviewers reference a configured model (see Configure Models/Reviewers)")
self.panel = [{"name": r["name"], "model": r["model"], "status": "pending"} for r in valid]
for r in panel:
if r["model"] not in catalog:
self.log(f"[runner] WARNING: grader '{r['name']}' uses unknown model "
f"'{r['model']}' — skipped")
if len(valid) < 2:
raise RuntimeError(
"need at least 2 graders referencing configured models — every deck "
"requires >= 2 valid grade reports (see Configure Models/Graders)")
extractor_model = (cfg.get("extractorModel") or "").strip() or \
(models[0]["alias"] if models else "")
if extractor_model not in catalog:
raise RuntimeError(f"extractor model '{extractor_model}' is not in the model catalog")
needed = {r["model"] for r in valid}
if cfg.get("synthesisEnabled"):
sm = synth_mod.pick_model(cfg)
if sm:
needed.add(sm)
needed = {r["model"] for r in valid} | {extractor_model}
adjudicate = bool(cfg.get("adjudicatorEnabled"))
adj_model = adj_mod.pick_model(cfg) if adjudicate else ""
needed_all = needed | ({adj_model} if adjudicate and adj_model else set())
# Air-gapped mode can't route to second-Spark models (internal net).
if cfg.get("networkMode") == "airgapped":
cat = {m["alias"]: m for m in (cfg.get("models") or [])}
offenders = [a for a in needed if cat.get(a, {}).get("spark") == "secondary"]
cat = {m["alias"]: m for m in models}
offenders = [a for a in needed_all if cat.get(a, {}).get("spark") == "secondary"]
if offenders:
raise RuntimeError(
"air-gapped mode requires all models on the head Spark, but these are "
f"on the secondary: {', '.join(sorted(offenders))}. Move them to the "
"primary Spark or switch to local-services mode.")
# 3. Ship text to the Spark + ensure infra.
self.phase = "reviewing"; self._persist()
push = sc.push_dir(sc.head(cfg), local_docs, f"{remote_job}/docs")
if push.returncode != 0:
raise RuntimeError(f"shipping documents to the Spark failed: {push.stderr}")
rev_mod.ensure_reviewer_image(cfg, self.log)
# 3. Infra once per job.
gr_mod.ensure_grader_image(cfg, self.log)
serving.ensure_network(cfg, self.log)
# In local_services mode, warn early if the optional web_search backend
# (SearXNG, self-signed HTTPS) is unreachable — non-fatal.
preflight.check_searxng(cfg, self.log)
# 4. Run the panel in waves (synthesis is handled separately, below).
review_aliases = {r["model"] for r in valid}
waves = serving.plan_waves(cfg, review_aliases)
self.waves_total = len(waves)
hf = bm_config.hf_token()
collected = []
for i, wave in enumerate(waves, 1):
self.wave_index = i
wave_aliases = {m["alias"] for m in wave}
wpanel = [r for r in valid if r["model"] in wave_aliases]
self.log(f"[runner] wave {i}/{len(waves)}: models={sorted(wave_aliases)} "
f"reviewers={[r['name'] for r in wpanel]}")
serving.bring_up_wave(cfg, wave, hf, self.log)
self._await_serving(cfg, wave)
preflight.check_wave(cfg, wave, self.log)
res = rev_mod.run_wave_reviewers(cfg, remote_job, wpanel, rubric, self.log)
collected.extend(res)
self._mark_panel(res)
serving.tear_down_wave(cfg, wave, self.log)
# 5. Synthesis (its own single-model wave).
synth_ok = False
if cfg.get("synthesisEnabled"):
self.phase = "synthesizing"; self._persist()
sm = synth_mod.pick_model(cfg)
swave = serving.plan_waves(cfg, {sm})
for wave in swave:
serving.bring_up_wave(cfg, wave, hf, self.log)
self._await_serving(cfg, wave)
preflight.check_wave(cfg, wave, self.log)
sres = synth_mod.run_synthesis(cfg, remote_job, rubric, self.log)
synth_ok = bool(sres.get("report"))
serving.tear_down_wave(cfg, wave, self.log)
# 4. Grade each deck unit, oldest first. One deck's failure never
# kills the job.
self.decks_total = len(units)
succeeded = 0
for idx, unit in enumerate(units, 1):
self.deck_index = idx
self.company = unit["company_slug"]
self.period = unit.get("period")
entry = {"company": unit["company_slug"], "period": unit.get("period"),
"status": "running"}
self.decks.append(entry)
self.panel = [{"name": r["name"], "model": r["model"], "status": "pending"}
for r in valid]
self._persist()
try:
result = self._grade_deck(cfg, led, job_id, remote_root, unit, idx,
valid, extractor_model, adjudicate, hf)
entry.update({"status": "done", "period": result["period"],
"composite": result["composite"]})
succeeded += 1
comp = result["composite"]
self.log(f"[runner] deck done: {unit['company_slug']} {result['period']}"
f" composite={comp if comp is not None else '?'}")
except Exception as e:
entry.update({"status": "failed", "error": str(e)[:300]})
self.log(f"[runner] DECK FAILED ({unit['company_slug']} "
f"{unit.get('period') or '?'}): {e}")
self.log(traceback.format_exc().splitlines()[-1])
self._persist()
# 6. Collect reports + assemble.
# 5. Job-level report.
self.phase = "collecting"; self._persist()
self._collect(cfg, job_id, remote_job, local_job, valid, manifest, synth_ok)
self._write_job_report(job_id)
# 7. Confidentiality: wipe the documents from the Spark.
# 6. Confidentiality: wipe the deck text from the Spark + teardown.
if cfg.get("wipeRemoteDocs", True):
sc.run(sc.head(cfg), f"rm -rf {remote_job}", timeout=60)
self.log("[runner] wiped document text from the Spark")
sc.run(sc.head(cfg), f"rm -rf {remote_root}", timeout=120)
self.log("[runner] wiped deck text from the Spark")
serving.tear_down_all(cfg, self.log)
# 8. Clear the inbox (move originals aside so they aren't re-reviewed).
self._drain_inbox(job_id)
# 7. Move the graded originals aside (the company folders stay).
self._drain_inbox(job_id, units)
self._last_done_sig = _inbox_signature()[1]
if succeeded:
self.phase = "done"
self.message = f"Reviewed {len(ok_docs)} document(s) with {len(valid)} reviewer(s)."
self.log(f"=== Review job {job_id} complete ===")
self.message = (f"Graded {succeeded}/{len(units)} deck(s) with "
f"{len(valid)} grader(s).")
else:
self.phase = "error"
self.message = f"All {len(units)} deck(s) failed — see the activity log."
self.log(f"=== Grading job {job_id} complete ({succeeded}/{len(units)} decks) ===")
self._persist()
except Exception as e:
self.phase = "error"
self.message = f"Review failed: {e}"
self.message = f"Grading failed: {e}"
self.log(f"[runner] JOB FAILED — {e}")
self.log(traceback.format_exc().splitlines()[-1])
try:
@@ -302,6 +370,205 @@ class JobRunner:
pass
self._persist()
# ------------------------------------------------------------- one deck
def _grade_deck(self, cfg: dict, led, job_id: str, remote_root: str, unit: dict,
idx: int, panel: list[dict], extractor_model: str,
adjudicate: bool, hf) -> dict:
"""Grade one deck unit end to end. Raises on deck failure (caller continues)."""
slug_c = unit["company_slug"]
# The staging/remote dir token; the final deck_id is the resolved period.
token = _token(unit["period"]) if unit.get("period") else f"deck-{idx:02d}"
local_deck = os.path.join(JOBS_DIR, job_id, slug_c, token)
remote_deck = f"{remote_root}/{slug_c}/{token}"
head = sc.head(cfg)
# --- extract locally + stage + push -----------------------------------
self.phase = "extracting"; self._persist()
manifest = extraction.extract_files(unit["files"], os.path.join(local_deck, "docs"),
self.log)
ok_docs = [m for m in manifest if m["ok"]]
for m in manifest:
if not m["ok"]:
self.log(f"[runner] WARNING: {m['source']}: {m['error']}")
if not ok_docs:
raise RuntimeError("no document text could be extracted from this deck")
gr_mod.stage_deck_files(cfg, local_deck, panel)
push = sc.push_dir(head, local_deck, remote_deck)
if push.returncode != 0:
raise RuntimeError(f"shipping deck text to the Spark failed: {push.stderr}")
# --- serve in waves; extractor + graders run in their model's wave ----
self.phase = "grading"; self._persist()
waves = serving.plan_waves(cfg, {r["model"] for r in panel} | {extractor_model})
self.waves_total = len(waves)
for i, wave in enumerate(waves, 1):
self.wave_index = i
aliases = {m["alias"] for m in wave}
wpanel = [r for r in panel if r["model"] in aliases]
self.log(f"[runner] wave {i}/{len(waves)}: models={sorted(aliases)} "
f"graders={[r['name'] for r in wpanel]}"
f"{' +extractor' if extractor_model in aliases else ''}")
serving.bring_up_wave(cfg, wave, hf, self.log)
try:
self._await_serving(cfg, wave)
preflight.check_wave(cfg, wave, self.log)
if extractor_model in aliases:
er = gr_mod.run_extractor(cfg, remote_deck, extractor_model, self.log)
if not er.get("report"):
raise RuntimeError("extractor produced no extraction.json")
if wpanel:
res = gr_mod.run_wave_graders(cfg, remote_deck, wpanel, self.log)
self._mark_panel(res)
finally:
serving.tear_down_wave(cfg, wave, self.log)
# --- pull the panel outputs + validate --------------------------------
local_out = os.path.join(local_deck, "out")
pull = sc.pull_dir(head, f"{remote_deck}/out", local_out)
if pull.returncode != 0:
raise RuntimeError(f"pulling panel outputs from the Spark failed: {pull.stderr}")
ext_path = os.path.join(local_out, "extraction.json")
ext_obj, ext_err = validate.validate_file(ext_path, "extraction")
if ext_obj is None or ext_err:
raise RuntimeError(f"extraction.json invalid: {ext_err or 'missing'}")
period, deck_id = self._resolve_period(unit, ext_obj, ext_path, token)
self.period = period; self._persist()
grades, panel_meta = [], []
for r in panel:
gpath = os.path.join(local_out, f"{r['rid']}.json")
gobj, gerr = (None, "no output file")
if os.path.exists(gpath) and not os.path.exists(gpath + ".invalid"):
gobj, gerr = validate.validate_file(gpath, "grades")
ok = gobj is not None and not gerr
if ok:
grades.append(gobj)
else:
self.log(f"[runner] grader {r['rid']} report dropped: {gerr}")
panel_meta.append({"rid": r["rid"], "model": r["model"], "valid": ok})
if len(grades) < 2:
raise RuntimeError(f"only {len(grades)} valid grade report(s) (need >= 2)")
# --- adjudication (non-fatal) ------------------------------------------
adjudication_md = None
if adjudicate:
self.phase = "adjudicating"; self._persist()
try:
adj_model = adj_mod.pick_model(cfg)
for wave in serving.plan_waves(cfg, {adj_model}):
serving.bring_up_wave(cfg, wave, hf, self.log)
try:
self._await_serving(cfg, wave)
preflight.check_wave(cfg, wave, self.log)
adj_mod.run_adjudication(cfg, remote_deck, self.log)
finally:
serving.tear_down_wave(cfg, wave, self.log)
local_adj = os.path.join(local_deck, "adjudicator-out")
sc.pull_dir(head, f"{remote_deck}/adjudicator-out", local_adj)
apath = os.path.join(local_adj, "ADJUDICATION.md")
if os.path.exists(apath):
adjudication_md = open(apath, errors="replace").read().strip() or None
if not adjudication_md:
self.log("[runner] WARNING: no adjudication produced (continuing without)")
except Exception as e:
self.log(f"[runner] WARNING: adjudication failed (non-fatal): {e}")
# --- deterministic scoring + ledger ------------------------------------
self.phase = "scoring"; self._persist()
company = led.ensure_company(slug_c)
pinned = company.get("pinned_targets") or []
aliases_map = company.get("kpi_aliases") or {}
prior = led.prior_targets(slug_c, period)
report_deck_dir = os.path.join(REPORTS_DIR, job_id, slug_c, deck_id)
meta = {
"company": slug_c,
"period": period,
"deck_id": deck_id,
"job_id": job_id,
"graded_at": datetime.now(timezone.utc).isoformat(),
"panel": panel_meta,
"artifacts": {
"report_dir": report_deck_dir,
"extraction": os.path.join(report_deck_dir, "extraction.json"),
"grades": [os.path.join(report_deck_dir, f"{p['rid']}.json")
for p in panel_meta if p["valid"]],
"adjudication": (os.path.join(report_deck_dir, "ADJUDICATION.md")
if adjudication_md else None),
},
}
record = scoring.score_deck(ext_obj, grades, pinned, prior, aliases_map,
cfg["weights"], meta)
rec_path = led.record_deck(slug_c, record, ext_obj.get("forward_targets") or [])
self.log(f"[runner] ledger updated: {rec_path}")
# --- reports ------------------------------------------------------------
self.phase = "collecting"; self._persist()
deck_md = scorecard.render_deck_report(record, ext_obj, adjudication_md)
if rec_path and str(rec_path).endswith(".json"):
ledger_md = os.path.splitext(str(rec_path))[0] + ".md"
else:
ledger_md = os.path.join(LEDGER_DIR, slug_c, "decks", f"{deck_id}.md")
os.makedirs(os.path.dirname(ledger_md), exist_ok=True)
with open(ledger_md, "w") as f:
f.write(deck_md)
os.makedirs(report_deck_dir, exist_ok=True)
with open(os.path.join(report_deck_dir, "DECK_REPORT.md"), "w") as f:
f.write(deck_md)
for fn in sorted(os.listdir(local_out)): # extraction + raw grader jsons (+ .invalid)
src = os.path.join(local_out, fn)
if os.path.isfile(src):
shutil.copyfile(src, os.path.join(report_deck_dir, fn))
if adjudication_md:
with open(os.path.join(report_deck_dir, "ADJUDICATION.md"), "w") as f:
f.write(adjudication_md + "\n")
# Refresh the company scorecard + the /data/reports latest copy.
sc_md = scorecard.render_scorecard(led.get_company(slug_c), led.deck_records(slug_c))
sc_path = os.path.join(LEDGER_DIR, slug_c, "SCORECARD.md")
os.makedirs(os.path.dirname(sc_path), exist_ok=True)
with open(sc_path, "w") as f:
f.write(sc_md)
with open(os.path.join(REPORTS_DIR, "latest-scorecard.md"), "w") as f:
f.write(sc_md)
self.log(f"[runner] reports saved to {report_deck_dir}")
return {"period": period, "deck_id": deck_id, "composite": _composite(record),
"report_dir": report_deck_dir}
def _resolve_period(self, unit: dict, ext_obj: dict, ext_path: str,
token: str) -> tuple[str, str]:
"""(period, deck_id) for this deck. Filename-derived period wins; else the
extractor's deck.period if canonical-ish; else the file's mtime month
(flagged as period_inferred in the extraction's red-flag candidates)."""
if unit.get("period"):
return unit["period"], _token(unit["period"])
p = ((ext_obj.get("deck") or {}).get("period") or "").strip()
if p and _PERIOD_RE.match(p):
self.log(f"[runner] period '{p}' taken from the deck text")
return p, _token(p)
try:
mtime = os.path.getmtime(unit["files"][0])
except OSError:
mtime = time.time()
period = time.strftime("%Y-%m", time.localtime(mtime))
ext_obj.setdefault("red_flag_candidates", []).append({
"code": "period_inferred",
"description": ("Reporting period was not stated in the filename or the deck "
f"text; inferred from the file's modification time as {period}."),
"severity": 2,
"evidence": "",
})
try:
with open(ext_path, "w") as f:
json.dump(ext_obj, f, indent=2)
except Exception:
pass
self.log(f"[runner] WARNING: period inferred from file mtime: {period}")
return period, (_token(period) or token)
# ------------------------------------------------------------- helpers
def _await_serving(self, cfg, wave, timeout=900):
self.log("[runner] waiting for wave serving to come online…")
@@ -329,50 +596,55 @@ class JobRunner:
p["status"] = "no-report"
self._persist()
def _collect(self, cfg, job_id, remote_job, local_job, valid, manifest, synth_ok):
out_local = os.path.join(REPORTS_DIR, job_id)
os.makedirs(out_local, exist_ok=True)
sc.pull_dir(sc.head(cfg), f"{remote_job}/out", os.path.join(out_local, "reviewers"))
if synth_ok:
sc.pull_dir(sc.head(cfg), f"{remote_job}/synth-out", os.path.join(out_local, "synthesis"))
# Assemble a single latest.md: the consolidated report if present, else a
# concatenation of the individual reports.
parts = [f"# Boardroom Map review — {job_id}\n",
"Documents reviewed: " + ", ".join(m["source"] for m in manifest if m["ok"]) + "\n",
"Panel: " + ", ".join(f"{r['name']} ({r['model']})" for r in valid) + "\n"]
consolidated = os.path.join(out_local, "synthesis", "CONSOLIDATED_REPORT.md")
if synth_ok and os.path.exists(consolidated):
parts.append("\n---\n\n## Consolidated report (lead reviewer)\n\n")
parts.append(open(consolidated, errors="replace").read())
parts.append("\n\n---\n")
parts.append("\n## Individual reviewer reports\n")
rev_local = os.path.join(out_local, "reviewers")
if os.path.isdir(rev_local):
for fn in sorted(os.listdir(rev_local)):
if fn.endswith(".md"):
parts.append(f"\n### {fn[:-3]}\n\n")
parts.append(open(os.path.join(rev_local, fn), errors="replace").read())
parts.append("\n")
assembled = "".join(parts)
with open(os.path.join(out_local, "report.md"), "w") as f:
def _write_job_report(self, job_id: str):
"""latest.md — one job summary across every deck graded (or failed)."""
lines = [f"# Boardroom Map grading job — {job_id}\n"]
done = [d for d in self.decks if d.get("status") == "done"]
failed = [d for d in self.decks if d.get("status") == "failed"]
lines.append(f"Decks graded: {len(done)}/{len(self.decks)}\n")
lines.append("\n## Results\n")
for d in self.decks:
if d.get("status") == "done":
comp = d.get("composite")
comp_s = f"{comp:.1f}" if isinstance(comp, (int, float)) else "?"
lines.append(f"- **{d['company']}** — {d.get('period') or '?'}: "
f"composite **{comp_s}** / 100\n")
else:
lines.append(f"- **{d['company']}** — {d.get('period') or '?'}: "
f"FAILED — {d.get('error', 'unknown error')}\n")
if done:
lines.append("\nPer-deck reports (DECK_REPORT.md, extraction, raw grades, "
f"adjudication): `/data/reports/{job_id}/<company>/<deck>/`.\n")
lines.append("Company scorecards: `/data/ledger/<company>/SCORECARD.md` "
"(latest copy at `/data/reports/latest-scorecard.md`).\n")
if failed:
lines.append("\nFailed decks were still moved to "
f"`/data/processed/{job_id}/` — re-drop them to regrade.\n")
assembled = "".join(lines)
out_dir = os.path.join(REPORTS_DIR, job_id)
os.makedirs(out_dir, exist_ok=True)
with open(os.path.join(out_dir, "report.md"), "w") as f:
f.write(assembled)
with open(os.path.join(REPORTS_DIR, "latest.md"), "w") as f:
f.write(assembled)
self.last_report_path = os.path.join(out_local, "report.md")
self.log(f"[runner] reports saved to {out_local}")
self.last_report_path = os.path.join(out_dir, "report.md")
def _drain_inbox(self, job_id):
dest = os.path.join(PROCESSED, job_id)
def _drain_inbox(self, job_id: str, units: list[dict]):
"""Move each unit's ORIGINAL files to /data/processed/<job>/<slug>/. The
per-company inbox folders are kept — the user reuses them next quarter."""
moved = 0
for unit in units:
dest = os.path.join(PROCESSED, job_id, unit["company_slug"])
os.makedirs(dest, exist_ok=True)
for fn in os.listdir(INBOX):
src = os.path.join(INBOX, fn)
for src in unit["files"]:
if os.path.isfile(src):
try:
shutil.move(src, os.path.join(dest, fn))
shutil.move(src, os.path.join(dest, os.path.basename(src)))
moved += 1
except Exception:
pass
self.log(f"[runner] inbox cleared (originals moved to processed/{job_id})")
self.log(f"[runner] inbox cleared ({moved} file(s) moved to processed/{job_id}; "
"company folders kept)")
# Module-level singleton used by app.py
orchestrator/ledger.py
+180
View File
@@ -0,0 +1,180 @@
"""The per-company running score ledger under /data/ledger.
Layout:
/data/ledger/<slug>/company.json identity, aliases, pinned targets,
extracted forward targets, history
/data/ledger/<slug>/decks/<id>.json full scoring record per graded deck
Config (bm_config `companies`) is the source of truth for name/aliases/pinned
targets; extracted_targets and history are owned by the grading pipeline.
Re-graded decks supersede (rename, never delete) the previous record.
"""
from __future__ import annotations
import json
import os
from datetime import datetime, timezone
import decks as decks_mod
def atomic_write_json(path: str, obj) -> None:
"""Write JSON via temp file + os.replace so readers never see a torn file."""
tmp = f"{path}.tmp"
with open(tmp, "w", encoding="utf-8") as f:
json.dump(obj, f, indent=2, sort_keys=False)
f.write("\n")
os.replace(tmp, path)
def parse_alias_lines(text: str) -> dict:
"""Parse newline-separated "canonical=alias1;alias2" lines into a dict."""
out: dict[str, list[str]] = {}
for line in (text or "").splitlines():
line = line.strip()
if not line or "=" not in line:
continue
canonical, _, rest = line.partition("=")
canonical = canonical.strip().lower()
aliases = [a.strip() for a in rest.split(";") if a.strip()]
if canonical and aliases:
out[canonical] = aliases
return out
class Ledger:
def __init__(self, base_dir: str):
self.base = base_dir
os.makedirs(self.base, exist_ok=True)
# ------------------------------------------------------------- paths
def _company_dir(self, slug: str) -> str:
return os.path.join(self.base, slug)
def _company_path(self, slug: str) -> str:
return os.path.join(self._company_dir(slug), "company.json")
def _decks_dir(self, slug: str) -> str:
return os.path.join(self._company_dir(slug), "decks")
# ------------------------------------------------------------- companies
def ensure_company(self, slug: str, name: str | None = None) -> dict:
"""Load the company, creating a skeleton entry on first sight."""
existing = self.get_company(slug)
if existing is not None:
return existing
company = {
"schema_version": 1,
"slug": slug,
"name": name or slug,
"auto_created": name is None,
"kpi_aliases": {},
"pinned_targets": [],
"extracted_targets": {},
"history": [],
}
os.makedirs(self._company_dir(slug), exist_ok=True)
atomic_write_json(self._company_path(slug), company)
return company
def get_company(self, slug: str) -> dict | None:
try:
with open(self._company_path(slug), encoding="utf-8") as f:
return json.load(f)
except (FileNotFoundError, json.JSONDecodeError):
return None
def all_slugs(self) -> list[str]:
if not os.path.isdir(self.base):
return []
return sorted(d for d in os.listdir(self.base)
if os.path.isfile(self._company_path(d)))
def all_companies(self) -> list[dict]:
return [c for c in (self.get_company(s) for s in self.all_slugs()) if c]
def merge_config_companies(self, companies_cfg: list) -> None:
"""Config wins for name/aliases/pinned targets; ledger keeps the rest."""
for cc in companies_cfg or []:
name = (cc.get("name") or "").strip()
slug = (cc.get("slug") or "").strip() or decks_mod.slugify(name)
company = self.ensure_company(slug, name or slug)
company["name"] = name or company["name"]
company["auto_created"] = False
company["kpi_aliases"] = parse_alias_lines(cc.get("kpiAliases") or "")
company["pinned_targets"] = list(cc.get("pinnedTargets") or [])
atomic_write_json(self._company_path(slug), company)
# ------------------------------------------------------------- targets
def prior_targets(self, slug: str, period: str) -> list[dict]:
"""Forward targets an earlier deck set for `period` (this deck's exam)."""
company = self.get_company(slug)
if not company or not period:
return []
return (company.get("extracted_targets", {}).get(period) or {}).get("targets", [])
# ------------------------------------------------------------- decks
def record_deck(self, slug: str, record: dict, forward_targets: list[dict]) -> str:
"""Persist a scoring record + its forward targets. Returns the deck path.
A re-graded deck_id supersedes (renames) the old record; the history
entry for the same period is replaced; a target period's forward
targets are replaced wholesale when set by a newer (or same) deck."""
company = self.ensure_company(slug)
deck_id = record["deck_id"]
ddir = self._decks_dir(slug)
os.makedirs(ddir, exist_ok=True)
deck_path = os.path.join(ddir, f"{deck_id}.json")
if os.path.exists(deck_path):
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
os.replace(deck_path, os.path.join(ddir, f"{deck_id}.superseded-{stamp}.json"))
atomic_write_json(deck_path, record)
period = record.get("period")
entry = {"period": period, "composite": record.get("composite"),
"graded_at": record.get("graded_at"),
"deck_file": f"decks/{deck_id}.json"}
history = [h for h in company.get("history", []) if h.get("period") != period]
history.append(entry)
history.sort(key=lambda h: decks_mod.period_sort_key(h.get("period")))
company["history"] = history
extracted = company.setdefault("extracted_targets", {})
from_key = decks_mod.period_sort_key(period)
by_period: dict[str, list[dict]] = {}
for ft in forward_targets or []:
tp = ft.get("target_period")
if tp:
by_period.setdefault(tp, []).append(ft)
for tp, targets in by_period.items():
cur = extracted.get(tp)
if cur is None or from_key >= decks_mod.period_sort_key(cur.get("from_deck")):
extracted[tp] = {"from_deck": period, "targets": targets}
atomic_write_json(self._company_path(slug), company)
return deck_path
def deck_record(self, slug: str, deck_id: str) -> dict | None:
try:
with open(os.path.join(self._decks_dir(slug), f"{deck_id}.json"),
encoding="utf-8") as f:
return json.load(f)
except (FileNotFoundError, json.JSONDecodeError):
return None
def deck_records(self, slug: str) -> list[dict]:
"""All live (non-superseded) deck records, oldest period first."""
ddir = self._decks_dir(slug)
if not os.path.isdir(ddir):
return []
out: list[dict] = []
for fn in sorted(os.listdir(ddir)):
if not fn.endswith(".json") or ".superseded-" in fn:
continue
try:
with open(os.path.join(ddir, fn), encoding="utf-8") as f:
out.append(json.load(f))
except (json.JSONDecodeError, OSError):
continue
out.sort(key=lambda r: decks_mod.period_sort_key(r.get("period")))
return out
orchestrator/prompts.py
+80
View File
@@ -0,0 +1,80 @@
"""Persona texts for the extractor, graders, and adjudicator.
These are the PERSONA.md contents written into each per-job dir and mounted
into the one-shot sandbox containers. The mechanical role instructions (read
/docs, emit JSON matching the schema at /schema.json, write to /out) live in
the sandbox agent itself — these texts only shape judgment and voice.
"""
from __future__ import annotations
def extractor_persona() -> str:
"""Stage-1 structured extractor: a forensic analyst, never a calculator."""
return (
"You are a forensic financial analyst extracting structured data from a "
"portfolio-company board deck. You are exhaustive and literal.\n\n"
"Extract:\n"
"- EVERY quantitative KPI actual reported for the deck's period: revenue, "
"ARR, margins, burn, cash, churn, NRR, headcount, pipeline — anything with "
"a number attached to a metric.\n"
"- Every stated forward target or guidance, with the exact period it "
"applies to (target_period).\n"
"- Red-flag candidates, using ONLY the taxonomy codes from the BDEF rubric "
"(adjusted_metrics, metric_redefinition, kpi_dropped, hockey_stick_forecast, "
"channel_stuffing_risk, short_term_comp, related_party, governance_gap, "
"cash_runway_silence, no_profitability_visibility, overreach_adjacency, "
"activity_bias, complexity_smokescreen, suppressed_dissent).\n"
"- Deck metadata: company hint, reporting period as printed, meeting date, "
"title.\n\n"
"Rules:\n"
"- canonical_name is lower_snake_case, GENERIC, and stable across quarters: "
"arr, ebitda_margin, churn_rate — not q2_arr_2026 or acme_revenue.\n"
"- profitability=true ONLY for profit/margin/cash metrics (EBITDA, net "
"margin, FCF, burn, runway) — never growth or activity metrics.\n"
"- NEVER compute, derive, or infer a number that is not printed in the "
"deck. If a margin is not printed, do not divide two numbers to get it.\n"
"- Copy the source location for every item (e.g. 'slide 6, financial "
"summary').\n"
"- direction: gte when higher is better, lte when lower is better "
"(churn, burn, CAC).\n"
"- target_in_deck is only a target printed NEXT TO the actual for the SAME "
"period; guidance for future periods goes in forward_targets."
)
def default_grader_persona(name: str) -> str:
"""Neutral BDEF lens for graders the operator has not customized."""
return (
f"You are '{name}', an owner's representative on the board grading this "
"deck strictly against the BDEF rubric provided.\n\n"
"- Score each category A-H from 1 to 5. A score above or below 3 REQUIRES "
"verbatim evidence quotes from the deck, with locations.\n"
"- Judge what the deck actually shows. Absence of evidence on a category "
"is itself information: score 2-3 and note the absence — never guess in "
"management's favor.\n"
"- Quote exactly; do not paraphrase inside quotes.\n"
"- Raise red flags only with the rubric's taxonomy codes, each with the "
"evidence that triggered it.\n"
"- Be specific and terse in rationales; write for a board member with "
"five minutes.\n"
"- Do NOT compute totals or a composite score; numbers are computed "
"elsewhere."
)
def adjudicator_persona() -> str:
"""Panel chair: consolidates the graders' verdicts, adds no new scores."""
return (
"You are the panel chair. You did not grade the deck yourself — you read "
"the graders' completed evaluations and adjudicate.\n\n"
"Produce a short markdown memo covering:\n"
"1. Consensus: what the panel agrees on, in one tight paragraph.\n"
"2. Disagreements: where graders diverge, which grader's evidence is "
"stronger and why (judge the quotes, not the adjectives).\n"
"3. Red flags: confirm or dismiss each raised flag based on the cited "
"evidence; say which deserve board attention.\n"
"4. Exactly 3 questions the board should ask management next quarter — "
"high-leverage, inversion-minded, answerable with data.\n\n"
"Attribute points to the grader(s) who raised them. Do not invent "
"findings, do not re-grade, and do NOT produce scores or totals."
)
orchestrator/requirements.txt
+2
View File
@@ -5,3 +5,5 @@ python-multipart==0.0.20
pyyaml==6.0.2
pypdf==5.1.0
python-docx==1.1.2
python-pptx==1.0.2
jsonschema==4.23.0
orchestrator/schemas/extraction.schema.json
+83
View File
@@ -0,0 +1,83 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "boardroom_extraction",
"type": "object",
"additionalProperties": false,
"required": ["schema_version", "deck", "kpis", "forward_targets", "red_flag_candidates", "narrative"],
"properties": {
"schema_version": {"type": "integer"},
"deck": {
"type": "object",
"additionalProperties": false,
"required": ["period"],
"properties": {
"company_hint": {"type": ["string", "null"]},
"period": {"type": ["string", "null"], "description": "Reporting period as printed on the deck, e.g. 2026-Q2, 2026-H1, FY2026, 2026-05"},
"meeting_date": {"type": ["string", "null"]},
"title": {"type": ["string", "null"]},
"truncated": {"type": "boolean", "default": false}
}
},
"kpis": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["name", "canonical_name", "actual", "direction", "profitability", "source"],
"properties": {
"name": {"type": "string", "description": "KPI label exactly as printed in the deck"},
"canonical_name": {"type": "string", "pattern": "^[a-z0-9_]+$", "description": "lower_snake_case, generic, stable across quarters (arr, ebitda_margin, churn_rate...)"},
"actual": {"type": "number"},
"unit": {"type": "string", "default": ""},
"period": {"type": ["string", "null"]},
"direction": {"type": "string", "enum": ["gte", "lte"], "description": "gte = higher is better, lte = lower is better"},
"profitability": {"type": "boolean", "description": "true only for profit/margin/cash metrics (EBITDA, net margin, FCF, burn...)"},
"target_in_deck": {"type": ["number", "null"], "description": "Target/plan value printed NEXT TO the actual for the SAME period, if any"},
"source": {"type": "string", "description": "Where in the deck, e.g. 'slide 6, financial summary'"},
"notes": {"type": "string", "default": ""}
}
}
},
"forward_targets": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["name", "canonical_name", "target", "target_period", "direction", "profitability", "source"],
"properties": {
"name": {"type": "string"},
"canonical_name": {"type": "string", "pattern": "^[a-z0-9_]+$"},
"target": {"type": "number"},
"unit": {"type": "string", "default": ""},
"target_period": {"type": "string", "description": "Period this guidance applies to, e.g. 2026-Q3"},
"direction": {"type": "string", "enum": ["gte", "lte"]},
"profitability": {"type": "boolean"},
"source": {"type": "string"}
}
}
},
"red_flag_candidates": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["code", "description", "severity"],
"properties": {
"code": {"type": "string", "pattern": "^[a-z0-9_]+$"},
"description": {"type": "string"},
"severity": {"type": "integer", "minimum": 1, "maximum": 5},
"evidence": {"type": "string", "default": ""}
}
}
},
"narrative": {
"type": "object",
"additionalProperties": false,
"required": ["summary"],
"properties": {
"summary": {"type": "string"},
"asks": {"type": "array", "items": {"type": "string"}, "default": []}
}
}
}
}
orchestrator/schemas/grades.schema.json
+53
View File
@@ -0,0 +1,53 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "boardroom_grades",
"type": "object",
"additionalProperties": false,
"required": ["schema_version", "grader", "categories", "red_flags", "overall_comment"],
"properties": {
"schema_version": {"type": "integer"},
"grader": {"type": "string"},
"categories": {
"type": "array",
"minItems": 8,
"maxItems": 8,
"items": {
"type": "object",
"additionalProperties": false,
"required": ["id", "score", "evidence", "rationale"],
"properties": {
"id": {"type": "string", "enum": ["A", "B", "C", "D", "E", "F", "G", "H"]},
"score": {"type": "integer", "minimum": 1, "maximum": 5},
"evidence": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["quote", "location"],
"properties": {
"quote": {"type": "string", "description": "Verbatim text from the deck"},
"location": {"type": "string", "description": "e.g. 'slide 3'"}
}
}
},
"rationale": {"type": "string"}
}
}
},
"red_flags": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["code", "description", "severity"],
"properties": {
"code": {"type": "string", "pattern": "^[a-z0-9_]+$"},
"description": {"type": "string"},
"severity": {"type": "integer", "minimum": 1, "maximum": 5},
"evidence": {"type": "string", "default": ""}
}
}
},
"overall_comment": {"type": "string"}
}
}
orchestrator/scorecard.py
+265
View File
@@ -0,0 +1,265 @@
"""Markdown renderers: per-deck DECK_REPORT and per-company SCORECARD.
Pure string builders over the scoring record shape (scoring.score_deck) and
ledger deck records — no I/O here; jobs.py decides where the files land.
"""
from __future__ import annotations
_CATEGORIES = "ABCDEFGH"
_CATEGORY_TITLES = {
"A": "Incentive Alignment & Skin in the Game",
"B": "Inversion Discipline & Margin of Safety",
"C": "Circle of Competence & Rational Learning",
"D": "Capital Allocation Quality",
"E": "Moat Durability & Competitive Reality",
"F": "Psychological & Cultural Health",
"G": "Simplicity, Clarity & Decision Velocity",
"H": "Board Value-Add & Governance Quality",
}
def _fmt(x, digits: int = 1) -> str:
if x is None:
return "—"
if isinstance(x, bool):
return "yes" if x else "no"
if isinstance(x, float):
return f"{x:.{digits}f}"
return str(x)
def _num(x, unit: str = "") -> str:
if x is None:
return "—"
s = f"{x:g}" if isinstance(x, (int, float)) else str(x)
return f"{s}{unit}" if unit and unit in ("%",) else (f"{s} {unit}".strip() if unit else s)
def _bucket_row(name: str, b: dict) -> str:
if b.get("na"):
return f"| {name} | — | — | {b.get('kpi_count', 0)} | NA — weight redistributed |"
return (f"| {name} | {_fmt(b.get('weight'))} | {_fmt(b.get('score'))} "
f"| {b.get('kpi_count', 0)} | |")
def render_deck_report(record: dict, extraction: dict, adjudication_md: str | None = None) -> str:
"""One deck's full markdown report."""
lines: list[str] = []
company = record.get("company") or "?"
period = record.get("period") or "unknown period"
lines.append(f"# Deck report — {company} · {period}")
lines.append("")
lines.append(f"## Composite: **{_fmt(record.get('composite'))} / 100**")
lines.append("")
q = record.get("quant", {})
lines.append(f"Quant {_fmt(q.get('score'))} · Qual {_fmt(record.get('qual', {}).get('score'))}"
f" · Penalties −{_fmt(record.get('penalties', {}).get('total'))}"
f" · graded {record.get('graded_at') or '?'} (job {record.get('job_id') or '?'})")
lines.append("")
# --- quantitative buckets
lines.append("## Quantitative (max 60)")
lines.append("")
lines.append("| Bucket | Weight | Score | KPIs | Note |")
lines.append("|---|---|---|---|---|")
lines.append(_bucket_row("Profitability KPIs", q.get("profitability", {})))
lines.append(_bucket_row("Other KPIs", q.get("other", {})))
lines.append(_bucket_row("Forecast integrity", q.get("forecast_integrity", {})))
lines.append("")
kpi_results = record.get("kpi_results") or []
if kpi_results:
lines.append("### KPI results")
lines.append("")
lines.append("| KPI | Actual | Target | Source | Credit |")
lines.append("|---|---|---|---|---|")
for r in kpi_results:
name = r.get("name") or r.get("canonical_name") or "?"
if r.get("matched_via") == "fuzzy":
name += " (≈ matched via fuzzy)"
src = r.get("target_source") or "—"
credit = "—" if r.get("credit") is None else _fmt(r.get("credit"), 2)
lines.append(f"| {name} | {_num(r.get('actual'), r.get('unit') or '')} "
f"| {_num(r.get('target'), r.get('unit') or '')} | {src} | {credit} |")
lines.append("")
fresults = record.get("forecast_results") or []
if fresults:
lines.append("### Forecast integrity (prior targets vs this period's actuals)")
lines.append("")
lines.append("| KPI | Prior target | Actual | Accuracy |")
lines.append("|---|---|---|---|")
for r in fresults:
lines.append(f"| {r.get('canonical_name')} | {_num(r.get('target'))} "
f"| {_num(r.get('actual'))} | {_fmt(r.get('accuracy'), 2)} |")
lines.append("")
# --- qualitative
lines.append("## Qualitative (max 40)")
lines.append("")
lines.append("| Category | Median | Evidence quality | Adjusted | Points |")
lines.append("|---|---|---|---|---|")
cats = record.get("qual", {}).get("categories", {})
for cid in _CATEGORIES:
c = cats.get(cid, {})
lines.append(f"| {cid}. {_CATEGORY_TITLES[cid]} | {_fmt(c.get('median'))} "
f"| {_fmt(c.get('evidence_quality'), 2)} | {_fmt(c.get('adjusted'), 2)} "
f"| {_fmt(c.get('points'), 2)} |")
lines.append("")
for cid in _CATEGORIES:
c = cats.get(cid, {})
rats = c.get("rationales") or []
if not rats:
continue
best = max(rats, key=lambda r: sum(len(e.get("quote") or "") for e in r.get("evidence") or []))
lines.append(f"### {cid}. {_CATEGORY_TITLES[cid]}")
lines.append("")
lines.append(f"**{best.get('grader')}**: {best.get('rationale')}")
for ev in best.get("evidence") or []:
loc = f" — {ev.get('location')}" if ev.get("location") else ""
lines.append(f"> \"{ev.get('quote')}\"{loc}")
lines.append("")
# --- red flags
flags = record.get("penalties", {}).get("flags") or []
lines.append(f"## Red flags (penalty −{_fmt(record.get('penalties', {}).get('total'))})")
lines.append("")
if flags:
lines.append("| Code | Severity | Points | Sources | Description |")
lines.append("|---|---|---|---|---|")
for f in flags:
lines.append(f"| `{f.get('code')}` | {f.get('severity')} | {_fmt(f.get('points'))} "
f"| {', '.join(f.get('sources') or [])} | {f.get('description')} |")
else:
lines.append("None raised.")
lines.append("")
# --- narrative
narrative = record.get("narrative") or extraction.get("narrative") or {}
if narrative.get("summary"):
lines.append("## Narrative")
lines.append("")
lines.append(narrative["summary"])
lines.append("")
asks = narrative.get("asks") or []
if asks:
lines.append("### Asks")
lines.append("")
for a in asks:
lines.append(f"- {a}")
lines.append("")
if adjudication_md:
lines.append("## Panel adjudication")
lines.append("")
lines.append(adjudication_md.strip())
lines.append("")
return "\n".join(lines).rstrip() + "\n"
def _arrow(delta: float) -> str:
if delta > 0:
return "↑"
if delta < 0:
return "↓"
return "→"
def render_scorecard(company: dict, records: list[dict]) -> str:
"""Company SCORECARD.md across all live deck records (oldest first)."""
name = company.get("name") or company.get("slug") or "?"
lines: list[str] = [f"# Scorecard — {name}", ""]
if not records:
lines.append("No graded decks yet.")
return "\n".join(lines) + "\n"
latest = records[-1]
prev = records[-2] if len(records) > 1 else None
comp = latest.get("composite") or 0.0
if prev is not None:
delta = round(comp - (prev.get("composite") or 0.0), 1)
lines.append(f"## Latest composite: **{_fmt(comp)}** ({latest.get('period')}) "
f"{_arrow(delta)} {'+' if delta > 0 else ''}{_fmt(delta)} vs {prev.get('period')}")
else:
lines.append(f"## Latest composite: **{_fmt(comp)}** ({latest.get('period')}) — first graded deck")
lines.append("")
# --- composite history
lines.append("## Composite history")
lines.append("")
lines.append("| Period | Composite | Quant | Qual | Penalties |")
lines.append("|---|---|---|---|---|")
for r in records:
lines.append(f"| {r.get('period') or '?'} | {_fmt(r.get('composite'))} "
f"| {_fmt(r.get('quant', {}).get('score'))} "
f"| {_fmt(r.get('qual', {}).get('score'))} "
f"| −{_fmt(r.get('penalties', {}).get('total'))} |")
lines.append("")
# --- categories latest vs previous
lines.append("## BDEF categories (latest vs previous)")
lines.append("")
lines.append("| Category | Latest | Previous | Δ |")
lines.append("|---|---|---|---|")
lcats = latest.get("qual", {}).get("categories", {})
pcats = (prev or {}).get("qual", {}).get("categories", {})
for cid in _CATEGORIES:
lp = lcats.get(cid, {}).get("points")
pp = pcats.get(cid, {}).get("points")
if lp is not None and pp is not None:
d = round(lp - pp, 2)
dcol = f"{_arrow(d)} {'+' if d > 0 else ''}{_fmt(d, 2)}"
else:
dcol = "—"
lines.append(f"| {cid}. {_CATEGORY_TITLES[cid]} | {_fmt(lp, 2)} | {_fmt(pp, 2)} | {dcol} |")
lines.append("")
# --- KPI hit-rate across records
order: list[str] = []
per_kpi: dict[str, list] = {}
for r in records:
for k in r.get("kpi_results") or []:
cn = k.get("canonical_name") or k.get("name") or "?"
if cn not in per_kpi:
per_kpi[cn] = []
order.append(cn)
per_kpi[cn].append(k.get("credit"))
lines.append("## KPI hit-rate")
lines.append("")
lines.append("| KPI | Attempts | Hits | Streak | Last credit |")
lines.append("|---|---|---|---|---|")
for cn in order:
credits = [c for c in per_kpi[cn] if c is not None]
if not credits:
continue
hits = sum(1 for c in credits if c >= 1)
streak = 0
for c in reversed(credits):
if c >= 1:
streak += 1
else:
break
lines.append(f"| {cn} | {len(credits)} | {hits} | {streak} | {_fmt(credits[-1], 2)} |")
lines.append("")
# --- open flags on the latest deck
flags = latest.get("penalties", {}).get("flags") or []
lines.append(f"## Open flags ({latest.get('period')})")
lines.append("")
if flags:
lines.append("| Code | Severity | Points | Sources | Description |")
lines.append("|---|---|---|---|---|")
for f in flags:
lines.append(f"| `{f.get('code')}` | {f.get('severity')} | {_fmt(f.get('points'))} "
f"| {', '.join(f.get('sources') or [])} | {f.get('description')} |")
else:
lines.append("None.")
lines.append("")
# --- deck reports
lines.append("## Deck reports")
lines.append("")
for r in records:
lines.append(f"- {r.get('period') or '?'} — composite {_fmt(r.get('composite'))} — "
f"decks/{r.get('deck_id')}.json / DECK_REPORT.md")
return "\n".join(lines).rstrip() + "\n"
orchestrator/scoring.py
+327
View File
@@ -0,0 +1,327 @@
"""Deterministic BDEF scoring — pure functions, stdlib only, no I/O.
Everything numeric happens here, never in a model: the panel supplies 1-5
category scores with verbatim evidence, the extractor supplies KPI actuals and
targets, and this module turns them into the 0-100 composite:
composite = quant (60) + qualitative (40) - red-flag penalties (cap 15)
All knobs come from the `weights` dict (bm_config.WEIGHTS_DEFAULTS shape) so
the operator can retune without a rebuild.
"""
from __future__ import annotations
import difflib
import re
import statistics
FUZZY_THRESHOLD = 0.85
_CATEGORIES = "ABCDEFGH"
# ---------------------------------------------------------------- KPI matching
def _norm(name: str) -> str:
return re.sub(r"[^a-z0-9]+", " ", (name or "").lower()).strip()
def match_kpi(canonical: str, candidates: list[dict], aliases: dict) -> tuple[dict | None, str | None]:
"""Match a canonical KPI name against candidate dicts ({canonical_name, name}).
Exact canonical -> alias map (either direction) -> fuzzy ratio >= 0.85.
Returns (candidate, via) with via in exact|alias|fuzzy, or (None, None)."""
canon = (canonical or "").strip().lower()
if not canon:
return None, None
for c in candidates:
if (c.get("canonical_name") or "").strip().lower() == canon:
return c, "exact"
amap = {(k or "").strip().lower(): {(a or "").strip().lower() for a in (v or [])}
for k, v in (aliases or {}).items()}
ours = amap.get(canon, set())
for c in candidates:
cn = (c.get("canonical_name") or "").strip().lower()
nm = (c.get("name") or "").strip().lower()
if cn in ours or nm in ours or canon in amap.get(cn, set()):
return c, "alias"
best, best_r = None, 0.0
for c in candidates:
for other in (c.get("canonical_name") or "", c.get("name") or ""):
r = difflib.SequenceMatcher(None, _norm(canon), _norm(other)).ratio()
if r > best_r:
best, best_r = c, r
if best is not None and best_r >= FUZZY_THRESHOLD:
return best, "fuzzy"
return None, None
# ---------------------------------------------------------------- KPI credit
def _passes(actual: float, target: float, direction: str) -> bool:
return actual <= target if direction == "lte" else actual >= target
def _credit(actual: float, target: float, direction: str, floor: float) -> float:
"""Partial credit for a targeted KPI: 0 below floor, linear to 1 at target."""
if target == 0 or (actual < 0) != (target < 0) or (direction == "lte" and actual == 0):
return 1.0 if _passes(actual, target, direction) else 0.0
r = target / actual if direction == "lte" else actual / target
if actual < 0 and target < 0:
# Both negative (EBITDA margin target -2, actual -3): the plain ratio
# inverts the ordering, so flip it back.
r = 1.0 / r
if r >= 1:
return 1.0
if floor >= 1 or r < floor:
return 0.0
return max(0.0, min(1.0, (r - floor) / (1 - floor)))
def _resolve_target(kpi: dict, pinned_targets: list[dict], prior_targets: list[dict],
aliases: dict) -> tuple[dict | None, str | None, str | None]:
"""(target dict {target, direction}, target_source, matched_via) for one KPI.
Precedence: pinned config target > prior deck's extracted forward target
for this period > target printed in the deck itself."""
canon = kpi.get("canonical_name") or ""
pinned_cands = [{"canonical_name": p.get("kpi"), "name": p.get("kpi"), "_src": p}
for p in (pinned_targets or [])]
cand, via = match_kpi(canon, pinned_cands, aliases)
if cand is not None:
p = cand["_src"]
return {"target": p.get("target"), "direction": p.get("direction") or kpi.get("direction")}, "pinned", via
cand, via = match_kpi(canon, prior_targets or [], aliases)
if cand is not None:
return {"target": cand.get("target"),
"direction": cand.get("direction") or kpi.get("direction")}, "extracted", via
if kpi.get("target_in_deck") is not None:
return {"target": kpi["target_in_deck"], "direction": kpi.get("direction")}, "in_deck", None
return None, None, None
# ---------------------------------------------------------------- scoring
def _bucket(results: list[dict], weight: float) -> dict:
if not results:
return {"score": 0.0, "weight": 0.0, "na": True, "kpi_count": 0}
mean = sum(r["credit"] for r in results) / len(results)
return {"score": mean, "weight": weight, "na": False, "kpi_count": len(results)}
def _qual(grades: list[dict], weights: dict) -> tuple[float, dict]:
"""(qual score, per-category detail). Evidence regresses medians toward 3."""
full_credit = max(1, int(weights.get("evidenceFullCredit", 400)))
cat_max = float(weights.get("qualCategoryMax", 5))
out: dict[str, dict] = {}
total = 0.0
for cid in _CATEGORIES:
scores: list[int] = []
equalities: list[float] = []
rationales: list[dict] = []
for g in grades or []:
cat = next((c for c in (g.get("categories") or []) if c.get("id") == cid), None)
if cat is None:
continue
evidence = cat.get("evidence") or []
quote_chars = sum(min(len(ev.get("quote") or ""), 200) for ev in evidence)
scores.append(int(cat.get("score", 3)))
equalities.append(min(1.0, quote_chars / full_credit))
rationales.append({
"grader": g.get("grader") or "grader",
"rationale": cat.get("rationale") or "",
"evidence": [{"quote": ev.get("quote") or "", "location": ev.get("location") or ""}
for ev in evidence],
})
if scores:
median = float(statistics.median(scores))
e_mean = sum(equalities) / len(equalities)
else:
median, e_mean = 3.0, 0.0
adjusted = 3.0 + (median - 3.0) * e_mean
points = adjusted * cat_max / 5.0
total += points
out[cid] = {
"panel_scores": scores,
"median": round(median, 2),
"evidence_quality": round(e_mean, 4),
"adjusted": round(adjusted, 4),
"points": round(points, 4),
"rationales": rationales,
}
return total, out
def score_deck(extraction: dict, grades: list[dict], pinned_targets: list[dict],
prior_targets: list[dict], kpi_aliases: dict, weights: dict,
meta: dict) -> dict:
"""Score one graded deck into the canonical ledger record. Pure."""
kpis = extraction.get("kpis") or []
floor = float(weights.get("kpiCreditFloor", 0.5))
scoring_flags: list[dict] = []
# --- per-KPI target resolution + credit
kpi_results: list[dict] = []
for k in kpis:
tgt, source, via = _resolve_target(k, pinned_targets, prior_targets, kpi_aliases)
credit = None
target = None
if tgt is not None and tgt.get("target") is not None:
target = float(tgt["target"])
credit = _credit(float(k.get("actual", 0)), target,
tgt.get("direction") or k.get("direction") or "gte", floor)
kpi_results.append({
"canonical_name": k.get("canonical_name"), "name": k.get("name"),
"actual": k.get("actual"), "unit": k.get("unit") or "",
"direction": k.get("direction"), "profitability": bool(k.get("profitability")),
"target": target, "target_source": source, "matched_via": via,
"credit": None if credit is None else round(credit, 4),
})
prof_all = [r for r in kpi_results if r["profitability"]]
prof_hit = [r for r in prof_all if r["credit"] is not None]
other_hit = [r for r in kpi_results if not r["profitability"] and r["credit"] is not None]
# --- forecast integrity: this deck's actuals vs the prior deck's targets
forecast_results: list[dict] = []
dropped: list[str] = []
for pt in prior_targets or []:
cand, _via = match_kpi(pt.get("canonical_name") or "", kpis, kpi_aliases)
if cand is None:
dropped.append(pt.get("canonical_name") or pt.get("name") or "kpi")
continue
actual = float(cand.get("actual", 0))
target = float(pt.get("target", 0))
direction = pt.get("direction") or "gte"
if target == 0:
acc = 1.0 if _passes(actual, target, direction) else 0.0
else:
err = (actual - target) / abs(target)
if direction == "lte":
err = -err
e = abs(err) if err < 0 else abs(err) / 2.0 # overshoot penalized half
acc = 1.0 - min(1.0, e)
forecast_results.append({
"canonical_name": pt.get("canonical_name"), "target": target,
"actual": actual, "accuracy": round(acc, 4),
})
# Pinned targets that no reported actual matches count as dropped too.
for p in pinned_targets or []:
cand, _via = match_kpi(p.get("kpi") or "", kpis, kpi_aliases)
if cand is None:
dropped.append(p.get("kpi") or "kpi")
seen: set[str] = set()
dropped_unique = [d for d in dropped
if not (d.strip().lower() in seen or seen.add(d.strip().lower()))]
for name in dropped_unique[: int(weights.get("droppedKpiMax", 3))]:
scoring_flags.append({
"code": "kpi_dropped",
"description": f"previously targeted KPI '{name}' is not reported this period",
"severity": int(weights.get("droppedKpiPenalty", 2)),
})
# --- quant buckets + renormalization (NA weight redistributes pro-rata)
prof = _bucket(prof_hit, float(weights.get("profitabilityKpi", 30)))
other = _bucket(other_hit, float(weights.get("otherKpi", 20)))
fmean = (sum(f["accuracy"] for f in forecast_results) / len(forecast_results)
if forecast_results else 0.0)
if forecast_results:
forecast = {"score": fmean, "weight": float(weights.get("forecastIntegrity", 10)),
"na": False, "kpi_count": len(forecast_results)}
else:
forecast = {"score": 0.0, "weight": 0.0, "na": True, "kpi_count": 0}
if not prof_all:
prof["na"] = True
prof["weight"] = 0.0
scoring_flags.append({
"code": "no_profitability_visibility",
"description": "no profit/margin/cash KPI reported at all",
"severity": 3,
})
total_quant_w = (float(weights.get("profitabilityKpi", 30))
+ float(weights.get("otherKpi", 20))
+ float(weights.get("forecastIntegrity", 10)))
present = [b for b in (prof, other, forecast) if not b["na"]]
if present:
scale = total_quant_w / sum(b["weight"] for b in present)
for b in present:
b["weight"] = round(b["weight"] * scale, 4)
b["score"] = round(b["score"] * b["weight"], 4)
quant_score = sum(b["score"] for b in present)
all_quant_na = False
else:
quant_score = 0.0
all_quant_na = True
scoring_flags.append({
"code": "no_quantitative_kpis",
"description": "no quantitative bucket could be scored (no targeted KPIs, "
"no prior targets)",
"severity": 4,
})
# --- qualitative
qual_score, categories = _qual(grades, weights)
qual_max = 8.0 * float(weights.get("qualCategoryMax", 5))
# --- red flags: extractor + graders + scoring; dedup, damp single-source
damp = float(weights.get("singleSourceFlagFactor", 0.5))
cap = float(weights.get("redFlagCap", 15))
flag_map: dict[str, dict] = {}
def add_flag(f: dict, source: str, scoring_flag: bool = False):
code = (f.get("code") or "flag").strip().lower()
key = f"{code}:{f.get('description', '')}" if scoring_flag and code == "kpi_dropped" else code
sev = int(f.get("severity", 1))
cur = flag_map.get(key)
if cur is None:
flag_map[key] = {"code": code, "description": f.get("description") or "",
"severity": sev, "sources": {source}, "scoring": scoring_flag}
else:
if sev > cur["severity"]:
cur["severity"] = sev
cur["description"] = f.get("description") or cur["description"]
cur["sources"].add(source)
cur["scoring"] = cur["scoring"] or scoring_flag
for f in extraction.get("red_flag_candidates") or []:
add_flag(f, "extractor")
for g in grades or []:
for f in g.get("red_flags") or []:
add_flag(f, g.get("grader") or "grader")
for f in scoring_flags:
add_flag(f, "scoring", scoring_flag=True)
flags: list[dict] = []
for f in flag_map.values():
full = f["scoring"] or len(f["sources"]) >= 2
points = float(f["severity"]) if full else float(f["severity"]) * damp
flags.append({"code": f["code"], "description": f["description"],
"severity": f["severity"], "points": round(points, 4),
"sources": sorted(f["sources"])})
flags.sort(key=lambda f: (-f["points"], f["code"]))
penalty_total = round(min(cap, sum(f["points"] for f in flags)), 4)
# --- composite
if all_quant_na:
base = (qual_score / qual_max * 100.0) if qual_max else 0.0
else:
base = quant_score + qual_score
composite = round(max(0.0, min(100.0, base - penalty_total)), 1)
return {
"schema_version": 1,
"company": meta.get("company"),
"period": meta.get("period"),
"deck_id": meta.get("deck_id"),
"job_id": meta.get("job_id"),
"graded_at": meta.get("graded_at"),
"composite": composite,
"quant": {"score": round(quant_score, 4), "profitability": prof,
"other": other, "forecast_integrity": forecast},
"qual": {"score": round(qual_score, 4), "categories": categories},
"penalties": {"total": penalty_total, "flags": flags},
"kpi_results": kpi_results,
"forecast_results": forecast_results,
"panel": meta.get("panel") or [],
"artifacts": meta.get("artifacts", {}),
"narrative": extraction.get("narrative", {}),
}
orchestrator/templates/index.html
+399 -39
View File
@@ -6,7 +6,7 @@
<title>Boardroom Map</title>
<style>
:root { --bg:#10131a; --panel:#181d27; --edge:#2a313f; --ink:#e9edf5; --dim:#97a1b5;
--accent:#c9a24b; --ok:#5fd08a; --warn:#ffcf66; --bad:#ff7a7a; }
--accent:#c9a24b; --ok:#5fd08a; --yg:#b9d05f; --warn:#ffcf66; --bad:#ff7a7a; }
* { box-sizing:border-box; }
body { margin:0; font:14px/1.5 system-ui,-apple-system,Segoe UI,Roboto,sans-serif;
background:var(--bg); color:var(--ink); }
@@ -20,7 +20,7 @@
.card h2 { margin:0 0 10px; font-size:13px; text-transform:uppercase; letter-spacing:1px; color:var(--dim); }
.pill { display:inline-block; padding:2px 10px; border-radius:999px; font-size:12px; font-weight:600; }
.pill.idle,.pill.done{background:#26303f;color:var(--dim)}
.pill.reviewing,.pill.synthesizing,.pill.collecting,.pill.extracting{background:#3b3414;color:var(--warn)}
.pill.busy{background:#3b3414;color:var(--warn)}
.pill.error{background:#3b1414;color:var(--bad)}
.kv { display:flex; justify-content:space-between; padding:4px 0; border-bottom:1px dashed var(--edge); }
.kv:last-child{border:0} .kv .k{color:var(--dim)}
@@ -28,15 +28,57 @@
.dot.on{background:var(--ok)} .dot.off{background:#55608f}
pre { background:#0c0f16; border:1px solid var(--edge); border-radius:10px; padding:12px; margin:0;
max-height:340px; overflow:auto; font:12px/1.5 ui-monospace,Menlo,monospace; color:#cdd6ff; white-space:pre-wrap; }
.row{display:flex;gap:8px;flex-wrap:wrap;margin-top:10px}
pre.tall{max-height:520px}
.row{display:flex;gap:8px;flex-wrap:wrap;margin-top:10px;align-items:center}
button{background:#222a3a;color:var(--ink);border:1px solid var(--edge);border-radius:9px;
padding:8px 12px;font-size:13px;cursor:pointer} button:hover{background:#2c3650}
button.primary{background:#3a2f12;border-color:#6b551f;color:#ffdf9a}
.drop{border:1.5px dashed var(--edge);border-radius:12px;padding:18px;text-align:center;color:var(--dim);cursor:pointer}
button.mini{padding:2px 8px;font-size:11px;border-radius:7px}
select,input[type=text]{background:#0c0f16;color:var(--ink);border:1px solid var(--edge);
border-radius:9px;padding:7px 10px;font-size:13px}
.drop{border:1.5px dashed var(--edge);border-radius:12px;padding:18px;text-align:center;color:var(--dim);cursor:pointer;margin-top:10px}
.drop.hot{border-color:var(--accent);color:var(--ink)}
.badge{font-size:11px;color:var(--dim)}
.muted{color:var(--dim);font-size:12px}
a{color:#9ab8ff}
/* portfolio */
table.port{width:100%;border-collapse:collapse}
table.port th{font-size:11px;text-transform:uppercase;letter-spacing:1px;color:var(--dim);
text-align:left;padding:4px 8px;border-bottom:1px solid var(--edge)}
table.port td{padding:7px 8px;border-bottom:1px dashed var(--edge);vertical-align:middle}
table.port tr.co{cursor:pointer} table.port tr.co:hover td{background:#1e2532}
.score{display:inline-block;min-width:46px;text-align:center;padding:2px 8px;border-radius:8px;
font-weight:700;font-size:13px}
.score.green{background:#15351f;color:var(--ok)} .score.yg{background:#2c3315;color:var(--yg)}
.score.amber{background:#3b3414;color:var(--warn)} .score.red{background:#3b1414;color:var(--bad)}
.score.none{background:#20263200;color:var(--dim);font-weight:400}
.delta{font-size:12px;font-weight:600;margin-left:4px}
.delta.up{color:var(--ok)} .delta.dn{color:var(--bad)} .delta.flat{color:var(--dim)}
.tag{display:inline-block;padding:1px 7px;border-radius:999px;font-size:10px;font-weight:600;
letter-spacing:.5px;text-transform:uppercase}
.tag.unreg{background:#3b3414;color:var(--warn)}
.tag.profit{background:#15351f;color:var(--ok)}
.chip{display:inline-block;padding:2px 9px;border-radius:999px;font-size:11px;margin:2px 4px 2px 0;
background:#222a3a;border:1px solid var(--edge)}
.chip.done{background:#15351f;color:var(--ok);border-color:#1f4a2c}
.chip.error{background:#3b1414;color:var(--bad);border-color:#552}
.chip.busy{background:#3b3414;color:var(--warn);border-color:#554416}
.chip.warn{background:#3b1414;color:var(--bad);border-color:#552222}
.chip.period{background:#1c2430;color:#9ab8ff;border-color:#2a3a55}
/* detail */
.detail-grid{display:grid;grid-template-columns:1fr 1fr;gap:18px;margin-top:12px}
@media (max-width:900px){ .detail-grid{grid-template-columns:1fr} .wrap{grid-template-columns:1fr} }
h3.sub{margin:14px 0 6px;font-size:12px;text-transform:uppercase;letter-spacing:1px;color:var(--dim)}
table.mini{width:100%;border-collapse:collapse;font-size:12px}
table.mini th{font-size:10px;text-transform:uppercase;letter-spacing:1px;color:var(--dim);
text-align:left;padding:3px 6px;border-bottom:1px solid var(--edge)}
table.mini td{padding:4px 6px;border-bottom:1px dashed var(--edge)}
.flag{border-left:3px solid var(--bad);background:#221318;border-radius:0 8px 8px 0;
padding:6px 10px;margin:6px 0;font-size:12px}
.flag .code{font:11px ui-monospace,Menlo,monospace;color:var(--bad)}
.inbox-co{margin-top:8px}
.inbox-co .co-name{font-weight:600;font-size:12px;color:var(--accent);letter-spacing:.5px}
details.jsonv summary{cursor:pointer;color:var(--dim);font-size:12px;margin:8px 0 4px}
</style>
</head>
<body>
@@ -48,16 +90,56 @@
</header>
<div class="wrap">
<div class="card full">
<h2>Portfolio</h2>
<div id="portfolio"><div class="muted">Loading…</div></div>
</div>
<div class="card full" id="companyCard" style="display:none">
<h2 style="display:flex;align-items:center;gap:10px">Company —
<span id="coName" style="color:var(--ink);letter-spacing:0;text-transform:none;font-size:15px"></span>
<span id="coBadges"></span>
<button class="mini" style="margin-left:auto" onclick="closeCompany()">close</button>
</h2>
<div id="coTrend"></div>
<div class="detail-grid">
<div>
<h3 class="sub">BDEF categories (latest, tick = previous)</h3>
<div id="coCats"><div class="muted">—</div></div>
<h3 class="sub">Open red flags (latest deck)</h3>
<div id="coFlags"><div class="muted">—</div></div>
</div>
<div>
<h3 class="sub">KPI hit rate</h3>
<div id="coKpis"><div class="muted">—</div></div>
</div>
</div>
<h3 class="sub">Deck history</h3>
<div id="coDecks"><div class="muted">—</div></div>
<div class="row">
<button onclick="toggleScorecard()" id="scBtn">View SCORECARD.md</button>
</div>
<div id="coViewer" style="margin-top:10px"></div>
</div>
<div class="card">
<h2>Documents</h2>
<div id="drop" class="drop">Drop confidential documents here, or click to choose<br>
<span class="badge">PDF · DOCX · TXT · MD</span></div>
<h2>Drop decks</h2>
<div class="row" style="margin-top:0">
<label class="muted" for="coSelect">Company</label>
<select id="coSelect" onchange="onCoSelect()"></select>
<span id="newCoWrap" style="display:none">
<input id="newCoName" type="text" placeholder="new company name" oninput="slugPreview()"/>
<span class="badge" id="slugPrev"></span>
</span>
</div>
<div id="drop" class="drop">Drop board decks here, or click to choose<br>
<span class="badge">PDF · PPTX · DOCX · TXT · MD — period parsed from filename (2026-Q2, FY2026…)</span></div>
<input id="file" type="file" multiple style="display:none"/>
<div id="inbox" style="margin-top:10px"></div>
<div class="row">
<button class="primary" onclick="runReview()">Run Review</button>
<button class="primary" onclick="act('/api/run','POST')">Grade decks</button>
<button onclick="act('/api/inbox/clear','POST')">Clear Inbox</button>
<button onclick="refresh()">Refresh</button>
<button onclick="refresh(); loadCompanies()">Refresh</button>
</div>
</div>
@@ -67,15 +149,15 @@
</div>
<div class="card">
<h2>Panel</h2>
<div id="panel"><div class="muted">No reviewers configured.</div></div>
<h2>Grading panel</h2>
<div id="panel"><div class="muted">No graders configured.</div></div>
</div>
<div class="card">
<h2>Serving / Job</h2>
<div id="job"><div class="muted">—</div></div>
<div class="row">
<button onclick="act('/api/reviewer/build-image','POST')">Build reviewer image</button>
<button onclick="act('/api/grader/build-image','POST')">Build grader image</button>
<button onclick="act('/api/stop','POST')">Stop serving</button>
</div>
</div>
@@ -84,72 +166,350 @@
<h2>Activity log</h2>
<pre id="log">loading…</pre>
</div>
<div class="card full">
<h2>Latest report</h2>
<pre id="report">—</pre>
</div>
</div>
<script>
// ------------------------------------------------------------------ helpers
function esc(s){ return String(s==null?'':s).replace(/[&<>"']/g,
c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'})[c]); }
async function getJSON(u){ const r=await fetch(u); if(!r.ok) throw new Error(await r.text()); return r.json(); }
async function act(u,m){ try{ const r=await fetch(u,{method:m}); const t=await r.text();
alert(r.ok? 'OK: '+t.slice(0,400) : 'Error: '+t.slice(0,400)); refresh(); }catch(e){ alert(e); } }
function dot(on,label){ return `<div><span class="dot ${on?'on':'off'}"></span>${label}</div>`; }
async function runReview(){ act('/api/run','POST'); }
function bandCls(c){ return c>=75?'green':c>=60?'yg':c>=45?'amber':'red'; }
function scoreBadge(c){ if(c==null||isNaN(+c)) return '<span class="score none">—</span>';
return `<span class="score ${bandCls(+c)}">${(+c).toFixed(1)}</span>`; }
function deltaArrow(d){ if(d==null||isNaN(+d)) return '';
if(Math.abs(+d)<0.05) return '<span class="delta flat">→ 0.0</span>';
return +d>0? `<span class="delta up">▲ ${(+d).toFixed(1)}</span>`
: `<span class="delta dn">▼ ${Math.abs(+d).toFixed(1)}</span>`; }
// Inline SVG sparkline: min-max normalized polyline (~120x28).
function sparkline(hist,w,h){
w=w||120; h=h||28;
const vals=(hist||[]).map(p=>+p.composite).filter(v=>!isNaN(v));
if(!vals.length) return '<span class="muted">—</span>';
if(vals.length===1) return `<svg width="${w}" height="${h}" viewBox="0 0 ${w} ${h}">`+
`<circle cx="${w/2}" cy="${h/2}" r="2.5" fill="var(--accent)"/></svg>`;
const mn=Math.min(...vals), mx=Math.max(...vals), span=(mx-mn)||1, pad=3;
const pts=vals.map((v,i)=>
`${(pad+i*(w-2*pad)/(vals.length-1)).toFixed(1)},${(h-pad-(v-mn)*(h-2*pad)/span).toFixed(1)}`).join(' ');
return `<svg width="${w}" height="${h}" viewBox="0 0 ${w} ${h}">`+
`<polyline points="${pts}" fill="none" stroke="var(--accent)" stroke-width="1.5"/></svg>`;
}
// Large composite trend with period labels + value dots.
function bigTrend(hist){
const pts=(hist||[]).filter(p=>p.composite!=null && !isNaN(+p.composite));
if(!pts.length) return '<div class="muted">No graded decks yet.</div>';
const w=680,h=150,padL=14,padR=14,padT=18,padB=26;
const vals=pts.map(p=>+p.composite);
const mn=Math.min(...vals), mx=Math.max(...vals), span=(mx-mn)||1;
const x=i=> pts.length===1? w/2 : padL+i*(w-padL-padR)/(pts.length-1);
const y=v=> h-padB-(v-mn)*(h-padT-padB)/span;
let svg=`<svg width="100%" viewBox="0 0 ${w} ${h}" style="max-width:${w}px">`;
if(pts.length>1)
svg+=`<polyline points="${pts.map((p,i)=>`${x(i).toFixed(1)},${y(+p.composite).toFixed(1)}`).join(' ')}" `+
`fill="none" stroke="var(--accent)" stroke-width="2"/>`;
const step=Math.max(1,Math.ceil(pts.length/10));
pts.forEach((p,i)=>{
const px=x(i), py=y(+p.composite);
svg+=`<circle cx="${px.toFixed(1)}" cy="${py.toFixed(1)}" r="3.5" fill="var(--accent)"/>`;
svg+=`<text x="${px.toFixed(1)}" y="${(py-8).toFixed(1)}" text-anchor="middle" `+
`font-size="10" fill="var(--ink)">${(+p.composite).toFixed(1)}</text>`;
if(i%step===0 || i===pts.length-1)
svg+=`<text x="${px.toFixed(1)}" y="${h-8}" text-anchor="middle" font-size="10" `+
`fill="var(--dim)">${esc(p.period||'?')}</text>`;
});
return svg+'</svg>';
}
// Horizontal BDEF A–H bars (0–5), previous value as a tick marker.
const BDEF={A:'Incentive alignment',B:'Inversion discipline',C:'Circle of competence',
D:'Capital allocation',E:'Moat durability',F:'Psych / cultural health',
G:'Simplicity & velocity',H:'Board value-add'};
function bdefBars(cats){
const ids=Object.keys(BDEF).filter(id=>cats[id]);
if(!ids.length) return '<div class="muted">—</div>';
const rowH=26,w=460,lab=170,barW=w-lab-46;
let svg=`<svg width="100%" viewBox="0 0 ${w} ${ids.length*rowH}" style="max-width:${w}px">`;
ids.forEach((id,r)=>{
const cy=r*rowH+rowH/2, c=cats[id]||{};
const cur=c.latest_adjusted, prev=c.previous_adjusted;
svg+=`<text x="0" y="${cy+4}" font-size="11" fill="var(--dim)">${id} · ${esc(BDEF[id])}</text>`;
svg+=`<rect x="${lab}" y="${cy-6}" width="${barW}" height="12" rx="6" fill="#0c0f16" stroke="var(--edge)"/>`;
if(cur!=null && !isNaN(+cur)){
const bw=Math.max(2,Math.min(1,+cur/5)*barW);
svg+=`<rect x="${lab}" y="${cy-6}" width="${bw.toFixed(1)}" height="12" rx="6" fill="var(--accent)"/>`;
svg+=`<text x="${lab+barW+8}" y="${cy+4}" font-size="11" fill="var(--ink)">${(+cur).toFixed(1)}</text>`;
}
if(prev!=null && !isNaN(+prev)){
const tx=lab+Math.min(1,+prev/5)*barW;
svg+=`<line x1="${tx.toFixed(1)}" y1="${cy-9}" x2="${tx.toFixed(1)}" y2="${cy+9}" `+
`stroke="var(--ink)" stroke-width="1.5" opacity="0.7"/>`;
}
});
return svg+'</svg>';
}
// Client-side mirror of decks.py period parsing (for inbox display).
const PERIOD_RES=[
[/(?:^|\D)((?:19|20)\d\d)[-_ ]?[Qq]([1-4])(?!\d)/, m=>m[1]+'-Q'+m[2]],
[/(?:^|[^A-Za-z0-9])[Qq]([1-4])[-_ ]((?:19|20)\d\d)(?!\d)/, m=>m[2]+'-Q'+m[1]],
[/(?:^|\D)((?:19|20)\d\d)[-_ ]?[Hh]([12])(?!\d)/, m=>m[1]+'-H'+m[2]],
[/(?:^|[^A-Za-z0-9])[Ff][Yy][-_ ]?((?:19|20)\d\d)(?!\d)/, m=>'FY'+m[1]],
[/(?:^|\D)((?:19|20)\d\d)[-_](0[1-9]|1[0-2])(?!\d)/, m=>m[1]+'-'+m[2]],
];
function parsePeriod(name){
for(const [re,f] of PERIOD_RES){ const m=String(name||'').match(re); if(m) return f(m); }
return null;
}
function slugify(s){ return String(s||'').toLowerCase().replace(/[^a-z0-9]+/g,'-')
.replace(/^-+|-+$/g,'') || 'company'; }
// ------------------------------------------------------------------ portfolio
let companiesCache=[], currentSlug=null;
async function loadCompanies(){
try{
const d=await getJSON('/api/companies');
companiesCache=d.companies||[];
renderPortfolio(); renderCoSelect();
if(currentSlug) openCompany(currentSlug,true);
}catch(e){
document.getElementById('portfolio').innerHTML='<div class="muted">companies error: '+esc(e.message||e)+'</div>';
}
}
function renderPortfolio(){
const el=document.getElementById('portfolio');
if(!companiesCache.length){
el.innerHTML='<div class="muted">No companies yet — drop a deck below to create one.</div>'; return;
}
el.innerHTML='<table class="port"><thead><tr>'+
'<th>Company</th><th>Latest</th><th>Δ</th><th>Trend</th><th>Decks</th></tr></thead><tbody>'+
companiesCache.map(c=>{
const l=c.latest||{};
return `<tr class="co" onclick="openCompany('${esc(c.slug)}')">`+
`<td>${esc(c.name)} ${c.auto_created?'<span class="tag unreg">unregistered</span>':''}</td>`+
`<td>${scoreBadge(l.composite)} <span class="badge">${esc(l.period||'')}</span></td>`+
`<td>${deltaArrow(l.delta)}</td>`+
`<td>${sparkline(c.history)}</td>`+
`<td class="muted">${c.deck_count||0}</td></tr>`;
}).join('')+'</tbody></table>';
}
// ------------------------------------------------------------------ company detail
async function openCompany(slug,silent){
try{
const d=await getJSON('/api/companies/'+encodeURIComponent(slug));
currentSlug=slug;
renderDetail(d);
document.getElementById('companyCard').style.display='';
if(!silent) document.getElementById('companyCard').scrollIntoView({behavior:'smooth'});
}catch(e){ if(!silent) alert('load company failed: '+(e.message||e)); }
}
function closeCompany(){ currentSlug=null; document.getElementById('companyCard').style.display='none'; }
function renderDetail(d){
const co=d.company||{}, recs=d.records||[];
document.getElementById('coName').textContent=co.name||co.slug||'?';
document.getElementById('coBadges').innerHTML=co.auto_created?'<span class="tag unreg">unregistered</span>':'';
document.getElementById('coTrend').innerHTML=
bigTrend(recs.map(r=>({period:r.period,composite:r.composite})));
document.getElementById('coCats').innerHTML=bdefBars(d.categories_latest||{});
// KPI hit-rate table — profitability KPIs pinned to top.
const kpis=Object.entries(d.kpi_hit_rate||{})
.map(([cn,s])=>Object.assign({canonical:cn},s))
.sort((a,b)=>(b.profitability?1:0)-(a.profitability?1:0)||a.canonical.localeCompare(b.canonical));
document.getElementById('coKpis').innerHTML=!kpis.length? '<div class="muted">—</div>' :
'<table class="mini"><thead><tr><th>KPI</th><th></th><th>Last credit</th><th>Hits</th><th>Streak</th></tr></thead><tbody>'+
kpis.map(k=>`<tr><td title="${esc(k.canonical)}">${esc(k.name||k.canonical)}</td>`+
`<td>${k.profitability?'<span class="tag profit">profit</span>':''}</td>`+
`<td>${k.last_credit==null?'—':(k.last_credit*100).toFixed(0)+'%'}</td>`+
`<td class="muted">${k.hits||0}/${k.attempts||0}</td>`+
`<td class="muted">${k.streak||0}</td></tr>`).join('')+'</tbody></table>';
// Open red flags (latest deck).
const flags=d.open_flags||[];
document.getElementById('coFlags').innerHTML=!flags.length? '<div class="muted">none</div>' :
flags.map(f=>`<div class="flag"><span class="code">${esc(f.code)}</span>`+
` <span class="badge">severity ${esc(f.severity)}${f.points!=null?' · -'+esc(f.points)+' pts':''}</span>`+
`<br>${esc(f.description)}</div>`).join('');
// Deck history (newest first).
document.getElementById('coDecks').innerHTML=!recs.length? '<div class="muted">no graded decks yet</div>' :
'<table class="mini"><thead><tr><th>Period</th><th>Composite</th><th>Graded</th><th></th></tr></thead><tbody>'+
recs.slice().reverse().map(r=>
`<tr><td>${esc(r.period||'?')}</td><td>${scoreBadge(r.composite)}</td>`+
`<td class="muted">${esc(String(r.graded_at||'').slice(0,16).replace('T',' '))}</td>`+
`<td><button class="mini" onclick="viewDeckReport('${esc(r.deck_id)}')">report</button> `+
`<button class="mini" onclick="viewDeckJson('${esc(r.deck_id)}')">json</button></td></tr>`).join('')+
'</tbody></table>';
document.getElementById('coViewer').innerHTML='';
document.getElementById('scBtn').textContent='View SCORECARD.md';
}
async function viewDeckReport(deckId){
if(!currentSlug) return;
const base='/api/companies/'+encodeURIComponent(currentSlug)+'/decks/'+encodeURIComponent(deckId);
try{
const r=await fetch(base+'/report');
const t=await r.text();
setViewer('Deck report — '+deckId, `<pre class="tall">${esc(r.ok?t:('error: '+t))}</pre>`);
}catch(e){ alert(e); }
}
async function viewDeckJson(deckId){
if(!currentSlug) return;
try{
const d=await getJSON('/api/companies/'+encodeURIComponent(currentSlug)+
'/decks/'+encodeURIComponent(deckId));
setViewer('Deck record — '+deckId,
`<details class="jsonv" open><summary>collapse / expand raw JSON</summary>`+
`<pre class="tall">${esc(JSON.stringify(d,null,2))}</pre></details>`);
}catch(e){ alert('load record failed: '+(e.message||e)); }
}
let scorecardOpen=false;
async function toggleScorecard(){
if(!currentSlug) return;
const v=document.getElementById('coViewer'), btn=document.getElementById('scBtn');
if(scorecardOpen){ v.innerHTML=''; scorecardOpen=false; btn.textContent='View SCORECARD.md'; return; }
try{
const r=await fetch('/api/companies/'+encodeURIComponent(currentSlug)+'/scorecard');
const t=await r.text();
setViewer('SCORECARD.md', `<pre class="tall">${esc(r.ok?t:('error: '+t))}</pre>`);
scorecardOpen=true; btn.textContent='Hide SCORECARD.md';
}catch(e){ alert(e); }
}
function setViewer(title,html){
scorecardOpen=false;
document.getElementById('scBtn').textContent='View SCORECARD.md';
document.getElementById('coViewer').innerHTML=
`<h3 class="sub" style="display:flex;align-items:center">${esc(title)}`+
`<button class="mini" style="margin-left:auto" onclick="document.getElementById('coViewer').innerHTML=''">close</button></h3>`+html;
}
// ------------------------------------------------------------------ drop card
function renderCoSelect(){
const sel=document.getElementById('coSelect');
const cur=sel.value;
sel.innerHTML=companiesCache.map(c=>
`<option value="${esc(c.slug)}">${esc(c.name)}</option>`).join('')+
'<option value="__new__">new company…</option>';
if(cur && [...sel.options].some(o=>o.value===cur)) sel.value=cur;
else if(!companiesCache.length) sel.value='__new__';
onCoSelect();
}
function onCoSelect(){
const isNew=document.getElementById('coSelect').value==='__new__';
document.getElementById('newCoWrap').style.display=isNew?'':'none';
slugPreview();
}
function slugPreview(){
const n=document.getElementById('newCoName').value.trim();
document.getElementById('slugPrev').textContent=n?('→ /data/inbox/'+slugify(n)+'/'):'';
}
function uploadCompany(){
const sel=document.getElementById('coSelect');
if(sel.value!=='__new__') return sel.value;
const n=document.getElementById('newCoName').value.trim();
return n? slugify(n) : null;
}
const drop=document.getElementById('drop'), file=document.getElementById('file');
drop.onclick=()=>file.click();
file.onchange=()=>upload(file.files);
file.onchange=()=>{ upload(file.files); file.value=''; };
['dragover','dragenter'].forEach(e=>drop.addEventListener(e,ev=>{ev.preventDefault();drop.classList.add('hot');}));
['dragleave','drop'].forEach(e=>drop.addEventListener(e,ev=>{ev.preventDefault();drop.classList.remove('hot');}));
drop.addEventListener('drop',ev=>{ if(ev.dataTransfer.files.length) upload(ev.dataTransfer.files); });
async function upload(files){
const co=uploadCompany();
if(!co){ alert('Pick a company (or type a new company name) before uploading — root-level files are not graded.'); return; }
const names=[...files].map(f=>f.name+(parsePeriod(f.name)?' ['+parsePeriod(f.name)+']':' [no period in name]'));
const fd=new FormData(); for(const f of files) fd.append('files',f);
try{ const r=await fetch('/api/upload',{method:'POST',body:fd});
if(!r.ok) alert('Upload failed: '+(await r.text()).slice(0,300)); refresh(); }catch(e){ alert(e); }
try{
const r=await fetch('/api/upload?company='+encodeURIComponent(co),{method:'POST',body:fd});
if(!r.ok) alert('Upload failed: '+(await r.text()).slice(0,300));
else console.log('uploaded to '+co+': '+names.join(', '));
refresh(); loadCompanies();
}catch(e){ alert(e); }
}
function renderInbox(ib){
const el=document.getElementById('inbox');
const cos=(ib&&ib.companies)||{}, skipped=(ib&&ib.skipped)||[];
const slugs=Object.keys(cos).sort();
let html='';
for(const slug of slugs){
html+=`<div class="inbox-co"><span class="co-name">${esc(slug)}</span>`+
cos[slug].map(f=>{
const per=f.period||parsePeriod(f.name);
return `<div class="kv"><span>${esc(f.name)} `+
(per?`<span class="chip period">${esc(per)}</span>`:'<span class="badge">period?</span>')+
(f.supported===false?' <span class="badge">unsupported</span>':'')+
`</span><span class="muted">${f.bytes?((f.bytes/1024).toFixed(0)+' KB'):''}</span></div>`;
}).join('')+'</div>';
}
if(skipped.length)
html+=`<div style="margin-top:8px"><span class="chip warn">skipped (no company): ${esc(skipped.join(', '))}</span></div>`;
el.innerHTML=html||'<div class="muted">Inbox empty.</div>';
}
// ------------------------------------------------------------------ polling
let lastPhase=null;
async function refresh(){
try{
const s = await getJSON('/api/status');
const rt = s.runtime||{};
const ph = document.getElementById('phase');
ph.textContent = rt.phase||'idle'; ph.className = 'pill '+(rt.phase||'idle');
const phase = rt.phase||'idle';
ph.textContent = phase;
ph.className = 'pill '+(phase==='idle'||phase==='done'?phase==='done'?'done':'idle':phase==='error'?'error':'busy');
if(lastPhase && lastPhase!==phase && (phase==='done'||phase==='idle')) loadCompanies();
lastPhase=phase;
document.getElementById('netmode').textContent =
(s.networkMode==='airgapped'?'air-gapped':'local-services')
+ (s.synthesis?' · synthesis on':'') + (s.autoRunOnDrop?' · auto-run':'');
+ (s.adjudicator?' · adjudicator on':'') + (s.autoRunOnDrop?' · auto-run':'');
document.getElementById('inbox').innerHTML = (s.inbox||[]).length
? (s.inbox||[]).map(f=>`<div class="kv"><span>${f.name} ${f.supported?'':'<span class="badge">unsupported</span>'}</span><span class="muted">${(f.bytes/1024).toFixed(0)} KB</span></div>`).join('')
: '<div class="muted">Inbox empty.</div>';
renderInbox(s.inbox);
document.getElementById('ready').innerHTML =
dot(s.configured.sparks,'Sparks configured') +
dot(s.configured.models>0,`${s.configured.models} model(s)`) +
dot(s.configured.reviewers>0,`${s.configured.reviewers} reviewer(s)`) +
`<div class="kv"><span class="k">network mode</span><span>${s.networkMode}</span></div>` +
dot(s.configured.graders>0,`${s.configured.graders} grader(s)`) +
`<div class="kv"><span class="k">network mode</span><span>${esc(s.networkMode)}</span></div>` +
`<div class="kv"><span class="k">wipe docs after</span><span>${s.wipeRemoteDocs?'yes':'no'}</span></div>` +
(s.models||[]).map(m=>`<div class="kv"><span>${m.alias}</span><span class="muted">${m.hfModel} · ${m.spark}</span></div>`).join('');
(s.models||[]).map(m=>`<div class="kv"><span>${esc(m.alias)}</span><span class="muted">${esc(m.hfModel)} · ${esc(m.spark)}</span></div>`).join('');
document.getElementById('panel').innerHTML = (s.panel||[]).length
? (s.panel||[]).map(r=>`<div class="kv"><span>${r.name} ${r.known?'':'<span class="badge">unknown model</span>'}</span><span class="muted">${r.model}${r.persona?' · persona ✓':''}</span></div>`).join('')
: '<div class="muted">No reviewers configured.</div>';
? (s.panel||[]).map(g=>`<div class="kv"><span>${esc(g.name)} ${g.known?'':'<span class="badge">unknown model</span>'}</span><span class="muted">${esc(g.model)}${g.persona?' · persona ✓':''}</span></div>`).join('')
: '<div class="muted">No graders configured.</div>';
const panelRt=(rt.panel||[]);
const deckChips=(rt.decks||[]).map(d=>{
const st=d.status||'pending';
const cls=st==='done'?'done':st==='error'?'error':(st==='pending'?'':'busy');
const tail=d.composite!=null?(' · '+(+d.composite).toFixed(1)):(d.error?' · error':'');
return `<span class="chip ${cls}">${esc(d.company)}${d.period?' '+esc(d.period):''}${tail}</span>`;
}).join('');
const curCo = rt.company||rt.current_company, curPer = rt.period||rt.current_period;
document.getElementById('job').innerHTML =
`<div class="kv"><span class="k">job</span><span>${rt.job_id||'—'}</span></div>` +
(rt.waves_total?`<div class="kv"><span class="k">wave</span><span>${rt.wave_index}/${rt.waves_total}</span></div>`:'') +
(rt.message?`<div class="muted" style="margin-top:6px">${rt.message}</div>`:'') +
panelRt.map(p=>`<div class="kv"><span>${p.name}</span><span class="muted">${p.status||''}</span></div>`).join('');
`<div class="kv"><span class="k">job</span><span>${esc(rt.job_id||'—')}</span></div>` +
(rt.decks_total?`<div class="kv"><span class="k">deck</span><span>${esc(rt.deck_index)}/${esc(rt.decks_total)}${curCo?' · '+esc(curCo)+(curPer?' '+esc(curPer):''):''}</span></div>`:'') +
(rt.waves_total?`<div class="kv"><span class="k">wave</span><span>${esc(rt.wave_index)}/${esc(rt.waves_total)}</span></div>`:'') +
(rt.message?`<div class="muted" style="margin-top:6px">${esc(rt.message)}</div>`:'') +
(deckChips?`<div style="margin-top:8px">${deckChips}</div>`:'') +
(rt.panel||[]).map(p=>`<div class="kv"><span>${esc(p.name)}</span><span class="muted">${esc(p.status||'')}</span></div>`).join('');
const ev = await getJSON('/api/events');
document.getElementById('log').textContent = (ev.events||[]).slice(-200).join('\n') || '(no activity yet)';
try{ const rep = await (await fetch('/api/report')).text();
document.getElementById('report').textContent = rep; }catch(e){}
}catch(e){ document.getElementById('log').textContent = 'status error: '+e; }
}
refresh(); setInterval(refresh, 6000);
refresh(); loadCompanies();
setInterval(refresh, 6000);
setInterval(loadCompanies, 60000);
</script>
</body>
</html>
orchestrator/tests/fixtures/extraction_q1.json
Vendored
+29
View File
@@ -0,0 +1,29 @@
{
"schema_version": 1,
"deck": {
"company_hint": "Acme Robotics",
"period": "2026-Q1",
"meeting_date": "2026-04-15",
"title": "Acme Robotics — Q1 2026 Board Deck",
"truncated": false
},
"kpis": [
{"name": "ARR", "canonical_name": "arr", "actual": 10.0, "unit": "$M", "period": "2026-Q1", "direction": "gte", "profitability": false, "target_in_deck": 9.5, "source": "slide 3, financial summary", "notes": ""},
{"name": "EBITDA Margin", "canonical_name": "ebitda_margin", "actual": -5.0, "unit": "%", "period": "2026-Q1", "direction": "gte", "profitability": true, "target_in_deck": -6.0, "source": "slide 4, P&L bridge", "notes": ""},
{"name": "Logo Churn", "canonical_name": "churn_rate", "actual": 4.0, "unit": "%", "period": "2026-Q1", "direction": "lte", "profitability": false, "target_in_deck": 5.0, "source": "slide 5, retention", "notes": ""},
{"name": "Cash Balance", "canonical_name": "cash_balance", "actual": 12.0, "unit": "$M", "period": "2026-Q1", "direction": "gte", "profitability": true, "target_in_deck": null, "source": "slide 4, balance sheet", "notes": ""}
],
"forward_targets": [
{"name": "ARR", "canonical_name": "arr", "target": 12.0, "unit": "$M", "target_period": "2026-Q2", "direction": "gte", "profitability": false, "source": "slide 9, guidance"},
{"name": "Logo Churn", "canonical_name": "churn_rate", "target": 4.0, "unit": "%", "target_period": "2026-Q2", "direction": "lte", "profitability": false, "source": "slide 9, guidance"},
{"name": "EBITDA Margin", "canonical_name": "ebitda_margin", "target": -2.0, "unit": "%", "target_period": "2026-Q2", "direction": "gte", "profitability": true, "source": "slide 9, guidance"},
{"name": "Qualified Pipeline", "canonical_name": "qualified_pipeline", "target": 30.0, "unit": "$M", "target_period": "2026-Q2", "direction": "gte", "profitability": false, "source": "slide 10, pipeline build"}
],
"red_flag_candidates": [
{"code": "hockey_stick_forecast", "description": "H2 revenue ramp shown with no downside case or stated falsifiers", "severity": 3, "evidence": "slide 9 guidance chart"}
],
"narrative": {
"summary": "Solid Q1: ARR beat plan at $10.0M, EBITDA margin improved to -5%, churn under plan. The H2 story rests entirely on the $30M qualified pipeline building as projected.",
"asks": ["Approve $2M expansion of the Austin integration facility"]
}
}
orchestrator/tests/fixtures/extraction_q2.json
Vendored
+26
View File
@@ -0,0 +1,26 @@
{
"schema_version": 1,
"deck": {
"company_hint": "Acme Robotics",
"period": "2026-Q2",
"meeting_date": "2026-07-14",
"title": "Acme Robotics — Q2 2026 Board Deck",
"truncated": false
},
"kpis": [
{"name": "ARR", "canonical_name": "arr", "actual": 11.0, "unit": "$M", "period": "2026-Q2", "direction": "gte", "profitability": false, "target_in_deck": null, "source": "slide 3, financial summary", "notes": ""},
{"name": "Logo Churn", "canonical_name": "churn_rate", "actual": 3.5, "unit": "%", "period": "2026-Q2", "direction": "lte", "profitability": false, "target_in_deck": null, "source": "slide 5, retention", "notes": ""},
{"name": "EBITDA Margin", "canonical_name": "ebitda_margin", "actual": -3.0, "unit": "%", "period": "2026-Q2", "direction": "gte", "profitability": true, "target_in_deck": null, "source": "slide 4, P&L bridge", "notes": ""},
{"name": "Cash Balance", "canonical_name": "cash_balance", "actual": 13.0, "unit": "$M", "period": "2026-Q2", "direction": "gte", "profitability": true, "target_in_deck": null, "source": "slide 4, balance sheet", "notes": ""}
],
"forward_targets": [
{"name": "ARR", "canonical_name": "arr", "target": 14.0, "unit": "$M", "target_period": "2026-Q3", "direction": "gte", "profitability": false, "source": "slide 9, guidance"}
],
"red_flag_candidates": [
{"code": "adjusted_metrics", "description": "EBITDA presented on an adjusted basis with no bridge to GAAP", "severity": 2, "evidence": "slide 4 footnote"}
],
"narrative": {
"summary": "Mixed Q2: ARR missed guidance at $11.0M vs $12.0M, churn beat, margin improved but missed the -2% target. Pipeline metric no longer reported.",
"asks": ["Approve revised FY2026 hiring plan"]
}
}
orchestrator/tests/fixtures/grade_a.json
Vendored
+103
View File
@@ -0,0 +1,103 @@
{
"schema_version": 1,
"grader": "grader-a",
"categories": [
{
"id": "A",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category A: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "B",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category B: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "C",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category C: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "D",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category D: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "E",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category E: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "F",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category F: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "G",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category G: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "H",
"score": 2,
"evidence": [
{
"quote": "We ask the board to approve the revised hiring plan as presented; supporting detail is available from management upon request after the meeting, and we recommend approval without further discussion given the compressed agenda for this session.",
"location": "slide 11"
}
],
"rationale": "Asks are listed without recommendations or the inversion of the decision."
}
],
"red_flags": [
{
"code": "governance_gap",
"description": "succession and incentive redesign get one bullet while product minutiae fill nine slides",
"severity": 2,
"evidence": "slides 12-20"
}
],
"overall_comment": "Strong disclosure discipline overall; governance asks remain the weak spot."
}
orchestrator/tests/fixtures/grade_b.json
Vendored
+98
View File
@@ -0,0 +1,98 @@
{
"schema_version": 1,
"grader": "grader-b",
"categories": [
{
"id": "A",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category A: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "B",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category B: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "C",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category C: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "D",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category D: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "E",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category E: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "F",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category F: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "G",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category G: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "H",
"score": 3,
"evidence": [],
"rationale": "Asks are listed without recommendations or the inversion of the decision."
}
],
"red_flags": [
{
"code": "governance_gap",
"description": "board asks lack recommendations and inversion",
"severity": 3,
"evidence": "slide 11"
}
],
"overall_comment": "Strong disclosure discipline overall; governance asks remain the weak spot."
}
orchestrator/tests/fixtures/grade_c.json
Vendored
+96
View File
@@ -0,0 +1,96 @@
{
"schema_version": 1,
"grader": "grader-c",
"categories": [
{
"id": "A",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category A: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "B",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category B: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "C",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category C: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "D",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category D: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "E",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category E: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "F",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category F: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "G",
"score": 4,
"evidence": [
{
"quote": "Our compensation plan ties 60 percent of executive bonus to three-year ARR retention and free-cash-flow milestones rather than annual bookings, and every vice president now holds equity vesting over four years with a one-year cliff, which we believe aligns the team with long-term owners.",
"location": "slide 7"
}
],
"rationale": "Category G: specific, quantified disclosure with owner-aligned framing."
},
{
"id": "H",
"score": 2,
"evidence": [
{
"quote": "We ask the board to approve the revised hiring plan as presented; supporting detail is available from management upon request after the meeting, and we recommend approval without further discussion given the compressed agenda for this session.",
"location": "slide 11"
}
],
"rationale": "Asks are listed without recommendations or the inversion of the decision."
}
],
"red_flags": [],
"overall_comment": "Strong disclosure discipline overall; governance asks remain the weak spot."
}
orchestrator/tests/test_decks.py
+118
View File
@@ -0,0 +1,118 @@
"""Tests for decks.py: slugify, period parsing/sorting, inbox discovery."""
import os
import sys
import tempfile
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import decks
class TestSlugify(unittest.TestCase):
def test_basic(self):
self.assertEqual(decks.slugify("Acme Robotics"), "acme-robotics")
self.assertEqual(decks.slugify(" Acme, Inc. (US) "), "acme-inc-us")
self.assertEqual(decks.slugify("ALLCAPS"), "allcaps")
self.assertEqual(decks.slugify(""), "company")
self.assertEqual(decks.slugify("---"), "company")
class TestParsePeriod(unittest.TestCase):
def test_quarters(self):
self.assertEqual(decks.parse_period_from_name("acme_2026-Q2_board.pdf"), "2026-Q2")
self.assertEqual(decks.parse_period_from_name("2026Q4 deck.pptx"), "2026-Q4")
self.assertEqual(decks.parse_period_from_name("Q3 2025 update.pptx"), "2025-Q3")
self.assertEqual(decks.parse_period_from_name("q1_2024_board.docx"), "2024-Q1")
def test_halves(self):
self.assertEqual(decks.parse_period_from_name("board-2026-H1.docx"), "2026-H1")
self.assertEqual(decks.parse_period_from_name("2025H2-review.pdf"), "2025-H2")
def test_months(self):
self.assertEqual(decks.parse_period_from_name("acme 2026-05 board.pdf"), "2026-05")
self.assertEqual(decks.parse_period_from_name("2026_12_flash.txt"), "2026-12")
self.assertIsNone(decks.parse_period_from_name("2026-13 notes.pdf"))
self.assertIsNone(decks.parse_period_from_name("2026-00 notes.pdf"))
def test_fiscal_year(self):
self.assertEqual(decks.parse_period_from_name("FY2025 review.pdf"), "FY2025")
self.assertEqual(decks.parse_period_from_name("fy-2024 plan.txt"), "FY2024")
self.assertEqual(decks.parse_period_from_name("FY 2026 budget.docx"), "FY2026")
def test_no_period(self):
self.assertIsNone(decks.parse_period_from_name("notes.txt"))
self.assertIsNone(decks.parse_period_from_name("budget_2027.xlsx"))
self.assertIsNone(decks.parse_period_from_name("Q5 2026.pdf"))
def test_quarter_wins_over_month(self):
# "2026-Q2" must not be misread; Q pattern is checked before YYYY-MM.
self.assertEqual(decks.parse_period_from_name("2026-Q2 and 2026-05.pdf"), "2026-Q2")
def test_not_inside_digit_runs(self):
self.assertIsNone(decks.parse_period_from_name("doc-20261-05.pdf"))
class TestPeriodSortKey(unittest.TestCase):
def test_ordering_mixed_granularities(self):
ordered = ["FY2025", "2025-Q4", "2026-H1", "2026-Q1", "2026-01",
"2026-Q2", "2026-05", "2026-H2", "2026-Q4"]
self.assertEqual(sorted(ordered, key=decks.period_sort_key), ordered)
def test_start_months(self):
self.assertEqual(decks.period_sort_key("2026-Q2")[:2], (2026, 4))
self.assertEqual(decks.period_sort_key("2026-H2")[:2], (2026, 7))
self.assertEqual(decks.period_sort_key("2026-11")[:2], (2026, 11))
self.assertEqual(decks.period_sort_key("FY2026")[:2], (2026, 1))
def test_unknown_sorts_last(self):
keys = [decks.period_sort_key(p) for p in ("2026-Q4", None, "garbage", "FY2026")]
self.assertEqual(max(keys), decks.period_sort_key(None))
self.assertEqual(decks.period_sort_key(None), decks.period_sort_key("garbage"))
self.assertGreater(decks.period_sort_key(None), decks.period_sort_key("2099-Q4"))
class TestDiscover(unittest.TestCase):
def _touch(self, *parts):
path = os.path.join(*parts)
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w") as f:
f.write("x")
def test_discover(self):
with tempfile.TemporaryDirectory() as tmp:
inbox = os.path.join(tmp, "inbox")
self._touch(inbox, "Acme Robotics", "acme-2026-Q1.pdf")
self._touch(inbox, "Acme Robotics", "acme-2026-Q1-appendix.txt")
self._touch(inbox, "Acme Robotics", "acme-2026-Q2.pptx")
self._touch(inbox, "Acme Robotics", "chart-2026-Q2.png")
self._touch(inbox, "Acme Robotics", "notes.txt")
self._touch(inbox, "Acme Robotics", ".DS_Store")
self._touch(inbox, "beta-corp", "deck 2026-H1.docx")
self._touch(inbox, "stray.pdf")
out = decks.discover(inbox)
self.assertEqual(out["skipped"], ["stray.pdf"])
units = out["units"]
keys = [(u["company_slug"], u["period"], u["period_source"]) for u in units]
self.assertEqual(keys, [
("acme-robotics", "2026-Q1", "filename"),
("acme-robotics", "2026-Q2", "filename"),
("acme-robotics", None, "unknown"),
("beta-corp", "2026-H1", "filename"),
])
q1 = units[0]
self.assertEqual([os.path.basename(f) for f in q1["files"]],
["acme-2026-Q1-appendix.txt", "acme-2026-Q1.pdf"])
self.assertTrue(all(os.path.isabs(f) for f in q1["files"]))
q2 = units[1]
self.assertEqual([os.path.basename(f) for f in q2["files"]], ["acme-2026-Q2.pptx"])
self.assertEqual(q2["ignored"], ["chart-2026-Q2.png"])
self.assertEqual([os.path.basename(f) for f in units[2]["files"]], ["notes.txt"])
def test_missing_inbox(self):
self.assertEqual(decks.discover("/nonexistent/inbox"), {"units": [], "skipped": []})
if __name__ == "__main__":
unittest.main()
orchestrator/tests/test_ledger.py
+116
View File
@@ -0,0 +1,116 @@
"""Tests for ledger.py: company lifecycle, deck records, forward targets."""
import glob
import os
import sys
import tempfile
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import ledger as ledger_mod
def _record(deck_id, period, composite=70.0):
return {"schema_version": 1, "deck_id": deck_id, "period": period,
"composite": composite, "graded_at": "2026-07-01T00:00:00Z"}
def _ft(canonical, target, target_period, direction="gte"):
return {"name": canonical, "canonical_name": canonical, "target": target,
"unit": "", "target_period": target_period, "direction": direction,
"profitability": False, "source": "slide 9"}
class TestLedger(unittest.TestCase):
def setUp(self):
self._tmp = tempfile.TemporaryDirectory()
self.ledger = ledger_mod.Ledger(os.path.join(self._tmp.name, "ledger"))
def tearDown(self):
self._tmp.cleanup()
def test_ensure_company_auto_created(self):
c = self.ledger.ensure_company("acme")
self.assertTrue(c["auto_created"])
self.assertEqual(c["name"], "acme")
c2 = self.ledger.ensure_company("acme", name="Acme Robotics")
self.assertEqual(c2["name"], "acme") # existing entry wins
named = self.ledger.ensure_company("beta", name="Beta Corp")
self.assertFalse(named["auto_created"])
self.assertEqual(named["name"], "Beta Corp")
self.assertEqual(self.ledger.all_slugs(), ["acme", "beta"])
self.assertEqual(len(self.ledger.all_companies()), 2)
def test_merge_config_companies(self):
self.ledger.ensure_company("acme")
self.ledger.merge_config_companies([{
"slug": "acme", "name": "Acme Robotics",
"kpiAliases": "arr=annual recurring revenue;run_rate_arr\nchurn_rate=logo_churn",
"pinnedTargets": [{"kpi": "cash_balance", "target": 12.0, "unit": "$M",
"direction": "gte", "profitability": True}],
}])
c = self.ledger.get_company("acme")
self.assertFalse(c["auto_created"])
self.assertEqual(c["name"], "Acme Robotics")
self.assertEqual(c["kpi_aliases"],
{"arr": ["annual recurring revenue", "run_rate_arr"],
"churn_rate": ["logo_churn"]})
self.assertEqual(c["pinned_targets"][0]["kpi"], "cash_balance")
# slug derived from name when absent
self.ledger.merge_config_companies([{"name": "Beta Corp", "kpiAliases": "",
"pinnedTargets": []}])
self.assertIsNotNone(self.ledger.get_company("beta-corp"))
def test_record_supersede_prior_targets_roundtrip(self):
path = self.ledger.record_deck("acme", _record("2026-Q1", "2026-Q1", 82.7),
[_ft("arr", 12.0, "2026-Q2"),
_ft("arr", 15.0, "2026-Q3")])
self.assertTrue(os.path.isfile(path))
self.assertEqual(
[t["target"] for t in self.ledger.prior_targets("acme", "2026-Q2")], [12.0])
self.assertEqual(self.ledger.prior_targets("acme", "2026-Q4"), [])
self.assertEqual(self.ledger.prior_targets("nobody", "2026-Q2"), [])
# Re-grade the same deck: old record superseded (renamed), one live record.
self.ledger.record_deck("acme", _record("2026-Q1", "2026-Q1", 80.0),
[_ft("arr", 12.5, "2026-Q2")])
ddir = os.path.dirname(path)
self.assertEqual(len(glob.glob(os.path.join(ddir, "*.superseded-*.json"))), 1)
live = self.ledger.deck_records("acme")
self.assertEqual(len(live), 1)
self.assertEqual(live[0]["composite"], 80.0)
self.assertEqual(
[t["target"] for t in self.ledger.prior_targets("acme", "2026-Q2")], [12.5])
# history keeps one entry per period
c = self.ledger.get_company("acme")
self.assertEqual([h["period"] for h in c["history"]], ["2026-Q1"])
self.assertEqual(c["history"][0]["composite"], 80.0)
def test_newer_deck_replaces_targets_older_does_not(self):
self.ledger.record_deck("acme", _record("2026-Q1", "2026-Q1"),
[_ft("arr", 15.0, "2026-Q3")])
self.ledger.record_deck("acme", _record("2026-Q2", "2026-Q2"),
[_ft("arr", 16.0, "2026-Q3"),
_ft("churn_rate", 3.0, "2026-Q3", "lte")])
targets = self.ledger.prior_targets("acme", "2026-Q3")
self.assertEqual(sorted(t["target"] for t in targets), [3.0, 16.0])
c = self.ledger.get_company("acme")
self.assertEqual(c["extracted_targets"]["2026-Q3"]["from_deck"], "2026-Q2")
# Re-recording the OLDER deck must not clobber the newer deck's targets.
self.ledger.record_deck("acme", _record("2026-Q1", "2026-Q1"),
[_ft("arr", 15.0, "2026-Q3")])
targets = self.ledger.prior_targets("acme", "2026-Q3")
self.assertEqual(sorted(t["target"] for t in targets), [3.0, 16.0])
# History is sorted oldest first.
c = self.ledger.get_company("acme")
self.assertEqual([h["period"] for h in c["history"]], ["2026-Q1", "2026-Q2"])
def test_deck_record_lookup(self):
self.ledger.record_deck("acme", _record("2026-Q1", "2026-Q1"), [])
self.assertEqual(self.ledger.deck_record("acme", "2026-Q1")["period"], "2026-Q1")
self.assertIsNone(self.ledger.deck_record("acme", "2026-Q9"))
self.assertEqual(self.ledger.deck_records("nobody"), [])
if __name__ == "__main__":
unittest.main()
orchestrator/tests/test_scoring.py
+423
View File
@@ -0,0 +1,423 @@
"""Tests for scoring.py (pure scorer), validate.py, and the fixture-driven
Q1 -> Q2 end-to-end flow through the ledger and scorecard renderers."""
import json
import os
import sys
import tempfile
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import ledger as ledger_mod
import scorecard
import scoring
import validate
FIXTURES = os.path.join(os.path.dirname(os.path.abspath(__file__)), "fixtures")
WEIGHTS = {
"profitabilityKpi": 30, "otherKpi": 20, "forecastIntegrity": 10,
"qualCategoryMax": 5, "redFlagCap": 15, "kpiCreditFloor": 0.5,
"droppedKpiPenalty": 2, "droppedKpiMax": 3, "evidenceFullCredit": 400,
"singleSourceFlagFactor": 0.5,
}
PINNED_CASH = [{"kpi": "cash_balance", "target": 12.0, "unit": "$M",
"direction": "gte", "profitability": True}]
def _fixture(name):
with open(os.path.join(FIXTURES, name), encoding="utf-8") as f:
return json.load(f)
def _kpi(canonical, actual, direction="gte", prof=False, tid=None, name=None, unit=""):
return {"name": name or canonical, "canonical_name": canonical, "actual": actual,
"unit": unit, "period": None, "direction": direction, "profitability": prof,
"target_in_deck": tid, "source": "slide 1", "notes": ""}
def _ft(canonical, target, direction="gte", target_period="2026-Q2", prof=False):
return {"name": canonical, "canonical_name": canonical, "target": target,
"unit": "", "target_period": target_period, "direction": direction,
"profitability": prof, "source": "slide 9"}
def _extraction(kpis=None, forward=None, flags=None, period="2026-Q2"):
return {"schema_version": 1, "deck": {"period": period},
"kpis": kpis or [], "forward_targets": forward or [],
"red_flag_candidates": flags or [],
"narrative": {"summary": "test deck", "asks": []}}
def _grade(grader="grader-a", score=3, quote_chars=0, red_flags=None, overrides=None):
cats = []
for cid in "ABCDEFGH":
s, qc = score, quote_chars
if overrides and cid in overrides:
s, qc = overrides[cid]
ev = [{"quote": "q" * qc, "location": "slide 1"}] if qc else []
cats.append({"id": cid, "score": s, "evidence": ev, "rationale": f"cat {cid}"})
return {"schema_version": 1, "grader": grader, "categories": cats,
"red_flags": red_flags or [], "overall_comment": "ok"}
def _meta(period="2026-Q2", deck_id="d1"):
return {"company": "acme", "period": period, "deck_id": deck_id, "job_id": "job-1",
"graded_at": "2026-07-06T12:00:00Z",
"panel": [{"rid": "grader-a", "model": "grader-a", "valid": True}],
"artifacts": {"extraction": "extraction.json"}}
def _score(extraction, grades=None, pinned=None, prior=None, aliases=None, meta=None):
return scoring.score_deck(extraction, grades if grades is not None else [_grade()],
pinned or [], prior or [], aliases or {}, WEIGHTS,
meta or _meta())
def _flag(rec, code):
return [f for f in rec["penalties"]["flags"] if f["code"] == code]
class TestMatchKpi(unittest.TestCase):
def test_exact(self):
cand, via = scoring.match_kpi("arr", [{"canonical_name": "arr", "name": "ARR"}], {})
self.assertEqual(via, "exact")
self.assertEqual(cand["name"], "ARR")
def test_alias_forward_and_reverse(self):
aliases = {"arr": ["Annual Recurring Revenue", "run_rate_arr"]}
cand, via = scoring.match_kpi(
"arr", [{"canonical_name": "revenue_annualized",
"name": "Annual Recurring Revenue"}], aliases)
self.assertEqual(via, "alias")
cand, via = scoring.match_kpi(
"run_rate_arr", [{"canonical_name": "arr", "name": "ARR"}], aliases)
self.assertEqual(via, "alias")
def test_fuzzy(self):
cand, via = scoring.match_kpi(
"ebitda_margin", [{"canonical_name": "ebitda_margins", "name": "x"}], {})
self.assertEqual(via, "fuzzy")
def test_no_match(self):
self.assertEqual(
scoring.match_kpi("arr", [{"canonical_name": "cash_balance", "name": "Cash"}], {}),
(None, None))
self.assertEqual(scoring.match_kpi("", [{"canonical_name": "arr"}], {}), (None, None))
class TestCredit(unittest.TestCase):
def test_lte_credit(self):
rec = _score(_extraction([_kpi("churn_rate", 6.0, "lte", tid=5.0)]))
self.assertAlmostEqual(rec["kpi_results"][0]["credit"], 0.6667, places=4)
rec = _score(_extraction([_kpi("churn_rate", 4.0, "lte", tid=5.0)]))
self.assertEqual(rec["kpi_results"][0]["credit"], 1.0)
def test_floor(self):
rec = _score(_extraction([_kpi("arr", 4.0, tid=10.0)])) # r=0.4 < floor
self.assertEqual(rec["kpi_results"][0]["credit"], 0.0)
rec = _score(_extraction([_kpi("arr", 7.5, tid=10.0)])) # r=0.75 -> 0.5
self.assertAlmostEqual(rec["kpi_results"][0]["credit"], 0.5, places=4)
def test_guards(self):
self.assertEqual(scoring._credit(5, 0, "gte", 0.5), 1.0) # zero target, passes
self.assertEqual(scoring._credit(-5, 0, "gte", 0.5), 0.0) # zero target, fails
self.assertEqual(scoring._credit(-1, 1, "gte", 0.5), 0.0) # sign mismatch, fails
self.assertEqual(scoring._credit(1, -1, "gte", 0.5), 1.0) # sign mismatch, passes
self.assertEqual(scoring._credit(0, 5, "lte", 0.5), 1.0) # lte zero actual
def test_negative_targets(self):
# EBITDA margin: target -2, actual -3 -> two thirds of the way -> 0.3333
self.assertAlmostEqual(scoring._credit(-3, -2, "gte", 0.5), 1 / 3, places=4)
self.assertEqual(scoring._credit(-1, -2, "gte", 0.5), 1.0)
class TestQuantBuckets(unittest.TestCase):
def test_first_deck_renormalization(self):
# No prior targets -> forecast NA -> its 10 points redistribute 36/24.
rec = _score(_extraction([_kpi("ebitda_margin", 5.0, prof=True, tid=5.0),
_kpi("arr", 10.0, tid=10.0)]))
q = rec["quant"]
self.assertTrue(q["forecast_integrity"]["na"])
self.assertAlmostEqual(q["profitability"]["weight"], 36.0)
self.assertAlmostEqual(q["profitability"]["score"], 36.0)
self.assertAlmostEqual(q["other"]["weight"], 24.0)
self.assertAlmostEqual(q["other"]["score"], 24.0)
self.assertAlmostEqual(q["score"], 60.0)
self.assertEqual(rec["penalties"]["flags"], [])
self.assertAlmostEqual(rec["composite"], 84.0) # 60 quant + 24 qual (all 3s)
def test_forecast_integrity_second_deck(self):
prior = [_ft("arr", 12.0), _ft("churn_rate", 4.0, "lte")]
rec = _score(_extraction([_kpi("arr", 11.0), _kpi("churn_rate", 3.5, "lte"),
_kpi("fcf", 1.0, prof=True, tid=1.0)]),
prior=prior)
fi = rec["quant"]["forecast_integrity"]
self.assertFalse(fi["na"])
self.assertEqual(fi["weight"], 10.0)
self.assertEqual(fi["kpi_count"], 2)
accs = {f["canonical_name"]: f["accuracy"] for f in rec["forecast_results"]}
self.assertAlmostEqual(accs["arr"], 0.9167, places=4) # 1/12 undershoot
self.assertAlmostEqual(accs["churn_rate"], 0.9375, places=4) # overshoot halved
self.assertAlmostEqual(fi["score"], (0.9167 + 0.9375) / 2 * 10, places=3)
def test_no_profitability_flag_and_redistribution(self):
rec = _score(_extraction([_kpi("arr", 10.0, tid=10.0)]))
q = rec["quant"]
self.assertTrue(q["profitability"]["na"])
self.assertTrue(q["forecast_integrity"]["na"])
self.assertAlmostEqual(q["other"]["weight"], 60.0)
self.assertAlmostEqual(q["score"], 60.0)
flags = _flag(rec, "no_profitability_visibility")
self.assertEqual(len(flags), 1)
self.assertEqual(flags[0]["points"], 3.0) # scoring flags never damped
self.assertEqual(flags[0]["sources"], ["scoring"])
def test_profitability_kpis_without_targets_na_no_flag(self):
rec = _score(_extraction([_kpi("ebitda_margin", -5.0, prof=True),
_kpi("arr", 10.0, tid=10.0)]))
self.assertTrue(rec["quant"]["profitability"]["na"])
self.assertEqual(_flag(rec, "no_profitability_visibility"), [])
def test_all_quant_na_scales_qual(self):
rec = _score(_extraction([]))
# qual 24 (all 3s) scaled to 60, minus no_profitability(3) + no_quantitative(4)
self.assertTrue(all(rec["quant"][b]["na"] for b in
("profitability", "other", "forecast_integrity")))
self.assertEqual(len(_flag(rec, "no_quantitative_kpis")), 1)
self.assertAlmostEqual(rec["composite"], 53.0)
class TestTargetPrecedence(unittest.TestCase):
def test_pinned_beats_extracted_beats_in_deck(self):
kpis = [_kpi("arr", 11.0, tid=9.0)]
pinned = [{"kpi": "arr", "target": 10.0, "unit": "$M",
"direction": "gte", "profitability": False}]
prior = [_ft("arr", 12.0)]
r = _score(_extraction(kpis), pinned=pinned, prior=prior)["kpi_results"][0]
self.assertEqual((r["target"], r["target_source"], r["matched_via"]),
(10.0, "pinned", "exact"))
self.assertEqual(r["credit"], 1.0)
r = _score(_extraction(kpis), prior=prior)["kpi_results"][0]
self.assertEqual((r["target"], r["target_source"]), (12.0, "extracted"))
self.assertAlmostEqual(r["credit"], 0.8333, places=4)
r = _score(_extraction(kpis))["kpi_results"][0]
self.assertEqual((r["target"], r["target_source"], r["matched_via"]),
(9.0, "in_deck", None))
def test_untargeted_kpi_reported_with_none(self):
r = _score(_extraction([_kpi("nps", 40.0)]))["kpi_results"][0]
self.assertIsNone(r["target"])
self.assertIsNone(r["credit"])
self.assertIsNone(r["target_source"])
class TestQualitative(unittest.TestCase):
def test_evidence_regression_both_directions(self):
# Median 5 with no quotes regresses to 3; so does median 1.
rec = _score(_extraction([]), grades=[_grade(score=5, quote_chars=0)])
self.assertEqual(rec["qual"]["categories"]["A"]["adjusted"], 3.0)
rec = _score(_extraction([]), grades=[_grade(score=1, quote_chars=0)])
self.assertEqual(rec["qual"]["categories"]["A"]["adjusted"], 3.0)
self.assertAlmostEqual(rec["qual"]["score"], 24.0)
def test_full_evidence_keeps_extreme_scores(self):
# Per-quote chars cap at 200, so full credit (400) needs two quotes.
g = _grade(score=5, quote_chars=200)
for cat in g["categories"]:
cat["evidence"].append({"quote": "q" * 200, "location": "slide 2"})
rec = _score(_extraction([]), grades=[g])
cat = rec["qual"]["categories"]["A"]
self.assertEqual(cat["evidence_quality"], 1.0)
self.assertEqual(cat["adjusted"], 5.0)
self.assertEqual(cat["points"], 5.0)
def test_quote_chars_capped_at_200_each(self):
# One 1000-char quote counts as 200 -> e = 0.5 -> adjusted 4.
rec = _score(_extraction([]), grades=[_grade(score=5, quote_chars=1000)])
self.assertEqual(rec["qual"]["categories"]["A"]["evidence_quality"], 0.5)
self.assertEqual(rec["qual"]["categories"]["A"]["adjusted"], 4.0)
def test_panel_median_and_rationales(self):
grades = [_grade("g1", score=4, quote_chars=400),
_grade("g2", score=4, quote_chars=400),
_grade("g3", score=2, quote_chars=400)]
rec = _score(_extraction([]), grades=grades)
cat = rec["qual"]["categories"]["B"]
self.assertEqual(cat["panel_scores"], [4, 4, 2])
self.assertEqual(cat["median"], 4.0)
self.assertEqual(len(cat["rationales"]), 3)
self.assertEqual(cat["rationales"][0]["grader"], "g1")
class TestPenalties(unittest.TestCase):
def test_single_source_damping(self):
rec = _score(_extraction([], flags=[{"code": "adjusted_metrics",
"description": "d", "severity": 4}]))
f = _flag(rec, "adjusted_metrics")[0]
self.assertEqual(f["points"], 2.0)
self.assertEqual(f["sources"], ["extractor"])
def test_two_sources_full_severity_max_wins(self):
grades = [_grade("g1", red_flags=[{"code": "governance_gap",
"description": "weak", "severity": 2}]),
_grade("g2", red_flags=[{"code": "governance_gap",
"description": "worse", "severity": 3}])]
rec = _score(_extraction([]), grades=grades)
f = _flag(rec, "governance_gap")[0]
self.assertEqual(f["severity"], 3)
self.assertEqual(f["points"], 3.0)
self.assertEqual(f["sources"], ["g1", "g2"])
def test_penalty_cap(self):
codes = ["related_party", "channel_stuffing_risk", "suppressed_dissent",
"metric_redefinition"]
flags = [{"code": c, "description": c, "severity": 5} for c in codes]
rec = _score(_extraction([_kpi("fcf", 1.0, prof=True, tid=1.0)], flags=flags),
grades=[_grade("g1", red_flags=flags)])
self.assertEqual(rec["penalties"]["total"], 15.0) # 4x5=20 capped
def test_dropped_kpi_flags_capped(self):
prior = [_ft(c, 1.0) for c in ("alpha_metric", "beta_metric", "gamma_metric",
"delta_metric", "epsilon_metric")]
rec = _score(_extraction([]), prior=prior)
dropped = _flag(rec, "kpi_dropped")
self.assertEqual(len(dropped), 3) # droppedKpiMax
for f in dropped:
self.assertEqual(f["points"], 2.0) # droppedKpiPenalty, never damped
class TestValidate(unittest.TestCase):
def test_parse_json_text(self):
self.assertEqual(validate.parse_json_text('{"a": 1}'), {"a": 1})
salvaged = validate.parse_json_text(
'Sure! Here is the JSON:\n```json\n{"a": {"b": "}"}}\n```\ntrailing prose')
self.assertEqual(salvaged, {"a": {"b": "}"}})
self.assertIsNone(validate.parse_json_text("no json here"))
self.assertIsNone(validate.parse_json_text("[1, 2, 3]"))
self.assertIsNone(validate.parse_json_text(""))
def test_schemas_load_and_fixtures_validate(self):
self.assertIn("properties", validate.load_schema("extraction"))
self.assertIn("properties", validate.load_schema("grades"))
for name, schema in (("extraction_q1.json", "extraction"),
("extraction_q2.json", "extraction"),
("grade_a.json", "grades"), ("grade_b.json", "grades"),
("grade_c.json", "grades")):
err = validate.validate_obj(_fixture(name), schema)
self.assertIsNone(err, f"{name}: {err}")
def test_validate_obj_rejects_bad(self):
self.assertIsNotNone(validate.validate_obj({"schema_version": 1}, "grades"))
def test_validate_file(self):
obj, err = validate.validate_file(os.path.join(FIXTURES, "grade_a.json"), "grades")
self.assertIsNone(err)
self.assertEqual(obj["grader"], "grader-a")
obj, err = validate.validate_file("/nonexistent.json", "grades")
self.assertIsNone(obj)
self.assertIsNotNone(err)
class TestEndToEnd(unittest.TestCase):
"""Fixture-driven Q1 -> Q2 flow: score, ledger round-trip, rendering."""
def setUp(self):
self._tmp = tempfile.TemporaryDirectory()
self.ledger = ledger_mod.Ledger(os.path.join(self._tmp.name, "ledger"))
self.grades = [_fixture("grade_a.json"), _fixture("grade_b.json"),
_fixture("grade_c.json")]
self.q1 = _fixture("extraction_q1.json")
self.q2 = _fixture("extraction_q2.json")
def tearDown(self):
self._tmp.cleanup()
def _score_q1(self):
return scoring.score_deck(self.q1, self.grades, PINNED_CASH, [], {}, WEIGHTS,
_meta("2026-Q1", "2026-Q1"))
def test_q1_first_deck(self):
rec = self._score_q1()
q = rec["quant"]
self.assertTrue(q["forecast_integrity"]["na"])
self.assertAlmostEqual(q["score"], 60.0) # every KPI at/above target
# qual: A-G 3.5 pts each (median 4, evidence 0.5), H 2.6667
self.assertAlmostEqual(rec["qual"]["score"], 27.1667, places=3)
self.assertAlmostEqual(rec["qual"]["categories"]["H"]["points"], 2.6667, places=3)
# hockey_stick (extractor only, sev 3 -> 1.5) + governance_gap (2 graders -> 3)
self.assertAlmostEqual(rec["penalties"]["total"], 4.5)
self.assertAlmostEqual(rec["composite"], 82.7)
cash = next(k for k in rec["kpi_results"] if k["canonical_name"] == "cash_balance")
self.assertEqual(cash["target_source"], "pinned")
def test_q2_against_q1_targets(self):
rec1 = self._score_q1()
self.ledger.record_deck("acme", rec1, self.q1["forward_targets"])
prior = self.ledger.prior_targets("acme", "2026-Q2")
self.assertEqual(len(prior), 4)
rec2 = scoring.score_deck(self.q2, self.grades, PINNED_CASH, prior, {}, WEIGHTS,
_meta("2026-Q2", "2026-Q2"))
by_name = {k["canonical_name"]: k for k in rec2["kpi_results"]}
self.assertAlmostEqual(by_name["arr"]["credit"], 0.8333, places=4)
self.assertEqual(by_name["churn_rate"]["credit"], 1.0)
self.assertAlmostEqual(by_name["ebitda_margin"]["credit"], 0.3333, places=4)
self.assertEqual(by_name["cash_balance"]["target_source"], "pinned")
self.assertEqual(by_name["cash_balance"]["credit"], 1.0)
q = rec2["quant"]
self.assertAlmostEqual(q["profitability"]["score"], 20.0, places=2)
self.assertAlmostEqual(q["other"]["score"], 18.333, places=2)
self.assertAlmostEqual(q["forecast_integrity"]["score"], 7.847, places=2)
self.assertEqual(len(rec2["forecast_results"]), 3)
# qualified_pipeline guided in Q1 but not reported in Q2 -> dropped flag
dropped = _flag(rec2, "kpi_dropped")
self.assertEqual(len(dropped), 1)
self.assertIn("qualified_pipeline", dropped[0]["description"])
# adjusted_metrics 1.0 + governance_gap 3.0 + kpi_dropped 2.0
self.assertAlmostEqual(rec2["penalties"]["total"], 6.0)
self.assertAlmostEqual(rec2["composite"], 67.3)
self.assertAlmostEqual(
rec2["composite"],
round(q["score"] + rec2["qual"]["score"] - rec2["penalties"]["total"], 1))
# ledger round-trip + rendering
self.ledger.record_deck("acme", rec2, self.q2["forward_targets"])
records = self.ledger.deck_records("acme")
self.assertEqual([r["period"] for r in records], ["2026-Q1", "2026-Q2"])
report = scorecard.render_deck_report(rec2, self.q2, adjudication_md="Chair memo.")
self.assertIn("67.3", report)
self.assertIn("pinned", report)
self.assertIn("## Panel adjudication", report)
self.assertIn("Chair memo.", report)
self.assertIn("kpi_dropped", report)
card = scorecard.render_scorecard(self.ledger.get_company("acme"), records)
self.assertIn("2026-Q1", card)
self.assertIn("2026-Q2", card)
self.assertIn("↓", card) # composite fell Q1 -> Q2
self.assertIn("KPI hit-rate", card)
self.assertIn("arr", card)
def test_meta_passthrough_and_record_shape(self):
rec = self._score_q1()
self.assertEqual(rec["company"], "acme")
self.assertEqual(rec["deck_id"], "2026-Q1")
self.assertEqual(rec["job_id"], "job-1")
self.assertEqual(rec["panel"][0]["rid"], "grader-a")
self.assertEqual(rec["artifacts"], {"extraction": "extraction.json"})
self.assertEqual(rec["schema_version"], 1)
self.assertIn("summary", rec["narrative"])
for key in ("composite", "quant", "qual", "penalties", "kpi_results",
"forecast_results"):
self.assertIn(key, rec)
# the record must be JSON-serializable as produced
json.dumps(rec)
if __name__ == "__main__":
unittest.main()
orchestrator/validate.py
+89
View File
@@ -0,0 +1,89 @@
"""JSON parsing + schema validation for extractor/grader outputs.
Local models occasionally wrap their JSON in prose or fences; parse_json_text
salvages the first brace-balanced top-level object before we give up. Schemas
live in orchestrator/schemas/ and are the single contract between the sandbox
agents and the deterministic scorer.
"""
from __future__ import annotations
import json
import os
SCHEMAS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "schemas")
_cache: dict[str, dict] = {}
def load_schema(name: str) -> dict:
"""Load "extraction" or "grades" schema (cached)."""
if name not in _cache:
path = os.path.join(SCHEMAS_DIR, f"{name}.schema.json")
with open(path, encoding="utf-8") as f:
_cache[name] = json.load(f)
return _cache[name]
def parse_json_text(text: str) -> dict | None:
"""Parse `text` as a JSON object; salvage the first balanced {...} block."""
if not text:
return None
try:
obj = json.loads(text)
return obj if isinstance(obj, dict) else None
except Exception:
pass
start = text.find("{")
while start != -1:
depth = 0
in_str = False
esc = False
for i in range(start, len(text)):
c = text[i]
if esc:
esc = False
elif in_str:
if c == "\\":
esc = True
elif c == '"':
in_str = False
elif c == '"':
in_str = True
elif c == "{":
depth += 1
elif c == "}":
depth -= 1
if depth == 0:
try:
obj = json.loads(text[start:i + 1])
if isinstance(obj, dict):
return obj
except Exception:
pass
break
start = text.find("{", start + 1)
return None
def validate_obj(obj, schema_name: str) -> str | None:
"""Validate against the named schema; error message or None if valid."""
import jsonschema
try:
jsonschema.validate(obj, load_schema(schema_name))
return None
except jsonschema.ValidationError as e:
path = ".".join(str(p) for p in e.absolute_path) or "(root)"
return f"{path}: {e.message}"[:500]
def validate_file(path: str, schema_name: str) -> tuple[dict | None, str | None]:
"""Read + parse (with salvage) + validate a JSON file -> (obj, error)."""
try:
with open(path, encoding="utf-8", errors="replace") as f:
text = f.read()
except Exception as e:
return None, f"read failed: {e}"
obj = parse_json_text(text)
if obj is None:
return None, "no parseable JSON object found"
return obj, validate_obj(obj, schema_name)
package.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "boardroom-map-startos",
"version": "0.1.0",
"description": "StartOS service: drop confidential documents in and have a panel of local LLMs on your DGX Sparks review them — fully air-gappable, no frontier oversight",
"description": "StartOS service: grade portfolio-company board decks with a panel of local LLMs on your DGX Sparks — BDEF v1.1 scoring, per-company running scorecards, fully air-gappable",
"scripts": {
"build": "rm -rf ./javascript && ncc build startos/index.ts -o ./javascript",
"check": "tsc --noEmit",
sandbox/build.sh
+3 -3
View File
@@ -1,17 +1,17 @@
#!/usr/bin/env bash
# Build the Boardroom Map reviewer image ON THE HEAD SPARK (aarch64/GB10).
# Build the Boardroom Map grader image ON THE HEAD SPARK (aarch64/GB10).
# The orchestrator does this automatically over SSH, but you can also run it by
# hand: copy this sandbox/ directory to the Spark and run:
#
# IMAGE=boardroom-grader:latest bash build.sh
#
# The tag must match the service's "Reviewer Image Tag" (Configure Sparks).
# The tag must match the service's "Grader Image Tag" (Configure Sparks).
set -euo pipefail
IMAGE="${IMAGE:-boardroom-grader:latest}"
DIR="$(cd "$(dirname "$0")" && pwd)"
echo ">> Building $IMAGE from $DIR"
docker build -t "$IMAGE" -f "$DIR/reviewer.Dockerfile" "$DIR"
docker build -t "$IMAGE" -f "$DIR/grader.Dockerfile" "$DIR"
echo ">> Done. Image: $IMAGE"
docker image inspect "$IMAGE" >/dev/null && echo ">> OK"
sandbox/grader.Dockerfile
+9 -7
View File
@@ -1,15 +1,17 @@
# Boardroom Map reviewer image — BUILT ON THE HEAD SPARK (aarch64), not packed into the
# Boardroom Map grader image — BUILT ON THE HEAD SPARK (aarch64), not packed into the
# s9pk (the orchestrator ships this build context and builds it on the Spark; see
# reviewers.ensure_reviewer_image).
# graders.ensure_grader_image).
#
# One-shot, read-only document reviewer (grader_agent.py) speaking the
# OpenAI-compatible API directly — a lean pure-Python image that builds fast.
# One-shot, read-only role agent (grader_agent.py; BM_ROLE = extractor | grader |
# adjudicator) speaking the OpenAI-compatible API directly — a lean pure-Python
# image that builds fast.
#
# At RUN time the orchestrator launches this HARDENED (non-root, --cap-drop ALL,
# --security-opt no-new-privileges, read-only rootfs, no docker socket, only
# /docs (ro), /persona (ro), /RUBRIC.md (ro) and /out (rw) mounted, cpu/mem/pid
# caps) and attached to the per-job network. In air-gapped mode that network is
# --internal, so the container can reach ONLY the on-Spark model proxy.
# /docs (ro), /BDEF.md (ro), /schema.json (ro), /persona (ro) and /out (rw)
# mounted — the adjudicator instead gets /grades (ro) + /extraction.json (ro) —
# cpu/mem/pid caps) and attached to the per-job network. In air-gapped mode that
# network is --internal, so the container can reach ONLY the on-Spark model proxy.
FROM python:3.11-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
sandbox/grader_agent.py
+371 -262
View File
@@ -1,43 +1,56 @@
"""Boardroom Map reviewer — a sandboxed, ONE-SHOT agent that reads confidential
documents through a LOCAL model and writes a single report, then exits.
"""Boardroom Map role agent — a sandboxed, ONE-SHOT, single-completion agent.
Unlike a swarm worker, this never loops forever and never writes to a shared
workspace. It mounts the documents read-only at /docs, runs one model (through the
on-Spark proxy) under its PERSONA + the shared RUBRIC, and writes exactly one
file to /out:
role=reviewer -> /out/<BM_REVIEWER_ID>.md
role=synthesizer -> /out/CONSOLIDATED_REPORT.md (also reads /reports)
One container = one role = one model call (plus a bounded reliability ladder /
repair round-trip). No tools, no loops, no shared writable workspace. The
orchestrator (orchestrator/graders.py + adjudicator.py) launches this hardened
(non-root, read-only rootfs, per-job network) with BM_ROLE set to:
It speaks the OpenAI-compatible /v1/chat/completions API directly (no Claude CLI,
no Anthropic translation — small local models handle this far better). The whole
document set is pre-loaded into the prompt up to a budget; for anything larger the
model can pull more with read_file. Native tool_calls are used when available,
with a JSON-action text fallback for models without a vLLM tool parser.
extractor reads /docs (ro) + the BDEF red-flag taxonomy, emits STRUCTURED
JSON per /schema.json (extraction schema) -> /out/extraction.json
grader reads /docs (ro) + the full /BDEF.md rubric, scores categories
A-H per /schema.json (grades schema) -> /out/<BM_GRADER_ID>.json
adjudicator reads /extraction.json (ro) + the panel's /grades/*.json (ro),
writes a MARKDOWN adjudication (no scores) -> /out/ADJUDICATION.md
In air-gapped mode the container is on an --internal Docker network: the only
thing reachable is the model proxy. web_search is offered ONLY when BM_SEARXNG_URL
is set (local-services mode).
Env (set by the orchestrator):
BM_ROLE, BM_GRADER_ID, BM_GRADER_NAME, BM_MODEL,
BM_LLM_BASE (http://boardroom-proxy:4000/v1), BM_LLM_KEY,
BM_TEMPERATURE (extractor forced to 0.0), BM_MAX_MODEL_LEN
Mounts: /docs (ro), /BDEF.md (ro), /schema.json (ro; role-appropriate),
/persona/PERSONA.md (ro, optional), /out (rw); adjudicator additionally
/grades (ro) and /extraction.json (ro).
JSON reliability ladder (extractor + grader):
1. response_format = {"type":"json_schema", ..., "strict": true}
2. on HTTP 4xx: retry with top-level {"guided_json": <schema>} (vLLM ext.)
3. on another 4xx: retry plain
Parse whole-reply JSON, else the first brace-balanced {...} block. If invalid,
ONE repair round-trip; if still bad, write the raw text to the output path plus
a sibling <output>.invalid marker containing the error. Full jsonschema
validation runs orchestrator-side; only lightweight structural checks here.
Air-gapped: the per-job Docker network is --internal, so the only reachable
endpoint is the model proxy. Pure Python stdlib (urllib) — no pip deps.
"""
from __future__ import annotations
import json
import os
import re
import ssl
import time
import traceback
import urllib.parse
import urllib.error
import urllib.request
_SSL_CTX = ssl._create_unverified_context() # LAN self-signed (SearXNG)
_SSL_CTX = ssl._create_unverified_context() # LAN self-signed certs
RID = os.environ.get("BM_REVIEWER_ID", "reviewer")
NAME = os.environ.get("BM_REVIEWER_NAME", RID)
ROLE = os.environ.get("BM_ROLE", "reviewer") # reviewer | synthesizer
MODEL = os.environ.get("BM_MODEL", "reviewer-a")
ROLE = os.environ.get("BM_ROLE", "grader") # extractor | grader | adjudicator
GID = os.environ.get("BM_GRADER_ID", ROLE)
NAME = os.environ.get("BM_GRADER_NAME", GID)
MODEL = os.environ.get("BM_MODEL", "grader-a")
LLM_BASE = os.environ.get("BM_LLM_BASE", "http://boardroom-proxy:4000/v1").rstrip("/")
LLM_KEY = os.environ.get("BM_LLM_KEY", "sk-local")
SEARXNG_URL = os.environ.get("BM_SEARXNG_URL", "").rstrip("/")
try:
TEMPERATURE = float(os.environ.get("BM_TEMPERATURE", "") or "0.3")
except ValueError:
@@ -48,156 +61,44 @@ except ValueError:
MAX_MODEL_LEN = 32768
DOCS = "/docs"
REPORTS = "/reports"
OUT_DIR = "/out"
BDEF_PATH = "/BDEF.md"
SCHEMA_PATH = "/schema.json"
PERSONA_PATH = "/persona/PERSONA.md"
RUBRIC_PATH = "/RUBRIC.md"
GRADES_DIR = "/grades"
EXTRACTION_PATH = "/extraction.json"
OUT_DIR = "/out"
# Leave headroom for the system/rubric/persona + the model's output; spend the
# rest on document text (~3 chars/token is a safe rough estimate).
DOC_BUDGET = max(8000, (MAX_MODEL_LEN - 3500) * 3)
MAX_STEPS = 8
MAX_OUTPUT_TOKENS = 2048
MAX_OUTPUT_TOKENS = 3072
TRANSPORT_RETRIES = 4
def log(msg: str) -> None:
print(f"[{GID}] {msg}", flush=True)
# ---------------------------------------------------------------- io helpers
def read_text(path: str, limit: int = 1_000_000) -> str:
def read_text(path: str, limit: int = 2_000_000) -> str:
try:
with open(path, errors="replace") as f:
return f.read()[:limit]
except FileNotFoundError:
except (FileNotFoundError, IsADirectoryError):
return ""
def _roots() -> list[str]:
return [DOCS, REPORTS] if ROLE == "synthesizer" else [DOCS]
def _safe(path: str) -> str:
"""Resolve a path inside an allowed read root; refuse escapes."""
cand = path or "."
for root in _roots():
p = os.path.realpath(os.path.join(root, cand) if not os.path.isabs(cand) else cand)
if p == root or p.startswith(root + os.sep):
return p
raise ValueError(f"path outside allowed roots: {path}")
def list_dir(root: str) -> list[str]:
out = []
if not os.path.isdir(root):
return out
for r, _dirs, files in os.walk(root):
for fn in files:
out.append(os.path.relpath(os.path.join(r, fn), root))
return sorted(out)
# ---------------------------------------------------------------- tools
def tool_list_files(args: dict) -> str:
lines = []
for root in _roots():
names = list_dir(root)
if names:
lines.append(f"{root}:")
lines += [f" {n}" for n in names]
return "\n".join(lines) or "(no files)"
def tool_read_file(args: dict) -> str:
p = _safe(args["path"])
try:
with open(p, errors="replace") as f:
return f.read()[:20000]
except FileNotFoundError:
return f"(no such file: {args['path']})"
except IsADirectoryError:
return f"(is a directory: {args['path']})"
def tool_web_search(args: dict) -> str:
if not SEARXNG_URL:
return "web search unavailable"
q = urllib.parse.urlencode({"q": args.get("query", ""), "format": "json"})
try:
req = urllib.request.Request(f"{SEARXNG_URL}/search?{q}", headers={"User-Agent": "boardroom-grader"})
with urllib.request.urlopen(req, timeout=20, context=_SSL_CTX) as resp:
data = json.loads(resp.read().decode())
lines = [f"- {r.get('title','')}\n {r.get('url','')}\n {r.get('content','')[:300]}"
for r in (data.get("results") or [])[:8]]
return "\n".join(lines) or "(no results)"
except Exception as e:
return f"search error: {e}"
DISPATCH = {"list_files": tool_list_files, "read_file": tool_read_file, "web_search": tool_web_search}
TOOLS = [
{"type": "function", "function": {
"name": "list_files", "description": "List the available document (and report) files.",
"parameters": {"type": "object", "properties": {}}}},
{"type": "function", "function": {
"name": "read_file", "description": "Read a document or report file by its path (from list_files).",
"parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}},
]
if SEARXNG_URL:
TOOLS.append({"type": "function", "function": {
"name": "web_search", "description": "Search the web via SearXNG; returns top results.",
"parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}})
def run_tool(name: str, args: dict) -> str:
fn = DISPATCH.get(name)
if not fn:
return f"(unknown tool: {name})"
try:
return fn(args)
except Exception as e:
return f"(tool error: {e})"
# ---------------------------------------------------------------- LLM
def chat(messages: list) -> dict:
body = {"model": MODEL, "messages": messages, "tools": TOOLS,
"tool_choice": "auto", "temperature": TEMPERATURE, "max_tokens": MAX_OUTPUT_TOKENS}
req = urllib.request.Request(
f"{LLM_BASE}/chat/completions",
data=json.dumps(body).encode(),
headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"},
method="POST")
with urllib.request.urlopen(req, timeout=300, context=_SSL_CTX) as resp:
data = json.loads(resp.read().decode())
return data["choices"][0]["message"]
_JSON_ACTION = re.compile(r"```(?:json)?\s*(\{.*?\})\s*```", re.DOTALL)
def _text_fallback_calls(content: str) -> list:
if not content:
return []
m = _JSON_ACTION.search(content)
if not m:
return []
try:
obj = json.loads(m.group(1))
except json.JSONDecodeError:
return []
name = obj.get("tool") or obj.get("name")
if name in DISPATCH:
return [{"id": "fallback", "function": {"name": name, "arguments": json.dumps(obj.get("args", {}))}}]
def _doc_names() -> list[str]:
if not os.path.isdir(DOCS):
return []
return sorted(fn for fn in os.listdir(DOCS)
if fn.endswith(".txt") and os.path.isfile(os.path.join(DOCS, fn)))
# ---------------------------------------------------------------- prompts
def _preload_docs() -> tuple[str, bool]:
"""Concatenate the document text up to DOC_BUDGET. Returns (text, truncated)."""
names = list_dir(DOCS)
def preload_docs(budget: int) -> tuple[str, bool]:
"""Concatenate /docs/*.txt up to `budget` chars. Returns (text, truncated)."""
chunks, used, truncated = [], 0, False
for n in names:
for n in _doc_names():
body = read_text(os.path.join(DOCS, n))
header = f"\n\n========== DOCUMENT: {n} ==========\n"
room = DOC_BUDGET - used
room = budget - used
if room <= 0:
truncated = True
break
@@ -209,129 +110,337 @@ def _preload_docs() -> tuple[str, bool]:
return "".join(chunks), truncated
def _preload_reports() -> str:
names = list_dir(REPORTS)
def out_path() -> str:
if ROLE == "adjudicator":
return os.path.join(OUT_DIR, "ADJUDICATION.md")
if ROLE == "extractor":
return os.path.join(OUT_DIR, "extraction.json")
return os.path.join(OUT_DIR, f"{GID}.json")
def write_out(text: str) -> None:
os.makedirs(OUT_DIR, exist_ok=True)
with open(out_path(), "w") as f:
f.write(text.rstrip() + "\n")
def write_invalid_marker(err: str) -> None:
try:
os.makedirs(OUT_DIR, exist_ok=True)
with open(out_path() + ".invalid", "w") as f:
f.write(err.strip() + "\n")
except Exception:
pass
# ---------------------------------------------------------------- LLM client
def _post(payload: dict, timeout: int = 600) -> dict:
req = urllib.request.Request(
f"{LLM_BASE}/chat/completions",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"},
method="POST")
with urllib.request.urlopen(req, timeout=timeout, context=_SSL_CTX) as resp:
return json.loads(resp.read().decode())
def chat(messages: list, extra: dict | None = None) -> str:
"""One completion. Retries with backoff on 5xx/connection errors; raises
HTTPError immediately on 4xx so the caller can walk the reliability ladder."""
payload = {"model": MODEL, "messages": messages,
"temperature": TEMPERATURE, "max_tokens": MAX_OUTPUT_TOKENS}
if extra:
payload.update(extra)
last: Exception | None = None
for attempt in range(1, TRANSPORT_RETRIES + 1):
try:
data = _post(payload)
return (data["choices"][0]["message"].get("content") or "").strip()
except urllib.error.HTTPError as e:
if 400 <= e.code < 500:
raise
last = e
except Exception as e: # URLError, timeout, bad JSON envelope...
last = e
sleep = 5 * attempt
log(f"transport error ({last}); retry {attempt}/{TRANSPORT_RETRIES} in {sleep}s")
time.sleep(sleep)
raise RuntimeError(f"model endpoint unreachable after {TRANSPORT_RETRIES} attempts: {last}")
def chat_json(messages: list, schema: dict) -> str:
"""The JSON reliability ladder: strict json_schema -> guided_json -> plain."""
ladder = [
("json_schema", {"response_format": {"type": "json_schema", "json_schema": {
"name": schema.get("title") or "output", "schema": schema, "strict": True}}}),
("guided_json", {"guided_json": schema}),
("plain", None),
]
last: Exception | None = None
for mode, extra in ladder:
try:
return chat(messages, extra=extra)
except urllib.error.HTTPError as e:
if 400 <= e.code < 500:
log(f"{mode} request rejected (HTTP {e.code}); trying next mode")
last = e
continue
raise
raise RuntimeError(f"all completion modes were rejected by the endpoint: {last}")
# ---------------------------------------------------------------- JSON parse
def parse_json(text: str):
"""Whole-reply json.loads, else the first brace-balanced {...} block."""
if not text:
return None
try:
return json.loads(text)
except json.JSONDecodeError:
pass
start = text.find("{")
while start != -1:
depth, in_str, esc = 0, False, False
for i in range(start, len(text)):
c = text[i]
if in_str:
if esc:
esc = False
elif c == "\\":
esc = True
elif c == '"':
in_str = False
elif c == '"':
in_str = True
elif c == "{":
depth += 1
elif c == "}":
depth -= 1
if depth == 0:
try:
return json.loads(text[start:i + 1])
except json.JSONDecodeError:
break
start = text.find("{", start + 1)
return None
def structural_error(obj) -> str | None:
"""Lightweight in-agent checks; the full jsonschema pass is orchestrator-side."""
if not isinstance(obj, dict):
return "top-level value is not a JSON object"
if ROLE == "extractor":
for k in ("schema_version", "deck", "kpis", "forward_targets",
"red_flag_candidates", "narrative"):
if k not in obj:
return f"missing required key: {k}"
if not isinstance(obj.get("deck"), dict) or "period" not in obj["deck"]:
return "deck.period is missing"
else: # grader
for k in ("schema_version", "grader", "categories", "red_flags", "overall_comment"):
if k not in obj:
return f"missing required key: {k}"
cats = obj.get("categories")
if not isinstance(cats, list) or len(cats) != 8:
return "categories must contain exactly 8 entries (A-H)"
return None
# ---------------------------------------------------------------- prompts
def _persona() -> str:
return read_text(PERSONA_PATH).strip()
def _bdef() -> str:
return read_text(BDEF_PATH).strip()
def _red_flag_taxonomy() -> str:
"""The '## Red-flag taxonomy' section of BDEF.md (falls back to the whole rubric)."""
bdef = _bdef()
low = bdef.lower()
i = low.find("## red-flag taxonomy")
return bdef[i:].strip() if i != -1 else bdef
def build_extractor_prompt(schema: dict) -> tuple[str, str, bool]:
schema_text = json.dumps(schema, indent=2)
system = (
"You are the structured-data EXTRACTOR for a board-deck grading pipeline. "
"You turn the deck text into machine-readable JSON; you do not grade.\n\n"
"HARD RULES:\n"
"- Reply with ONLY one JSON object conforming exactly to the schema below. "
"No prose, no markdown fences, no comments.\n"
"- NEVER invent numbers. Every actual/target must appear in the deck text; "
"record where in \"source\".\n"
"- canonical_name is lower_snake_case, generic, and stable across quarters "
"(arr, ebitda_margin, churn_rate...).\n"
"- deck.period is the reporting period as printed on the deck "
"(e.g. 2026-Q2, 2026-H1, FY2026, 2026-05); null if truly absent.\n"
"- If the deck text you were given was truncated, set deck.truncated = true.\n\n"
"# OUTPUT SCHEMA (JSON Schema)\n" + schema_text
)
persona = _persona()
if persona:
system = persona + "\n\n" + system
taxonomy = _red_flag_taxonomy()
prefix = ("# RED-FLAG TAXONOMY\nUse ONLY these codes when populating "
"red_flag_candidates:\n\n" + taxonomy + "\n\n# BOARD DECK TEXT\n")
suffix = ("\n\n# TASK\nExtract the deck metadata, every KPI actual, every "
"forward-looking target, red-flag candidates (taxonomy codes above), "
"and the narrative summary. Output the JSON object now.")
budget = max(8000, (MAX_MODEL_LEN - 3500) * 3 - len(system) - len(prefix) - len(suffix))
docs, truncated = preload_docs(budget)
note = ("\n\n(NOTE: the deck text above was TRUNCATED to fit the context window — "
"set deck.truncated = true.)" if truncated else "")
return system, prefix + docs + note + suffix, truncated
def build_grader_prompt(schema: dict) -> tuple[str, str, bool]:
schema_text = json.dumps(schema, indent=2)
system = (
f"You are '{NAME}', one grader on a panel scoring a portfolio-company board "
"deck against the BDEF rubric.\n\n"
"HARD RULES:\n"
"- Score every BDEF category A-H with an integer 1-5.\n"
"- Any score ABOVE or BELOW 3 REQUIRES verbatim evidence quotes from the deck, "
"each with a location (e.g. 'slide 6'). Unsupported non-3 scores will be "
"regressed to 3 by the pipeline.\n"
"- Do NOT compute totals or composite scores; numbers are computed elsewhere.\n"
f"- Set \"grader\" to exactly \"{GID}\".\n"
"- Use only the red-flag taxonomy codes defined in the rubric.\n"
"- Reply with ONLY one JSON object conforming exactly to the provided schema. "
"No prose, no markdown fences."
)
persona = _persona()
if persona:
system += "\n\n# YOUR LENS — how YOU specifically read this deck\n" + persona
bdef = _bdef()
prefix = "# BDEF RUBRIC\n" + bdef + "\n\n# BOARD DECK TEXT\n"
suffix = ("\n\n# OUTPUT SCHEMA (JSON Schema)\n" + schema_text +
"\n\n# TASK\nGrade the deck per the rubric and your lens. "
"Output the JSON object now.")
budget = max(8000, (MAX_MODEL_LEN - 3500) * 3 - len(system) - len(prefix) - len(suffix))
docs, truncated = preload_docs(budget)
note = ("\n\n(NOTE: the deck text above was TRUNCATED to fit the context window — "
"grade what is shown and mention the truncation in overall_comment.)"
if truncated else "")
return system, prefix + docs + note + suffix, truncated
def _load_panel_grades() -> str:
"""All /grades/*.json panel reports, skipping extraction.json and anything
flagged invalid by the agent that produced it (sibling .invalid marker)."""
parts = []
budget = DOC_BUDGET
used = 0
for n in names:
body = read_text(os.path.join(REPORTS, n))
header = f"\n\n========== REVIEWER REPORT: {n} ==========\n"
seg = (header + body)[: max(0, budget - used)]
parts.append(seg)
used += len(seg)
if not os.path.isdir(GRADES_DIR):
return ""
for fn in sorted(os.listdir(GRADES_DIR)):
p = os.path.join(GRADES_DIR, fn)
if not fn.endswith(".json") or not os.path.isfile(p):
continue
if fn == "extraction.json" or os.path.exists(p + ".invalid"):
continue
parts.append(f"\n\n========== GRADER REPORT: {fn} ==========\n" + read_text(p))
return "".join(parts)
def system_prompt() -> str:
persona = read_text(PERSONA_PATH).strip()
if ROLE == "synthesizer":
base = (f"You are '{NAME}', the lead reviewer chairing a document-review panel. "
"You are given the panel members' individual reports (and the source "
"documents for reference). Produce ONE consolidated report in Markdown.")
else:
base = (f"You are '{NAME}', an expert confidential-document reviewer. Read the "
"document(s) provided and produce ONE written report in Markdown. Base every "
"statement on the documents; never invent facts. Be specific and cite the "
"document/section for each point.")
if persona:
base += "\n\n# YOUR LENS — how YOU specifically read this\n" + persona
tools_note = (
"\n\nYou can call list_files and read_file to pull more content if what was "
"pre-loaded is truncated"
+ (", and web_search for external context" if SEARXNG_URL else "")
+ ". When done, reply with the FINAL report only — no tool call. If your client "
'cannot emit tool calls, reply with a single fenced block: '
'```json\\n{"tool":"read_file","args":{"path":"..."}}\\n``` and nothing else.'
def build_adjudicator_prompt() -> tuple[str, str]:
system = _persona() or (
"You are the adjudicator chairing a panel of board-deck graders. You did not "
"read the deck first-hand for a fresh opinion — you weigh the panel's evidence."
)
return base + tools_note
def first_user_message() -> str:
rubric = read_text(RUBRIC_PATH).strip() or "Produce a thorough review report."
if ROLE == "synthesizer":
reports = _preload_reports()
docs, truncated = _preload_docs()
return (f"# REVIEW RUBRIC\n{rubric}\n\n# PANEL REPORTS\n{reports}\n\n"
f"# SOURCE DOCUMENTS (for reference){' (truncated)' if truncated else ''}\n{docs}\n\n"
"# YOUR TASK\nConsolidate the panel's reports into one authoritative report per the "
"rubric: shared findings, conflicts (and your adjudication), anything only one "
"reviewer caught, and a prioritized overall recommendation. Attribute points to "
"reviewers. Output the final consolidated report now.")
docs, truncated = _preload_docs()
note = ("\n\n(Note: the documents were truncated to fit context — use read_file to pull any "
"section you need in full.)" if truncated else "")
return (f"# REVIEW RUBRIC\n{rubric}\n\n# DOCUMENT(S)\n{docs}{note}\n\n"
"# YOUR TASK\nReview the document(s) above per the rubric and your lens. Output your "
"final report now.")
extraction_txt = read_text(EXTRACTION_PATH)
grades_txt = _load_panel_grades()
body = ("# STRUCTURED EXTRACTION (ground truth pulled from the deck)\n" +
extraction_txt + "\n\n# PANEL GRADE REPORTS\n" + grades_txt)
budget = max(8000, (MAX_MODEL_LEN - 3500) * 3 - len(system) - 1500)
if len(body) > budget:
body = body[:budget] + "\n\n(NOTE: input truncated to fit the context window.)"
task = (
"\n\n# TASK\nWrite a MARKDOWN adjudication of the panel:\n"
"- Consensus per BDEF category A-H (one line each).\n"
"- Material disagreements: where graders diverge, what each cites, and whose "
"evidence is stronger (verbatim deck quotes beat assertions).\n"
"- Red flags: which are CONFIRMED and which are DISMISSED, and why.\n"
"- Exactly 3 questions the board should ask management next quarter.\n"
"Do NOT output numeric scores, totals, or JSON — narrative Markdown only."
)
return system, body + task
# ---------------------------------------------------------------- run
def out_path() -> str:
name = "CONSOLIDATED_REPORT.md" if ROLE == "synthesizer" else f"{RID}.md"
return os.path.join(OUT_DIR, name)
def finalize(obj: dict, truncated: bool) -> dict:
if ROLE == "grader":
obj["grader"] = GID
elif ROLE == "extractor" and truncated:
deck = obj.get("deck")
if isinstance(deck, dict):
deck["truncated"] = True
return obj
def write_report(text: str) -> None:
os.makedirs(OUT_DIR, exist_ok=True)
with open(out_path(), "w") as f:
f.write(text.strip() + "\n")
def run_json_role() -> None:
schema = json.loads(read_text(SCHEMA_PATH) or "{}")
if ROLE == "extractor":
system, user, truncated = build_extractor_prompt(schema)
else:
system, user, truncated = build_grader_prompt(schema)
if truncated:
log("document text truncated to fit the context window")
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
text = chat_json(messages, schema)
obj = parse_json(text)
err = structural_error(obj) if obj is not None else "reply was not parseable JSON"
if obj is not None and err is None:
write_out(json.dumps(finalize(obj, truncated), indent=2))
log(f"wrote {out_path()}")
return
# ONE repair round-trip.
log(f"invalid reply ({err}); attempting one repair round-trip")
repair = messages + [
{"role": "assistant", "content": text or "(empty reply)"},
{"role": "user", "content": (
f"Your previous reply was not valid JSON or failed validation: {err}. "
"Reply with ONLY the corrected JSON.")},
]
text2 = chat_json(repair, schema)
obj2 = parse_json(text2)
err2 = structural_error(obj2) if obj2 is not None else "reply was not parseable JSON"
if obj2 is not None and err2 is None:
write_out(json.dumps(finalize(obj2, truncated), indent=2))
log(f"wrote {out_path()} (after repair)")
return
# Still bad: leave the raw text + an .invalid marker for the orchestrator.
write_out(text2 or text or "")
write_invalid_marker(f"invalid after repair round-trip: {err2}")
log(f"FAILED to produce valid JSON: {err2} (raw text + .invalid marker written)")
def run() -> str:
messages = [{"role": "system", "content": system_prompt()},
{"role": "user", "content": first_user_message()}]
last_text = ""
for _ in range(MAX_STEPS):
msg = chat(messages)
content = msg.get("content") or ""
calls = msg.get("tool_calls") or []
if content.strip():
last_text = content.strip()
if not calls:
calls = _text_fallback_calls(content)
if not calls:
break # final report
messages.append({"role": "assistant", "content": content})
for c in calls:
args = json.loads(c["function"]["arguments"] or "{}")
res = run_tool(c["function"]["name"], args)
messages.append({"role": "user", "content": f"[tool {c['function']['name']} result]\n{res[:20000]}"})
continue
messages.append({"role": "assistant", "content": content or None, "tool_calls": calls})
for c in calls:
try:
args = json.loads(c["function"]["arguments"] or "{}")
except json.JSONDecodeError:
args = {}
res = run_tool(c["function"]["name"], args)
messages.append({"role": "tool", "tool_call_id": c.get("id", ""), "content": res[:20000]})
# If the model ended on a tool turn with no report text, ask once more plainly.
if not last_text.strip():
messages.append({"role": "user", "content": "Now output your final report in Markdown."})
try:
last_text = (chat(messages).get("content") or "").strip()
except Exception:
pass
return last_text
def run_adjudicator() -> None:
system, user = build_adjudicator_prompt()
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
text = chat(messages)
if not text.strip():
raise RuntimeError("model returned an empty adjudication")
write_out(text)
log(f"wrote {out_path()} ({len(text)} chars)")
def main() -> None:
print(f"[{RID}] reviewer starting (role={ROLE} model={MODEL} base={LLM_BASE})", flush=True)
log(f"starting (role={ROLE} model={MODEL} base={LLM_BASE} temp={TEMPERATURE})")
try:
report = run()
if not report.strip():
report = f"# {NAME}\n\n(The model returned no report text.)"
write_report(report)
print(f"[{RID}] report written to {out_path()} ({len(report)} chars)", flush=True)
if ROLE == "adjudicator":
run_adjudicator()
else:
run_json_role()
except Exception as e:
print(traceback.format_exc(), flush=True)
# Always leave a file so the orchestrator can see this reviewer ran.
try:
write_report(f"# {NAME} — ERROR\n\nThis reviewer failed: {e}\n")
except Exception:
pass
if ROLE != "adjudicator":
# Leave a marker so the orchestrator sees this agent ran and failed.
write_invalid_marker(f"agent error: {e}")
raise
startos/actions/configure-companies.ts
+185
View File
@@ -0,0 +1,185 @@
import { sdk } from '../sdk'
import { configFile } from '../file-models/config'
const { InputSpec, Value, List } = sdk
const inputSpec = InputSpec.of({
companies: Value.list(
List.obj(
{
name: 'Portfolio Companies',
description:
'One entry per portfolio company. The SLUG is the folder you drop its ' +
'decks into (inbox/<slug>/2026-Q2-deck.pdf) and the key its running ' +
'scorecard ledger lives under. Pinned targets are the KPIs the scorer ' +
'holds the company to even when a deck goes quiet about them — pin the ' +
'profitability thresholds especially, since they carry the heaviest ' +
'weight (30 of the quant 60).',
default: [],
minLength: 0,
maxLength: 64,
},
{
uniqueBy: 'slug',
displayAs: '{{slug}}',
spec: InputSpec.of({
slug: Value.text({
name: 'Slug',
description:
'Folder name under the inbox and the ledger key. Stable — do not ' +
'rename once decks have been graded.',
required: true,
default: null,
placeholder: 'acme-widgets',
patterns: [
{ regex: '^[a-z0-9][a-z0-9-]{0,40}$',
description: 'Lowercase letters, numbers, dashes (max 41 chars).' },
],
}),
name: Value.text({
name: 'Display Name (optional)',
description: 'Shown on the dashboard and scorecards. Empty = the slug.',
required: false,
default: null,
placeholder: 'Acme Widgets, Inc.',
}),
kpiAliases: Value.textarea({
name: 'KPI Aliases (optional)',
description:
'One line per KPI: canonical=alias1;alias2 — maps the names a deck ' +
'uses onto the canonical KPI name, so "Adj. EBITDA" and "EBITDA ' +
'(adj)" both land on the same pinned target.',
required: false,
default: null,
minRows: 2,
maxRows: 12,
placeholder: 'ebitda=Adj. EBITDA;EBITDA (adj)\nmrr=Monthly Recurring Revenue;MRR',
}),
pinnedTargets: Value.list(
List.obj(
{
name: 'Pinned KPI Targets',
description:
'Targets graded every quarter regardless of what the deck ' +
'chooses to show. Mark profitability KPIs (EBITDA, net margin, ' +
'FCF...) so they score in the heavier profitability bucket.',
default: [],
minLength: 0,
maxLength: 32,
},
{
uniqueBy: 'kpi',
displayAs: '{{kpi}} → {{target}}',
spec: InputSpec.of({
kpi: Value.text({
name: 'KPI Name',
description: 'Canonical KPI name (see the aliases field above).',
required: true,
default: null,
placeholder: 'ebitda',
}),
target: Value.number({
name: 'Target',
description: 'The numeric target for the period.',
required: true,
default: null,
integer: false,
}),
unit: Value.text({
name: 'Unit (optional)',
description: 'For display only, e.g. "USD", "%", "customers".',
required: false,
default: null,
placeholder: 'USD',
}),
direction: Value.select({
name: 'Direction',
description: 'Whether hitting the target means being at-or-above it (revenue) or at-or-below it (churn, burn).',
default: 'gte',
values: {
gte: 'At or above target',
lte: 'At or below target',
},
}),
profitability: Value.toggle({
name: 'Profitability KPI',
description: 'Score this KPI in the profitability bucket (heaviest weight) instead of the general KPI bucket.',
default: false,
}),
}),
},
),
),
}),
},
),
),
})
export const configureCompanies = sdk.Action.withInput(
'configure-companies',
async ({ effects }) => ({
name: 'Configure Companies',
description: 'Define the portfolio companies, their inbox slugs, KPI aliases, and pinned KPI targets.',
warning: null,
allowedStatuses: 'any',
group: 'Grading',
visibility: 'enabled',
}),
inputSpec,
async ({ effects }) => {
const cfg = await configFile.read().const(effects)
if (!cfg) return {}
return {
companies: cfg.companies.map((c) => ({
slug: c.slug,
name: c.name || undefined,
kpiAliases: c.kpiAliases || undefined,
pinnedTargets: c.pinnedTargets.map((t) => ({
kpi: t.kpi,
target: t.target,
unit: t.unit || undefined,
direction: t.direction,
profitability: t.profitability,
})),
})),
}
},
async ({ effects, input }) => {
await configFile.merge(effects, {
companies: input.companies.map((c) => ({
slug: c.slug,
name: c.name ?? '',
kpiAliases: c.kpiAliases ?? '',
pinnedTargets: c.pinnedTargets.map((t) => ({
kpi: t.kpi,
target: t.target,
unit: t.unit ?? '',
direction: t.direction,
profitability: t.profitability,
})),
})),
})
return {
version: '1',
title: 'Companies Configured',
message:
'Saved ' + input.companies.length + ' company(ies). Drop each company\'s ' +
'decks into inbox/<slug>/ (e.g. inbox/' +
(input.companies[0]?.slug || 'acme-widgets') +
'/2026-Q2-deck.pdf), then run "Grade Decks".',
result: {
type: 'single',
value: input.companies.map((c) => c.slug).join(', ') || '(none)',
copyable: false,
qr: false,
masked: false,
},
}
},
)
startos/actions/configure-graders.ts
+31 -28
View File
@@ -4,14 +4,15 @@ import { configFile } from '../file-models/config'
const { InputSpec, Value, List } = sdk
const inputSpec = InputSpec.of({
reviewers: Value.list(
graders: Value.list(
List.obj(
{
name: 'Review Panel',
name: 'Grading Panel',
description:
'Who sits on the panel — one entry per review. Add as many as you like. ' +
'Each reviewer is a model from your catalog plus a PERSONA: the lens it ' +
'reads through, so the same document gets examined from different angles.',
'Who grades the decks — one entry per grader. Add as many as you like. ' +
'Each grader is a model from your catalog plus a PERSONA: the lens it ' +
'grades through, so the same deck gets scored from different angles ' +
'before the deterministic composite is computed.',
default: [],
minLength: 1,
maxLength: 32,
@@ -22,10 +23,10 @@ const inputSpec = InputSpec.of({
spec: InputSpec.of({
name: Value.text({
name: 'Name',
description: 'Unique reviewer name. Becomes its container and report filename.',
description: 'Unique grader name. Becomes its container and grade-sheet filename.',
required: true,
default: null,
placeholder: 'risk-counsel',
placeholder: 'munger-lens',
patterns: [
{ regex: '^[A-Za-z0-9][A-Za-z0-9 _-]{0,40}$',
description: 'Letters, numbers, spaces, dashes, underscores (max 41 chars).' },
@@ -33,31 +34,32 @@ const inputSpec = InputSpec.of({
}),
model: Value.text({
name: 'Model Alias',
description: 'Which catalog model this reviewer uses (must match an alias from "Configure Models").',
description: 'Which catalog model this grader uses (must match an alias from "Configure Models").',
required: true,
default: null,
placeholder: 'reviewer-a',
placeholder: 'grader-a',
}),
persona: Value.textarea({
name: 'Persona / Lens',
description:
'How THIS reviewer should read the documents — its priorities and ' +
'weighting. Injected into its system prompt. e.g. "You are skeptical ' +
'legal counsel: weight liability, ambiguous obligations, and missing ' +
'clauses above everything." Leave empty for a neutral reviewer.',
'How THIS grader should read the deck — its priorities and ' +
'weighting within the BDEF rubric. Injected into its system prompt. ' +
'e.g. "You are a Munger-style inversion skeptic: ask what would have ' +
'to be true for this deck to be hiding a deteriorating business." ' +
'Leave empty for a neutral grader.',
required: false,
default: null,
minRows: 3,
maxRows: 16,
placeholder:
'You are a financial-controls reviewer. Focus on numbers that do not ' +
'reconcile, unstated assumptions behind projections, and anything that ' +
'would concern an auditor. Organize findings by severity.',
'You are a Girdley-style operator. Weight unit economics, owner ' +
'accountability, and whether the KPIs the board was promised last ' +
'quarter are still being reported. Flag every silently dropped metric.',
}),
temperature: Value.number({
name: 'Sampling Temperature (optional)',
description:
'Best-effort per-reviewer sampling temperature for extra diversity. ' +
'Best-effort per-grader sampling temperature for extra diversity. ' +
'Persona is the primary lever. Leave empty to use the model default.',
required: false,
default: null,
@@ -71,15 +73,15 @@ const inputSpec = InputSpec.of({
),
})
export const configureReviewers = sdk.Action.withInput(
'configure-reviewers',
export const configureGraders = sdk.Action.withInput(
'configure-graders',
async ({ effects }) => ({
name: 'Configure Reviewers',
description: 'Define the review panel: which models and which personas, and how many reviews.',
name: 'Configure Graders',
description: 'Define the grading panel: which models and which personas grade each deck.',
warning: null,
allowedStatuses: 'any',
group: null,
group: 'Grading',
visibility: 'enabled',
}),
@@ -88,22 +90,23 @@ export const configureReviewers = sdk.Action.withInput(
async ({ effects }) => {
const cfg = await configFile.read().const(effects)
if (!cfg) return {}
return { reviewers: cfg.reviewers }
return { graders: cfg.graders }
},
async ({ effects, input }) => {
await configFile.merge(effects, { reviewers: input.reviewers })
await configFile.merge(effects, { graders: input.graders })
return {
version: '1',
title: 'Panel Configured',
message:
'Saved a panel of ' + input.reviewers.length + ' reviewer(s). Each model ' +
'alias must exist in "Configure Models". Set the rubric in "Configure ' +
'Review", then drop documents and run a review.',
'Saved a panel of ' + input.graders.length + ' grader(s). Each model ' +
'alias must exist in "Configure Models". Set the scoring knobs in ' +
'"Configure Grading", add companies in "Configure Companies", then drop ' +
'decks into inbox/<company-slug>/ and run "Grade Decks".',
result: {
type: 'single',
value: input.reviewers.map((r) => r.name).join(', '),
value: input.graders.map((g) => g.name).join(', '),
copyable: false,
qr: false,
masked: false,
startos/actions/configure-grading.ts
+171 -48
View File
@@ -4,26 +4,33 @@ import { configFile } from '../file-models/config'
const { InputSpec, Value } = sdk
const inputSpec = InputSpec.of({
reviewInstructions: Value.textarea({
name: 'Review Rubric',
bdefOverride: Value.textarea({
name: 'BDEF Rubric Override (optional)',
description:
'What every reviewer should look for and produce. Layered above each ' +
'reviewer\'s persona. Be concrete about the structure you want back.',
required: true,
'Leave EMPTY to grade against the baked-in BDEF v1.1 framework ' +
'(Girdley + Munger/Buffett). Non-empty text replaces the rubric wholesale, ' +
'so include the qualitative categories A-H if you customize it.',
required: false,
default: null,
minRows: 5,
maxRows: 20,
placeholder:
'Review the attached document(s). Produce: a short summary, key findings, ' +
'risks/red flags, open questions, and recommendations. Cite the document and ' +
'section for each point. Never invent facts not present in the documents.',
maxRows: 24,
placeholder: '(empty = the built-in BDEF v1.1 rubric)',
}),
extractorModel: Value.text({
name: 'Extractor Model (optional)',
description:
'Catalog alias of the model that runs the stage-1 structured KPI ' +
'extraction over each deck. Empty = use the first model.',
required: false,
default: null,
placeholder: 'grader-a',
}),
networkMode: Value.select({
name: 'Network Mode',
description:
'Air-gapped: reviewers reach ONLY the on-Spark model proxy — zero internet, ' +
'documents never leave your hardware (models must be pre-pulled into the ' +
'Spark HF cache, all on the head Spark). Local services: reviewers may also ' +
'Air-gapped: graders reach ONLY the on-Spark model proxy — zero internet, ' +
'board decks never leave your hardware (models must be pre-pulled into the ' +
'Spark HF cache, all on the head Spark). Local services: graders may also ' +
'reach LAN services like SearXNG and the second Spark (this network has ' +
'egress unless you firewall it).',
default: 'airgapped',
@@ -34,59 +41,151 @@ const inputSpec = InputSpec.of({
}),
searxngUrl: Value.text({
name: 'SearXNG URL (local-services only)',
description: 'JSON-search endpoint to give reviewers a web_search tool. Ignored in air-gapped mode. Empty = no web search.',
description: 'JSON-search endpoint to give graders a web_search tool. Ignored in air-gapped mode. Empty = no web search.',
required: false,
default: null,
placeholder: 'https://searxng.local',
}),
synthesisEnabled: Value.toggle({
name: 'Synthesize a Consolidated Report',
adjudicatorEnabled: Value.toggle({
name: 'Run an Adjudicator',
description:
'After the panel finishes, run a local "lead reviewer" that reads all the ' +
'individual reports and writes one consolidated report (themes, conflicts, ' +
'consensus, recommendation). No frontier model — stays on the Sparks.',
'After the panel finishes, run a local "lead grader" that reads every ' +
'grade sheet, reconciles disagreements, and settles the qualitative scores ' +
'the deterministic composite uses. No frontier model — stays on the Sparks.',
default: true,
}),
synthesisModel: Value.text({
name: 'Lead Reviewer Model (optional)',
description: 'Catalog alias of the model that writes the consolidated report. Empty = use the first model.',
adjudicatorModel: Value.text({
name: 'Adjudicator Model (optional)',
description: 'Catalog alias of the model that adjudicates the panel. Empty = use the first model.',
required: false,
default: null,
placeholder: 'reviewer-a',
placeholder: 'grader-a',
}),
synthesisPersona: Value.textarea({
name: 'Lead Reviewer Instructions (optional)',
description: 'Override how the consolidated report is written. Empty = a sensible built-in default.',
adjudicatorPersona: Value.textarea({
name: 'Adjudicator Instructions (optional)',
description: 'Override how the adjudicator reconciles the panel. Empty = a sensible built-in default.',
required: false,
default: null,
minRows: 3,
maxRows: 14,
}),
wipeRemoteDocs: Value.toggle({
name: 'Wipe Documents From Sparks After Review',
name: 'Wipe Decks From Sparks After Grading',
description:
'Delete the extracted document text from the Sparks when a job finishes. ' +
'Reports are always kept on this StartOS box. Recommended for confidential material.',
'Delete the extracted deck text from the Sparks when a job finishes. ' +
'Scorecards and ledgers are always kept on this StartOS box. Recommended ' +
'for confidential board material.',
default: true,
}),
autoRunOnDrop: Value.toggle({
name: 'Auto-run When Documents Are Dropped',
name: 'Auto-grade When Decks Are Dropped',
description:
'Start a review automatically (after a short debounce) whenever new files ' +
'land in the inbox. Off by default so you trigger reviews explicitly.',
'Start grading automatically (after a short debounce) whenever new decks ' +
'land in the inbox. Off by default so you trigger grading explicitly.',
default: false,
}),
// --- Deterministic-scorer weights (composite = quant 60 + qual 40 - flags) ---
profitabilityKpi: Value.number({
name: 'Weight: Profitability KPIs',
description: 'Points for the profitability KPI attainment bucket — the heaviest slice of the quant 60. Default 30.',
required: true,
default: 30,
integer: false,
min: 0,
max: 100,
}),
otherKpi: Value.number({
name: 'Weight: Other KPIs',
description: 'Points for the non-profitability measurable-KPI bucket. Default 20.',
required: true,
default: 20,
integer: false,
min: 0,
max: 100,
}),
forecastIntegrity: Value.number({
name: 'Weight: Forecast Integrity',
description: 'Points for deck N actuals hitting what deck N-1 promised. Default 10.',
required: true,
default: 10,
integer: false,
min: 0,
max: 100,
}),
qualCategoryMax: Value.number({
name: 'Weight: Max Per Qualitative Category',
description: 'Max points per BDEF category A-H (8 categories x 5 = the qualitative 40). Default 5.',
required: true,
default: 5,
integer: false,
min: 0,
max: 100,
}),
redFlagCap: Value.number({
name: 'Red-flag Penalty Cap',
description: 'Maximum total points red flags can subtract from the composite. Default 15.',
required: true,
default: 15,
integer: false,
min: 0,
max: 100,
}),
kpiCreditFloor: Value.number({
name: 'KPI Credit Floor',
description: 'actual/target ratio below which a KPI earns zero credit (linear credit above it). Default 0.5.',
required: true,
default: 0.5,
integer: false,
min: 0,
max: 1,
}),
droppedKpiPenalty: Value.number({
name: 'Dropped-KPI Penalty',
description: 'Penalty per KPI that silently disappeared from the deck. Default 2.',
required: true,
default: 2,
integer: false,
min: 0,
max: 100,
}),
droppedKpiMax: Value.number({
name: 'Dropped-KPI Flag Limit',
description: 'Count at most this many dropped-KPI flags per deck. Default 3.',
required: true,
default: 3,
integer: true,
min: 0,
max: 50,
}),
evidenceFullCredit: Value.number({
name: 'Evidence Full-credit Threshold',
description: 'Quote characters a qualitative finding needs for full weight (thinner evidence scales down). Default 400.',
required: true,
default: 400,
integer: true,
min: 0,
max: 100000,
}),
singleSourceFlagFactor: Value.number({
name: 'Single-source Flag Damping',
description: 'Multiplier applied to red flags raised by only one grader. Default 0.5.',
required: true,
default: 0.5,
integer: false,
min: 0,
max: 1,
}),
})
export const configureReview = sdk.Action.withInput(
'configure-review',
export const configureGrading = sdk.Action.withInput(
'configure-grading',
async ({ effects }) => ({
name: 'Configure Review',
description: 'Set the rubric, air-gap mode, synthesis, and document retention.',
name: 'Configure Grading',
description: 'Set the BDEF rubric, air-gap mode, adjudication, retention, and the scoring weights.',
warning: null,
allowedStatuses: 'any',
group: null,
group: 'Grading',
visibility: 'enabled',
}),
@@ -96,36 +195,60 @@ export const configureReview = sdk.Action.withInput(
const cfg = await configFile.read().const(effects)
if (!cfg) return {}
return {
reviewInstructions: cfg.reviewInstructions,
bdefOverride: cfg.bdefOverride || undefined,
extractorModel: cfg.extractorModel || undefined,
networkMode: cfg.networkMode,
searxngUrl: cfg.searxngUrl || undefined,
synthesisEnabled: cfg.synthesisEnabled,
synthesisModel: cfg.synthesisModel || undefined,
synthesisPersona: cfg.synthesisPersona || undefined,
adjudicatorEnabled: cfg.adjudicatorEnabled,
adjudicatorModel: cfg.adjudicatorModel || undefined,
adjudicatorPersona: cfg.adjudicatorPersona || undefined,
wipeRemoteDocs: cfg.wipeRemoteDocs,
autoRunOnDrop: cfg.autoRunOnDrop,
profitabilityKpi: cfg.weights.profitabilityKpi,
otherKpi: cfg.weights.otherKpi,
forecastIntegrity: cfg.weights.forecastIntegrity,
qualCategoryMax: cfg.weights.qualCategoryMax,
redFlagCap: cfg.weights.redFlagCap,
kpiCreditFloor: cfg.weights.kpiCreditFloor,
droppedKpiPenalty: cfg.weights.droppedKpiPenalty,
droppedKpiMax: cfg.weights.droppedKpiMax,
evidenceFullCredit: cfg.weights.evidenceFullCredit,
singleSourceFlagFactor: cfg.weights.singleSourceFlagFactor,
}
},
async ({ effects, input }) => {
await configFile.merge(effects, {
reviewInstructions: input.reviewInstructions,
bdefOverride: input.bdefOverride ?? '',
extractorModel: input.extractorModel ?? '',
networkMode: input.networkMode,
searxngUrl: input.searxngUrl ?? '',
synthesisEnabled: input.synthesisEnabled,
synthesisModel: input.synthesisModel ?? '',
synthesisPersona: input.synthesisPersona ?? '',
adjudicatorEnabled: input.adjudicatorEnabled,
adjudicatorModel: input.adjudicatorModel ?? '',
adjudicatorPersona: input.adjudicatorPersona ?? '',
wipeRemoteDocs: input.wipeRemoteDocs,
autoRunOnDrop: input.autoRunOnDrop,
weights: {
profitabilityKpi: input.profitabilityKpi,
otherKpi: input.otherKpi,
forecastIntegrity: input.forecastIntegrity,
qualCategoryMax: input.qualCategoryMax,
redFlagCap: input.redFlagCap,
kpiCreditFloor: input.kpiCreditFloor,
droppedKpiPenalty: input.droppedKpiPenalty,
droppedKpiMax: input.droppedKpiMax,
evidenceFullCredit: input.evidenceFullCredit,
singleSourceFlagFactor: input.singleSourceFlagFactor,
},
})
return {
version: '1',
title: 'Review Settings Saved',
title: 'Grading Settings Saved',
message:
input.networkMode === 'airgapped'
? 'Saved. Reviewers will run air-gapped (no internet). Ensure all models are on the head Spark and pre-pulled into its HF cache.'
: 'Saved. Reviewers run in local-services mode and may reach the network — make sure that is acceptable for these documents.',
? 'Saved. Graders will run air-gapped (no internet). Ensure all models are on the head Spark and pre-pulled into its HF cache.'
: 'Saved. Graders run in local-services mode and may reach the network — make sure that is acceptable for these board decks.',
result: { type: 'single', value: input.networkMode, copyable: false, qr: false, masked: false },
}
},
startos/actions/configure-models.ts
+8 -8
View File
@@ -9,7 +9,7 @@ const inputSpec = InputSpec.of({
{
name: 'Model Catalog',
description:
'The local models this service can serve on your Sparks. Each reviewer ' +
'The local models this service can serve on your Sparks. Each grader ' +
'references one of these by its alias. The job runner loads models in ' +
'waves so you can run a panel across more models than fit in GPU memory ' +
'at once.',
@@ -23,10 +23,10 @@ const inputSpec = InputSpec.of({
spec: InputSpec.of({
alias: Value.text({
name: 'Alias',
description: 'Short name reviewers use to pick this model (e.g. "qwen-32b").',
description: 'Short name graders use to pick this model (e.g. "qwen-32b").',
required: true,
default: null,
placeholder: 'reviewer-a',
placeholder: 'grader-a',
patterns: [
{ regex: '^[a-z0-9][a-z0-9-]{0,30}$',
description: 'Lowercase letters, numbers, dashes (max 31 chars).' },
@@ -42,7 +42,7 @@ const inputSpec = InputSpec.of({
spark: Value.select({
name: 'Served On',
description:
'Which Spark serves this model. Air-gapped review mode requires the ' +
'Which Spark serves this model. Air-gapped grading mode requires the ' +
'head (primary) Spark; the secondary is used only in local-services mode.',
default: 'primary',
values: { primary: 'Primary (head) Spark', secondary: 'Secondary Spark' },
@@ -68,7 +68,7 @@ const inputSpec = InputSpec.of({
}),
maxModelLen: Value.number({
name: 'Max Model Length',
description: 'vLLM --max-model-len (context window). Documents are chunked to fit.',
description: 'vLLM --max-model-len (context window). Deck text is chunked to fit.',
required: true,
default: 32768,
integer: true,
@@ -77,7 +77,7 @@ const inputSpec = InputSpec.of({
toolCallParser: Value.text({
name: 'Tool-Call Parser',
description:
'vLLM tool-call parser for the reviewer\'s read-file tool loop. Match the ' +
'vLLM tool-call parser for the grader\'s read-file tool loop. Match the ' +
'served model family (Qwen3 → "hermes"). Empty disables native tool-calling.',
required: false,
default: 'hermes',
@@ -112,7 +112,7 @@ export const configureModels = sdk.Action.withInput(
description: 'Define the local model catalog served on your Sparks and the serving knobs.',
warning: null,
allowedStatuses: 'any',
group: null,
group: 'Setup',
visibility: 'enabled',
}),
@@ -146,7 +146,7 @@ export const configureModels = sdk.Action.withInput(
title: 'Models Configured',
message:
'Saved ' + input.models.length + ' model(s). Make sure each is present in ' +
'the Spark HF cache for air-gapped runs, then set "Configure Reviewers".',
'the Spark HF cache for air-gapped runs, then set "Configure Graders".',
result: {
type: 'single',
value: input.models.map((m) => m.alias).join(', '),
startos/actions/configure-sparks.ts
+8 -8
View File
@@ -7,7 +7,7 @@ const { InputSpec, Value } = sdk
const inputSpec = InputSpec.of({
primarySparkHost: Value.text({
name: 'Primary Spark Host',
description: 'Hostname or IP of the head DGX Spark (reachable over SSH). Serves models, hosts the model proxy, and runs the reviewer panel.',
description: 'Hostname or IP of the head DGX Spark (reachable over SSH). Serves models, hosts the model proxy, and runs the grading panel.',
required: true,
default: null,
placeholder: 'spark-01.local',
@@ -46,8 +46,8 @@ const inputSpec = InputSpec.of({
name: 'Use Both Sparks',
description:
'Allow models to be served on a second Spark (over ConnectX/200GbE) for ' +
'extra capacity. NOTE: in air-gapped review mode all models must run on the ' +
'head Spark; the second Spark is used only in local-services mode.',
'extra capacity. NOTE: in air-gapped grading mode all models must run on ' +
'the head Spark; the second Spark is used only in local-services mode.',
default: false,
}),
secondarySparkHost: Value.text({
@@ -67,7 +67,7 @@ const inputSpec = InputSpec.of({
}),
remoteWorkDir: Value.text({
name: 'Remote Work Directory',
description: 'Absolute path on the head Spark for staged document text, the HF cache, and logs.',
description: 'Absolute path on the head Spark for staged deck text, the HF cache, and logs.',
required: true,
default: '/home/nvidia/boardroom-map',
}),
@@ -78,8 +78,8 @@ const inputSpec = InputSpec.of({
default: 'boardroom-vllm:latest',
}),
graderImage: Value.text({
name: 'Reviewer Image Tag',
description: 'The sandboxed reviewer image built on the head Spark from sandbox/build.sh.',
name: 'Grader Image Tag',
description: 'The sandboxed grader image built on the head Spark from sandbox/build.sh.',
required: true,
default: 'boardroom-grader:latest',
}),
@@ -103,7 +103,7 @@ export const configureSparks = sdk.Action.withInput(
description: 'Set the DGX Spark connection details, SSH credentials, and image tags.',
warning: null,
allowedStatuses: 'any',
group: null,
group: 'Setup',
visibility: 'enabled',
}),
@@ -154,7 +154,7 @@ export const configureSparks = sdk.Action.withInput(
title: 'Sparks Configured',
message:
'Saved. Use "Test Spark Connection" to verify SSH + GPU access, then set ' +
'"Configure Models" and "Configure Reviewers".',
'"Configure Models" and "Configure Graders".',
result: { type: 'single', value: input.primarySparkHost, copyable: false, qr: false, masked: false },
}
},
startos/actions/grade-decks.ts
+12 -12
View File
@@ -2,17 +2,17 @@ import { startSdk } from '@start9labs/start-sdk'
import { sdk } from '../sdk'
/**
* Trigger a review of whatever is currently in the inbox. The running job-runner
* Trigger grading of whatever is currently in the inbox. The running job-runner
* thread polls for /data/state/run_request and starts a job when it appears, so
* this action just drops that request file (decoupled from the daemon — no need
* to reach its HTTP port from the action's one-shot container).
*/
export const runReview = sdk.Action.withoutInput(
'run-review',
export const gradeDecks = sdk.Action.withoutInput(
'grade-decks',
async ({ effects }) => ({
name: 'Run Review',
description: 'Convene the panel now over the documents currently in the inbox.',
name: 'Grade Decks',
description: 'Grade all decks currently in the inbox (inbox/<company-slug>/...).',
warning: null,
allowedStatuses: 'only-running',
group: null,
@@ -36,23 +36,23 @@ export const runReview = sdk.Action.withoutInput(
'sh',
'-c',
'mkdir -p /data/state && date +%s > /data/state/run_request && ' +
'n=$(ls -1 /data/inbox 2>/dev/null | wc -l | tr -d " "); ' +
'echo "Review requested. $n file(s) in the inbox."',
'n=$(find /data/inbox -type f 2>/dev/null | wc -l | tr -d " "); ' +
'echo "Grading requested. $n deck file(s) in the inbox."',
],
{ mounts, env: { BM_DATA_DIR: '/data' } },
'run-review',
'grade-decks',
)
output = (stdout?.toString() || '').trim() || 'Review requested.'
output = (stdout?.toString() || '').trim() || 'Grading requested.'
} catch (e: any) {
output = 'Could not request a review: ' + (e?.message || String(e))
output = 'Could not request grading: ' + (e?.message || String(e))
}
return {
version: '1',
title: 'Review Requested',
title: 'Grading Requested',
message:
output +
' Watch the Web UI for progress; reports appear there and via "View Latest Report".',
' Watch the Web UI for progress; scorecards appear there and via "View Latest Scorecard".',
result: { type: 'single', value: output, copyable: false, qr: false, masked: false },
}
},
startos/actions/index.ts
+15 -10
View File
@@ -1,17 +1,22 @@
import { sdk } from '../sdk'
import { configureSparks } from './configure-sparks'
import { configureModels } from './configure-models'
import { configureReviewers } from './configure-reviewers'
import { configureReview } from './configure-review'
import { runReview } from './run-review'
import { testConnection } from './test-connection'
import { latestReport } from './latest-report'
import { configureModels } from './configure-models'
import { configureGraders } from './configure-graders'
import { configureGrading } from './configure-grading'
import { configureCompanies } from './configure-companies'
import { gradeDecks } from './grade-decks'
import { latestScorecard } from './latest-scorecard'
// Setup group: Sparks -> connection test -> model catalog.
// Grading group: panel -> scoring knobs -> portfolio companies.
// Ungrouped (day-to-day): grade the inbox, read the latest scorecard.
export const actions = sdk.Actions.of()
.addAction(configureSparks)
.addAction(configureModels)
.addAction(configureReviewers)
.addAction(configureReview)
.addAction(runReview)
.addAction(testConnection)
.addAction(latestReport)
.addAction(configureModels)
.addAction(configureGraders)
.addAction(configureGrading)
.addAction(configureCompanies)
.addAction(gradeDecks)
.addAction(latestScorecard)
startos/actions/latest-scorecard.ts
+21 -14
View File
@@ -2,17 +2,18 @@ import { startSdk } from '@start9labs/start-sdk'
import { sdk } from '../sdk'
/**
* Returns the latest report as a copyable result, so you can read it straight
* from the StartOS service page without opening the Web UI. The job runner saves
* the most recent report (consolidated if synthesis is on, else the panel
* digest) to /data/reports/latest.md on the StartOS host — no Spark round-trip.
* Returns the latest scorecard as a copyable result, so you can read it straight
* from the StartOS service page without opening the dashboard. The job runner
* saves the most recent scorecard to /data/reports/latest-scorecard.md (with
* /data/reports/latest.md as the legacy fallback) on the StartOS host — no
* Spark round-trip.
*/
export const latestReport = sdk.Action.withoutInput(
'latest-report',
export const latestScorecard = sdk.Action.withoutInput(
'latest-scorecard',
async ({ effects }) => ({
name: 'View Latest Report',
description: 'Show the most recent review report produced by the panel.',
name: 'View Latest Scorecard',
description: 'Show the most recent deck scorecard produced by the grading panel.',
warning: null,
allowedStatuses: 'any',
group: null,
@@ -32,19 +33,25 @@ export const latestReport = sdk.Action.withoutInput(
const { stdout } = await startSdk.runCommand<typeof sdk.manifest>(
effects,
{ imageId: 'main' },
['sh', '-c', 'cat /data/reports/latest.md 2>/dev/null || echo "(no report yet — drop documents in the inbox and run a review)"'],
[
'sh',
'-c',
'cat /data/reports/latest-scorecard.md 2>/dev/null || ' +
'cat /data/reports/latest.md 2>/dev/null || ' +
'echo "(no scorecard yet — drop decks into inbox/<company-slug>/ and run Grade Decks)"',
],
{ mounts, env: { BM_DATA_DIR: '/data' } },
'latest-report',
'latest-scorecard',
)
report = (stdout?.toString() || '').trim() || '(no report yet)'
report = (stdout?.toString() || '').trim() || '(no scorecard yet)'
} catch (e: any) {
report = 'Could not read report: ' + (e?.message || String(e))
report = 'Could not read scorecard: ' + (e?.message || String(e))
}
return {
version: '1',
title: 'Latest Boardroom Map Report',
message: 'The panel\'s most recent review.',
title: 'Latest Scorecard',
message: 'The panel\'s most recent deck scorecard.',
result: { type: 'single', value: report, copyable: true, qr: false, masked: false },
}
},
startos/actions/test-connection.ts
+5 -4
View File
@@ -3,18 +3,19 @@ import { sdk } from '../sdk'
/**
* Runs a one-shot in the orchestrator image that SSHes into the configured
* Spark(s) and reports `nvidia-smi` plus whether the vLLM + reviewer images are
* built. Reuses orchestrator/spark_client.py so SSH logic lives in one place.
* Spark(s) and reports `nvidia-smi` plus whether the vLLM (boardroom-vllm) and
* grader (boardroom-grader) images are built. Reuses orchestrator/spark_client.py
* so SSH logic lives in one place.
*/
export const testConnection = sdk.Action.withoutInput(
'test-connection',
async ({ effects }) => ({
name: 'Test Spark Connection',
description: 'SSH into the configured Spark(s) and verify GPU + serving/reviewer image access.',
description: 'SSH into the configured Spark(s) and verify GPU + serving/grader image access.',
warning: null,
allowedStatuses: 'any',
group: null,
group: 'Setup',
visibility: 'enabled',
}),
startos/file-models/config.ts
+124 -43
View File
@@ -3,16 +3,21 @@ import { FileHelper, z } from '@start9labs/start-sdk'
/**
* Boardroom Map configuration, persisted to the `main` volume as config.json.
*
* Written by the StartOS actions (Configure Sparks / Models / Reviewers /
* Review) and read by the Python orchestrator inside the container, which mounts
* the same volume at /data and reads /data/config.json. Keep field names in sync
* with orchestrator/bm_config.py (CONFIG_DEFAULTS).
* Written by the StartOS actions (Configure Sparks / Models / Graders /
* Grading / Companies) and read by the Python orchestrator inside the
* container, which mounts the same volume at /data and reads /data/config.json.
* Keep field names in sync with orchestrator/bm_config.py (CONFIG_DEFAULTS).
*
* Boardroom Map is a CONTROL PLANE: a GPU-free orchestrator on StartOS that SSHes into
* one or two DGX Sparks to serve local models and run a panel of sandboxed
* "reviewer" containers over confidential documents you drop in. There is NO
* frontier model and NO cloud key — everything stays on your hardware. The only
* secrets are the Spark SSH key and an optional Hugging Face token (secrets.ts).
* Boardroom Map is a CONTROL PLANE: a GPU-free orchestrator on StartOS that
* SSHes into one or two DGX Sparks to serve local models and grade the board
* decks you drop into /data/inbox/<company-slug>/. A panel of sandboxed
* "grader" containers scores each deck against the BDEF v1.1 framework
* (Girdley + Munger/Buffett); Python then computes a deterministic composite
* (quant KPI attainment 60 incl. profitability 30, qualitative categories 40,
* red-flag penalties up to -15) and appends it to the company's running
* ledger. There is NO frontier model and NO cloud key — everything stays on
* your hardware. The only secrets are the Spark SSH key and an optional
* Hugging Face token (secrets.ts).
*/
export const configShape = z.object({
// --- Spark connection (mirrors LLaMA-Factory / Nightshift) ---
@@ -35,20 +40,21 @@ export const configShape = z.object({
// --- Serving (vLLM on the Sparks) ---
gpuMemoryUtilization: z.string().default('0.85'),
maxModelLen: z.number().int().positive().default(32768),
// vLLM tool-call parser for native function-calling (the reviewer's read-file
// vLLM tool-call parser for native function-calling (the grader's read-file
// tool loop relies on it). Must match the served model family — Qwen3 →
// 'hermes'. Empty disables native tool-calling (reviewers fall back to a
// 'hermes'. Empty disables native tool-calling (graders fall back to a
// JSON-action text protocol).
toolCallParser: z.string().default('hermes'),
// LiteLLM router exposing every model alias on one OpenAI-compatible endpoint.
proxyPort: z.number().int().positive().default(4000),
// How many distinct models may be co-resident on the HEAD Spark at once. The
// job runner loads models in WAVES so it never exceeds this — letting you run a
// panel across more models than fit in GPU memory simultaneously. 1 is safest.
// job runner loads models in WAVES so it never exceeds this — letting you run
// a panel across more models than fit in GPU memory simultaneously. 1 is
// safest.
maxConcurrentModels: z.number().int().positive().default(1),
// The MODEL CATALOG: the set of local models the service can serve. Each
// reviewer (below) references one of these by `alias`. Mirrors
// grader (below) references one of these by `alias`. Mirrors
// bm_config.py CONFIG_DEFAULTS["models"].
models: z
.array(
@@ -62,13 +68,14 @@ export const configShape = z.object({
}),
)
.default([
{ alias: 'reviewer-a', hfModel: 'Qwen/Qwen3-32B-FP8', spark: 'primary', port: 8001 },
{ alias: 'grader-a', hfModel: 'Qwen/Qwen3-32B-FP8', spark: 'primary', port: 8001 },
]),
// --- The review panel: one entry per reviewer ("number of reviews") ---
// Each reviewer is a model + a persona (the lens it reads through) + an
// optional temperature. Mirrors bm_config.py CONFIG_DEFAULTS["reviewers"].
reviewers: z
// --- The grading panel: one entry per grader ---
// Each grader is a model + a persona (the lens it grades through — e.g. a
// Munger-style inversion skeptic or a Girdley-style operator) + an optional
// temperature. Mirrors bm_config.py CONFIG_DEFAULTS["graders"].
graders: z
.array(
z.object({
name: z.string(),
@@ -79,50 +86,124 @@ export const configShape = z.object({
}),
)
.default([
{ name: 'reviewer-1', model: 'reviewer-a', persona: '', temperature: null },
{ name: 'munger-lens', model: 'grader-a', persona: '', temperature: null },
]),
// --- Review job settings ---
// The rubric: what every reviewer should look for / produce. Layered above
// each reviewer's persona.
reviewInstructions: z.string().default(
'Review the attached document(s). Produce a structured report: a 3-5 sentence ' +
'summary, the key findings and insights, risks or red flags, open questions, ' +
'and concrete recommendations. Cite the document and section for each point. ' +
'Be honest about uncertainty; never invent facts not present in the documents.',
),
// Confidentiality posture for the reviewer containers:
// 'airgapped' — reviewers join an --internal Docker network: they can
// Which catalog model runs the stage-1 structured KPI extractor over each
// deck. Empty = first model in the catalog.
extractorModel: z.string().default(''),
// --- Grading job settings ---
// The rubric override. Empty = the baked-in BDEF v1.1 framework
// (orchestrator/bdef.md — Girdley + Munger/Buffett). Non-empty text replaces
// it wholesale, so include scoring categories A-H if you customize.
bdefOverride: z.string().default(''),
// Deterministic-scorer knobs. The composite is 0-100 = quant 60 (profitability
// 30 + other KPIs 20 + forecast integrity 10) + qualitative 40 (8 BDEF
// categories x 5) - red-flag penalties (capped at 15). Mirrors
// bm_config.py WEIGHTS_DEFAULTS; keep both in sync.
weights: z
.object({
// Points for the profitability KPI attainment bucket (heaviest weight).
profitabilityKpi: z.number().default(30),
// Points for the non-profitability measurable-KPI bucket.
otherKpi: z.number().default(20),
// Points for forecast integrity: deck N actuals vs deck N-1 stated targets.
forecastIntegrity: z.number().default(10),
// Max points per qualitative BDEF category A-H (8 x 5 = 40).
qualCategoryMax: z.number().default(5),
// Cap on total red-flag penalty.
redFlagCap: z.number().default(15),
// actual/target ratio below which a KPI earns zero credit.
kpiCreditFloor: z.number().default(0.5),
// Penalty per KPI that silently disappeared from the deck.
droppedKpiPenalty: z.number().default(2),
// Count at most this many dropped-KPI flags.
droppedKpiMax: z.number().default(3),
// Quote characters required for full qualitative-evidence weight.
evidenceFullCredit: z.number().default(400),
// Damping factor for red flags raised by a single grader only.
singleSourceFlagFactor: z.number().default(0.5),
})
.default({
profitabilityKpi: 30,
otherKpi: 20,
forecastIntegrity: 10,
qualCategoryMax: 5,
redFlagCap: 15,
kpiCreditFloor: 0.5,
droppedKpiPenalty: 2,
droppedKpiMax: 3,
evidenceFullCredit: 400,
singleSourceFlagFactor: 0.5,
}),
// Confidentiality posture for the grader containers:
// 'airgapped' — graders join an --internal Docker network: they can
// reach ONLY the on-Spark model proxy, with zero internet
// egress. Models must be pre-pulled into the Spark's HF
// cache (no live download). All models must be on the head
// Spark. Strongest confidentiality.
// 'local_services' — reviewers may also reach configured LAN services
// 'local_services' — graders may also reach configured LAN services
// (e.g. SearXNG) and the second Spark. NOTE: this network
// has egress unless you firewall it — use only when you
// accept that reviewers can reach the network.
// accept that graders can reach the network.
networkMode: z.enum(['airgapped', 'local_services']).default('airgapped'),
// SearXNG JSON endpoint, used ONLY in local_services mode to give reviewers a
// SearXNG JSON endpoint, used ONLY in local_services mode to give graders a
// web_search tool. Empty = no web search.
searxngUrl: z.string().default(''),
// --- Synthesis (a local lead reviewer; no frontier model) ---
synthesisEnabled: z.boolean().default(true),
// Alias of the model that writes the consolidated report. Empty = first model.
synthesisModel: z.string().default(''),
// Optional persona/instructions for the lead reviewer. Empty = built-in default.
synthesisPersona: z.string().default(''),
// --- Adjudication (a local lead grader; no frontier model) ---
adjudicatorEnabled: z.boolean().default(true),
// Alias of the model that reconciles the panel's grades. Empty = first model.
adjudicatorModel: z.string().default(''),
// Optional persona/instructions for the adjudicator. Empty = built-in default.
adjudicatorPersona: z.string().default(''),
// --- Document handling ---
// After a job, wipe the extracted document text from the Sparks. Reports are
// After a job, wipe the extracted deck text from the Sparks. Scorecards are
// kept on the StartOS box regardless. Default true for confidentiality.
wipeRemoteDocs: z.boolean().default(true),
// Watch /data/inbox and auto-start a review when files land (debounced).
// Default false: you trigger reviews explicitly with "Run Review".
// Watch /data/inbox and auto-start grading when decks land (debounced).
// Default false: you trigger grading explicitly with "Grade Decks".
autoRunOnDrop: z.boolean().default(false),
// Name of the per-job Docker network created on the head Spark.
networkName: z.string().default('boardroom-net'),
// --- Portfolio companies ---
// The authoritative source of pinned KPI targets and KPI-name aliases. Decks
// are dropped into /data/inbox/<slug>/ and each company keeps its own running
// scorecard ledger. Mirrors bm_config.py CONFIG_DEFAULTS["companies"].
companies: z
.array(
z.object({
// Directory name under /data/inbox and the ledger key. Stable — do not
// rename once decks have been graded.
slug: z.string(),
// Display name for the dashboard. Empty = the slug.
name: z.string().default(''),
// Newline-separated "canonical=alias1;alias2" lines mapping the names a
// deck uses for a KPI onto its canonical name.
kpiAliases: z.string().default(''),
// Targets the scorer holds the company to even when a deck goes quiet
// about them. `profitability: true` marks the KPI as part of the
// heavier profitability bucket.
pinnedTargets: z
.array(
z.object({
kpi: z.string(),
target: z.number(),
unit: z.string().default(''),
direction: z.enum(['gte', 'lte']).default('gte'),
profitability: z.boolean().default(false),
}),
)
.default([]),
}),
)
.default([]),
// --- Auth flags (the secret itself lives in secrets.ts) ---
hfTokenSet: z.boolean().default(false),
})
startos/interfaces.ts
+3 -2
View File
@@ -13,8 +13,9 @@ export const setInterfaces = sdk.setupInterfaces(async ({ effects }) => {
name: 'Web UI',
id: 'webui',
description:
'The Boardroom Map control panel: drop documents in, convene the reviewer ' +
'panel, watch the job run on your Sparks, and read the reports.',
'The Boardroom Map control panel: drop board decks into each company\'s ' +
'inbox, run the grading panel on your Sparks, and watch per-company ' +
'scorecard trends.',
type: 'ui',
username: null,
path: '',
startos/main.ts
+6 -5
View File
@@ -3,8 +3,8 @@ import { WEB_UI_PORT } from './interfaces'
export const main = sdk.setupMain(async ({ effects }) => {
// Mount the persistent volume at /data: config.json, ssh key, optional HF
// token, the dropped-document inbox, job run state, and saved reports all live
// here.
// token, the per-company deck inbox, job run state, and saved scorecards /
// ledgers all live here.
const mounts = sdk.Mounts.of().mountVolume({
volumeId: 'main',
mountpoint: '/data',
@@ -19,9 +19,10 @@ export const main = sdk.setupMain(async ({ effects }) => {
'boardroom-webui',
)
// The web UI runs the FastAPI app AND, in a background thread, the Boardroom Map job
// runner (which extracts dropped documents, serves the chosen models on the
// Sparks in waves, runs the reviewer panel, and synthesizes a report).
// The web UI runs the FastAPI app AND, in a background thread, the Boardroom
// Map job runner (which extracts dropped decks, serves the chosen models on
// the Sparks in waves, runs the grading panel + adjudicator, computes the
// deterministic composite, and updates each company's scorecard ledger).
return sdk.Daemons.of(effects).addDaemon('webui', {
subcontainer: sub,
exec: {
startos/manifest/index.ts
+33 -24
View File
@@ -7,16 +7,19 @@ import { setupManifest } from '@start9labs/start-sdk'
* does not run any GPU workload itself. It is a small web UI + job runner that
* SSHes into one or two NVIDIA DGX Sparks to:
* 1. serve a panel of local LLMs with vLLM (loaded in waves to fit GPU memory),
* 2. extract text from documents you drop in (PDF/DOCX/TXT/MD — done on the
* StartOS box), ship it to the Sparks, and launch a panel of sandboxed
* "reviewer" containers (each a model + a persona) that read the documents
* and write a report,
* 3. optionally run a local "lead reviewer" that synthesizes the panel's
* reports into one consolidated report.
* 2. extract text from the board decks you drop into inbox/<company-slug>/
* (PDF/DOCX/TXT/MD — done on the StartOS box), ship it to the Sparks, and
* launch a panel of sandboxed "grader" containers (each a model + a
* persona) that grade each deck against the BDEF v1.1 framework
* (Girdley + Munger/Buffett),
* 3. optionally run a local "adjudicator" that reconciles the panel, after
* which Python computes a deterministic composite (quant KPI attainment 60
* incl. profitability 30, qualitative categories 40, red-flag penalties
* up to -15) and appends it to the company's running scorecard ledger.
*
* There is NO frontier model and NO cloud API key. In the default `airgapped`
* network mode the reviewer containers can reach ONLY the on-Spark model proxy —
* the documents and their reviews never touch the internet.
* network mode the grader containers can reach ONLY the on-Spark model proxy —
* the decks and their grades never touch the internet.
*
* NOTE: s9pk.mk extracts the package identifier from the single-quoted value on
* the line below, so keep that field on one line and avoid stray quotes above it.
@@ -30,17 +33,21 @@ export const manifest = setupManifest({
marketingUrl: 'https://github.com/ten31/boardroom-map',
donationUrl: null,
description: {
short: 'A private panel of local LLMs that reviews your confidential documents on your DGX Sparks',
short: 'Grade portfolio-company board decks with local LLMs on your DGX Sparks — BDEF scoring, per-company running scorecards',
long:
'Boardroom Map lets you drop confidential documents in and convene a panel of ' +
'local LLMs running on your NVIDIA DGX Sparks to review them. You choose ' +
'which models and which personas (lenses) sit on the panel and how many ' +
'reviews to run. Each reviewer reads the documents and writes a report; an ' +
'optional local lead reviewer synthesizes them into one consolidated ' +
'report. There is no frontier model and no cloud key: in the default ' +
'air-gapped mode the reviewers reach only the on-Spark model endpoint, so ' +
'your documents and their reviews never leave your hardware. No GPU is ' +
'needed on the StartOS host.',
'Boardroom Map turns your DGX Sparks into a private board-deck grading ' +
'panel. Drop each portfolio company\'s deck into its inbox folder and a ' +
'panel of local LLMs (each a model + a persona) grades it against the ' +
'BDEF v1.1 framework (Girdley + Munger/Buffett); an optional local ' +
'adjudicator reconciles the panel, then a deterministic scorer computes a ' +
'0-100 composite — quantitative KPI attainment worth 60 (profitability ' +
'alone 30, plus forecast integrity: deck N actuals vs deck N-1 promises), ' +
'qualitative categories worth 40, and red-flag penalties up to -15. Each ' +
'company keeps a running scorecard ledger, and a web dashboard shows the ' +
'trends. There is no frontier model and no cloud key: in the default ' +
'air-gapped mode the graders reach only the on-Spark model endpoint, so ' +
'your confidential decks never leave your hardware. No GPU is needed on ' +
'the StartOS host.',
},
// Arch-agnostic orchestrator. Docker build paths are relative to the PROJECT
// ROOT (where the Makefile runs), matching the Start9 convention.
@@ -53,7 +60,7 @@ export const manifest = setupManifest({
},
},
arch: ['x86_64', 'aarch64'],
// The orchestrator only SSHes out + extracts document text on CPU; it never
// The orchestrator only SSHes out + extracts deck text on CPU; it never
// touches a local GPU.
nvidiaContainer: false,
},
@@ -66,10 +73,12 @@ export const manifest = setupManifest({
alerts: {
install:
'Boardroom Map drives work on REMOTE machines (your DGX Sparks) over SSH; ' +
'nothing serves or runs on your StartOS server. After install: ' +
'(1) "Configure Sparks" for SSH access, (2) "Configure Models" for the ' +
'local models to serve, (3) "Configure Reviewers" for the panel + personas, ' +
'(4) "Configure Review" for the rubric and air-gap mode. Then drop ' +
'documents in the inbox and run "Run Review".',
'nothing serves or runs on your StartOS server, and your confidential ' +
'board decks stay on your LAN. After install: (1) "Configure Sparks" for ' +
'SSH access, (2) "Configure Models" for the local models to serve, ' +
'(3) "Configure Graders" for the panel + personas, (4) "Configure Grading" ' +
'for the BDEF rubric, air-gap mode, and scoring weights, (5) "Configure ' +
'Companies" for slugs and pinned KPI targets. Then drop decks into ' +
'inbox/<company-slug>/ and run "Grade Decks".',
},
})
startos/versions/v_0_1_0.ts
+7 -5
View File
@@ -7,10 +7,12 @@ import { VersionInfo } from '@start9labs/start-sdk'
export const v_0_1_0 = VersionInfo.of({
version: '0.1.0:0',
releaseNotes:
'Initial release: drop confidential documents in and convene a panel of ' +
'local LLMs on your DGX Sparks to review them. Choose the models, the ' +
'personas, and how many reviews; an optional local lead reviewer ' +
'synthesizes a consolidated report. Default air-gapped mode keeps documents ' +
'and reviews entirely on your hardware.',
'Initial version — BDEF v1.1 deck grading with per-company scorecards. ' +
'Drop board decks into inbox/<company-slug>/ and a panel of local LLMs on ' +
'your DGX Sparks grades them against the BDEF framework (Girdley + ' +
'Munger/Buffett); a deterministic scorer computes the 0-100 composite ' +
'(quant 60 incl. profitability 30, qualitative 40, red flags to -15) and ' +
'appends it to each company\'s running ledger. Default air-gapped mode ' +
'keeps decks and grades entirely on your hardware.',
migrations: {},
})