Jonathan KirkwoodandClaude Fable 5
6bf0786993
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
142 lines
7.8 KiB
Markdown
142 lines
7.8 KiB
Markdown
# Boardroom Map — private board-deck grading on your DGX Sparks
|
||
|
||
Boardroom Map is a StartOS service (`.s9pk`) that **grades portfolio-company
|
||
board decks with a panel of local LLMs** on your NVIDIA DGX Sparks. Drop each
|
||
company's deck into `inbox/<company-slug>/`; the panel grades it against the
|
||
**BDEF v1.1 framework** (Girdley + Munger/Buffett), an optional local
|
||
**adjudicator** reconciles the panel, and a deterministic Python scorer computes
|
||
a 0–100 composite that lands on the company's **running scorecard ledger**. A
|
||
web dashboard shows per-company trends. There is **no frontier model and no
|
||
cloud API key** — in the default air-gapped mode the decks and their grades
|
||
never leave your hardware.
|
||
|
||
It is a sibling of [Chambers](../chambers) and reuses the same control-plane
|
||
pattern (a GPU-free orchestrator on StartOS driving the Sparks over SSH), but
|
||
swaps the free-form document-review panel for a **deterministic deck-grading
|
||
pipeline** with pinned KPI targets and per-company ledgers.
|
||
|
||
## The scoring model
|
||
|
||
`composite (0–100) = quant 60 + qual 40 − red flags (capped at 15)`
|
||
|
||
- **Quantitative 60:** profitability KPI attainment **30** (heaviest slice),
|
||
other measurable KPIs **20**, **forecast integrity 10** — deck N's actuals are
|
||
chained against deck N−1's stated targets, so moved goalposts cost points.
|
||
KPI credit is linear above a floor ratio (default 0.5 → zero credit below).
|
||
- **Qualitative 40:** eight BDEF categories (A–H) × 5 points, scored by the
|
||
panel with evidence quotes; thin evidence scales down.
|
||
- **Red flags:** up to **−15**; silently dropped KPIs are auto-flagged (capped),
|
||
and flags raised by a single grader are damped by 0.5.
|
||
|
||
Every knob lives in config (`weights`, per-company `pinnedTargets` and
|
||
`kpiAliases`) so the model can be retuned without a rebuild.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
StartOS box (control plane, no GPU) DGX Spark(s)
|
||
┌────────────────────────────────────┐ ┌───────────────────────────────┐
|
||
│ FastAPI dashboard + job runner │ SSH │ per-job Docker network │
|
||
│ • inbox/<company-slug>/ (decks) │ ───────▶│ (──internal in airgapped) │
|
||
│ • extract text (PDF/PPTX/DOCX/…) │ rsync │ ┌─────────┐ ┌────────────┐ │
|
||
│ • plan model "waves" │ ───────▶│ │ vLLM(s) │◀─│ LiteLLM │ │
|
||
│ • extractor → graders → adjudicator│ │ └─────────┘ │ router │ │
|
||
│ • deterministic composite scorer │◀─────── │ ┌──────────────┐ ▲ │ │
|
||
│ • per-company ledgers + scorecards │ rsync │ │ grader ×N │──┘ │ │
|
||
└────────────────────────────────────┘ │ │ (read-only, │ │ │
|
||
│ │ sandboxed) │ │ │
|
||
│ └──────────────┘ │ │
|
||
└───────────────────────────────┘
|
||
```
|
||
|
||
- **Graders** are one-shot, read-only, hardened containers (non-root,
|
||
`--cap-drop ALL`, read-only rootfs, no docker socket). In air-gapped mode they
|
||
sit on an `--internal` network and can reach only the model proxy.
|
||
- **Waves:** the job runner serves models in waves bounded by
|
||
`maxConcurrentModels`, so a panel can span more models than fit in GPU memory
|
||
at once.
|
||
- **Air-gap modes:** `airgapped` (default — graders reach only the on-Spark
|
||
model proxy, zero egress, models pre-pulled) or `local_services` (graders may
|
||
reach LAN services like SearXNG and the second Spark — has egress unless
|
||
firewalled).
|
||
- **Confidentiality:** decks are extracted to text on the StartOS box; only
|
||
text crosses to the Sparks, and it is wiped from the Sparks after the job.
|
||
Scorecards and ledgers live only on the StartOS box.
|
||
|
||
## Setup order
|
||
|
||
Configure Sparks → Test Spark Connection → Configure Models → Configure Graders
|
||
→ Configure Grading (rubric, air-gap, weights) → Configure Companies (slugs,
|
||
KPI aliases, pinned targets — especially profitability thresholds) → drop decks
|
||
into `inbox/<company-slug>/` → **Grade Decks** → watch the dashboard.
|
||
|
||
Two operational notes:
|
||
|
||
- **Deck filenames must contain the period with the year** — e.g.
|
||
`Board Deck - Q4 2025.pdf` parses; a bare `Q4` does not. Periods are
|
||
canonicalized (`2025-Q4`, `2026-H1`, `2026-05`, `FY2026`) and drive the
|
||
forecast-target chaining between consecutive decks.
|
||
- If a Spark already runs resident vLLM containers, list their names in
|
||
**Configure Grading → `preJobStopContainers`** — they are docker-stopped on
|
||
the head Spark at job start to free GPU memory (and deliberately *not*
|
||
restarted afterward; re-warm them from whatever job owns them).
|
||
|
||
## Repo layout
|
||
|
||
```
|
||
startos/ StartOS package definition (TypeScript / start-sdk)
|
||
manifest/ main.ts interfaces.ts versions/ file-models/ actions/
|
||
orchestrator/ The control-plane app (Python)
|
||
app.py FastAPI dashboard + JSON API
|
||
jobs.py the job runner (extract → serve waves → grade → adjudicate → score)
|
||
serving.py vLLM + LiteLLM router on the Sparks, in waves
|
||
graders.py launch the grading panel
|
||
adjudicator.py the local lead grader
|
||
extraction.py PDF/PPTX/DOCX/TXT/MD → text (on the StartOS box)
|
||
preflight.py probe models before launching graders
|
||
spark_client.py SSH/rsync helpers
|
||
bdef.md the baked-in BDEF v1.1 rubric
|
||
bm_config.py config defaults (mirrors startos/file-models/config.ts)
|
||
sandbox/ grader image (built ON the Spark, not packed in the s9pk)
|
||
grader_agent.py grader.Dockerfile build.sh
|
||
```
|
||
|
||
## Build
|
||
|
||
Local build with `start-cli` (on a Mac: colima + start-cli in a VM — see the
|
||
s9pk-build-on-mac recipe). The vLLM and grader images are built **on the
|
||
Sparks**, not packed into the `.s9pk`:
|
||
|
||
- `boardroom-vllm:latest` — the vLLM serving image with `ENTRYPOINT []`
|
||
(serving.py passes the full `vllm serve …` command as the container CMD).
|
||
- `boardroom-grader:latest` — from `sandbox/build.sh`.
|
||
|
||
```
|
||
npm ci && npm run check && npm run build # type-check + bundle
|
||
make # pack the .s9pk (needs start-cli)
|
||
```
|
||
|
||
Canonical repo: `https://gitea.ten31.ai/Ten31AI/boardroom-map`.
|
||
|
||
## Status
|
||
|
||
**v0.1.5 — live in production.** Deployed on a StartOS box driving a DGX Spark
|
||
in single-spark air-gapped mode (gemma-4-31B panel: munger-lens /
|
||
girdley-operator / buffett-owner). First full grading run completed
|
||
2026-07-29: a three-deck company history graded end-to-end into a running
|
||
ledger with quarter-over-quarter forecast chaining.
|
||
|
||
Version history: v0.1.1 fixed config persistence (absolute `/media/startos/…`
|
||
paths — relative paths resolved into an ephemeral cwd); v0.1.2 added
|
||
`preJobStopContainers`; v0.1.3 hardened preflight (authed model probes,
|
||
poll-until-loaded, crash fast-fail); v0.1.4 fixed air-gapped serving
|
||
(`HF_HUB_OFFLINE` — the `--internal` network has no DNS) and raised grader
|
||
timeouts for ~3.6 tok/s local generation; v0.1.5 fixed the dashboard viewer
|
||
(stays open across background refresh) and added report/JSON/scorecard
|
||
downloads.
|
||
|
||
Known optimization not yet done: the wave is torn down per deck, so the 31B
|
||
reloads from disk (~6 min) between decks even when the model set is unchanged.
|
||
|
||
HF model pre-pull on the head Spark is required for air-gapped runs.
|