Implement BDEF v1.1 grading: scoring core, per-deck pipeline, ledger, dashboard, StartOS layer
- Deterministic scoring.py (quant 60 / qual 40 / flags -15, profitability heaviest) - Per-company JSON ledger with forecast-target chaining deck N-1 -> N - Single-shot sandbox agent with guided-JSON fallback ladder (no tool loop) - Portfolio dashboard with sparklines, KPI hit rates, BDEF category bars - 48 unit tests green; endpoints smoke-tested; npm check+build green Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jonathan Kirkwood
co-authored by
Claude Fable 5
Claude Fable 5
parent
1dde915540
commit
b1d7aed9f4
+124
-43
@@ -3,16 +3,21 @@ import { FileHelper, z } from '@start9labs/start-sdk'
|
||||
/**
|
||||
* Boardroom Map configuration, persisted to the `main` volume as config.json.
|
||||
*
|
||||
* Written by the StartOS actions (Configure Sparks / Models / Reviewers /
|
||||
* Review) and read by the Python orchestrator inside the container, which mounts
|
||||
* the same volume at /data and reads /data/config.json. Keep field names in sync
|
||||
* with orchestrator/bm_config.py (CONFIG_DEFAULTS).
|
||||
* Written by the StartOS actions (Configure Sparks / Models / Graders /
|
||||
* Grading / Companies) and read by the Python orchestrator inside the
|
||||
* container, which mounts the same volume at /data and reads /data/config.json.
|
||||
* Keep field names in sync with orchestrator/bm_config.py (CONFIG_DEFAULTS).
|
||||
*
|
||||
* Boardroom Map is a CONTROL PLANE: a GPU-free orchestrator on StartOS that SSHes into
|
||||
* one or two DGX Sparks to serve local models and run a panel of sandboxed
|
||||
* "reviewer" containers over confidential documents you drop in. There is NO
|
||||
* frontier model and NO cloud key — everything stays on your hardware. The only
|
||||
* secrets are the Spark SSH key and an optional Hugging Face token (secrets.ts).
|
||||
* Boardroom Map is a CONTROL PLANE: a GPU-free orchestrator on StartOS that
|
||||
* SSHes into one or two DGX Sparks to serve local models and grade the board
|
||||
* decks you drop into /data/inbox/<company-slug>/. A panel of sandboxed
|
||||
* "grader" containers scores each deck against the BDEF v1.1 framework
|
||||
* (Girdley + Munger/Buffett); Python then computes a deterministic composite
|
||||
* (quant KPI attainment 60 incl. profitability 30, qualitative categories 40,
|
||||
* red-flag penalties up to -15) and appends it to the company's running
|
||||
* ledger. There is NO frontier model and NO cloud key — everything stays on
|
||||
* your hardware. The only secrets are the Spark SSH key and an optional
|
||||
* Hugging Face token (secrets.ts).
|
||||
*/
|
||||
export const configShape = z.object({
|
||||
// --- Spark connection (mirrors LLaMA-Factory / Nightshift) ---
|
||||
@@ -35,20 +40,21 @@ export const configShape = z.object({
|
||||
// --- Serving (vLLM on the Sparks) ---
|
||||
gpuMemoryUtilization: z.string().default('0.85'),
|
||||
maxModelLen: z.number().int().positive().default(32768),
|
||||
// vLLM tool-call parser for native function-calling (the reviewer's read-file
|
||||
// vLLM tool-call parser for native function-calling (the grader's read-file
|
||||
// tool loop relies on it). Must match the served model family — Qwen3 →
|
||||
// 'hermes'. Empty disables native tool-calling (reviewers fall back to a
|
||||
// 'hermes'. Empty disables native tool-calling (graders fall back to a
|
||||
// JSON-action text protocol).
|
||||
toolCallParser: z.string().default('hermes'),
|
||||
// LiteLLM router exposing every model alias on one OpenAI-compatible endpoint.
|
||||
proxyPort: z.number().int().positive().default(4000),
|
||||
// How many distinct models may be co-resident on the HEAD Spark at once. The
|
||||
// job runner loads models in WAVES so it never exceeds this — letting you run a
|
||||
// panel across more models than fit in GPU memory simultaneously. 1 is safest.
|
||||
// job runner loads models in WAVES so it never exceeds this — letting you run
|
||||
// a panel across more models than fit in GPU memory simultaneously. 1 is
|
||||
// safest.
|
||||
maxConcurrentModels: z.number().int().positive().default(1),
|
||||
|
||||
// The MODEL CATALOG: the set of local models the service can serve. Each
|
||||
// reviewer (below) references one of these by `alias`. Mirrors
|
||||
// grader (below) references one of these by `alias`. Mirrors
|
||||
// bm_config.py CONFIG_DEFAULTS["models"].
|
||||
models: z
|
||||
.array(
|
||||
@@ -62,13 +68,14 @@ export const configShape = z.object({
|
||||
}),
|
||||
)
|
||||
.default([
|
||||
{ alias: 'reviewer-a', hfModel: 'Qwen/Qwen3-32B-FP8', spark: 'primary', port: 8001 },
|
||||
{ alias: 'grader-a', hfModel: 'Qwen/Qwen3-32B-FP8', spark: 'primary', port: 8001 },
|
||||
]),
|
||||
|
||||
// --- The review panel: one entry per reviewer ("number of reviews") ---
|
||||
// Each reviewer is a model + a persona (the lens it reads through) + an
|
||||
// optional temperature. Mirrors bm_config.py CONFIG_DEFAULTS["reviewers"].
|
||||
reviewers: z
|
||||
// --- The grading panel: one entry per grader ---
|
||||
// Each grader is a model + a persona (the lens it grades through — e.g. a
|
||||
// Munger-style inversion skeptic or a Girdley-style operator) + an optional
|
||||
// temperature. Mirrors bm_config.py CONFIG_DEFAULTS["graders"].
|
||||
graders: z
|
||||
.array(
|
||||
z.object({
|
||||
name: z.string(),
|
||||
@@ -79,50 +86,124 @@ export const configShape = z.object({
|
||||
}),
|
||||
)
|
||||
.default([
|
||||
{ name: 'reviewer-1', model: 'reviewer-a', persona: '', temperature: null },
|
||||
{ name: 'munger-lens', model: 'grader-a', persona: '', temperature: null },
|
||||
]),
|
||||
|
||||
// --- Review job settings ---
|
||||
// The rubric: what every reviewer should look for / produce. Layered above
|
||||
// each reviewer's persona.
|
||||
reviewInstructions: z.string().default(
|
||||
'Review the attached document(s). Produce a structured report: a 3-5 sentence ' +
|
||||
'summary, the key findings and insights, risks or red flags, open questions, ' +
|
||||
'and concrete recommendations. Cite the document and section for each point. ' +
|
||||
'Be honest about uncertainty; never invent facts not present in the documents.',
|
||||
),
|
||||
// Confidentiality posture for the reviewer containers:
|
||||
// 'airgapped' — reviewers join an --internal Docker network: they can
|
||||
// Which catalog model runs the stage-1 structured KPI extractor over each
|
||||
// deck. Empty = first model in the catalog.
|
||||
extractorModel: z.string().default(''),
|
||||
|
||||
// --- Grading job settings ---
|
||||
// The rubric override. Empty = the baked-in BDEF v1.1 framework
|
||||
// (orchestrator/bdef.md — Girdley + Munger/Buffett). Non-empty text replaces
|
||||
// it wholesale, so include scoring categories A-H if you customize.
|
||||
bdefOverride: z.string().default(''),
|
||||
|
||||
// Deterministic-scorer knobs. The composite is 0-100 = quant 60 (profitability
|
||||
// 30 + other KPIs 20 + forecast integrity 10) + qualitative 40 (8 BDEF
|
||||
// categories x 5) - red-flag penalties (capped at 15). Mirrors
|
||||
// bm_config.py WEIGHTS_DEFAULTS; keep both in sync.
|
||||
weights: z
|
||||
.object({
|
||||
// Points for the profitability KPI attainment bucket (heaviest weight).
|
||||
profitabilityKpi: z.number().default(30),
|
||||
// Points for the non-profitability measurable-KPI bucket.
|
||||
otherKpi: z.number().default(20),
|
||||
// Points for forecast integrity: deck N actuals vs deck N-1 stated targets.
|
||||
forecastIntegrity: z.number().default(10),
|
||||
// Max points per qualitative BDEF category A-H (8 x 5 = 40).
|
||||
qualCategoryMax: z.number().default(5),
|
||||
// Cap on total red-flag penalty.
|
||||
redFlagCap: z.number().default(15),
|
||||
// actual/target ratio below which a KPI earns zero credit.
|
||||
kpiCreditFloor: z.number().default(0.5),
|
||||
// Penalty per KPI that silently disappeared from the deck.
|
||||
droppedKpiPenalty: z.number().default(2),
|
||||
// Count at most this many dropped-KPI flags.
|
||||
droppedKpiMax: z.number().default(3),
|
||||
// Quote characters required for full qualitative-evidence weight.
|
||||
evidenceFullCredit: z.number().default(400),
|
||||
// Damping factor for red flags raised by a single grader only.
|
||||
singleSourceFlagFactor: z.number().default(0.5),
|
||||
})
|
||||
.default({
|
||||
profitabilityKpi: 30,
|
||||
otherKpi: 20,
|
||||
forecastIntegrity: 10,
|
||||
qualCategoryMax: 5,
|
||||
redFlagCap: 15,
|
||||
kpiCreditFloor: 0.5,
|
||||
droppedKpiPenalty: 2,
|
||||
droppedKpiMax: 3,
|
||||
evidenceFullCredit: 400,
|
||||
singleSourceFlagFactor: 0.5,
|
||||
}),
|
||||
|
||||
// Confidentiality posture for the grader containers:
|
||||
// 'airgapped' — graders join an --internal Docker network: they can
|
||||
// reach ONLY the on-Spark model proxy, with zero internet
|
||||
// egress. Models must be pre-pulled into the Spark's HF
|
||||
// cache (no live download). All models must be on the head
|
||||
// Spark. Strongest confidentiality.
|
||||
// 'local_services' — reviewers may also reach configured LAN services
|
||||
// 'local_services' — graders may also reach configured LAN services
|
||||
// (e.g. SearXNG) and the second Spark. NOTE: this network
|
||||
// has egress unless you firewall it — use only when you
|
||||
// accept that reviewers can reach the network.
|
||||
// accept that graders can reach the network.
|
||||
networkMode: z.enum(['airgapped', 'local_services']).default('airgapped'),
|
||||
// SearXNG JSON endpoint, used ONLY in local_services mode to give reviewers a
|
||||
// SearXNG JSON endpoint, used ONLY in local_services mode to give graders a
|
||||
// web_search tool. Empty = no web search.
|
||||
searxngUrl: z.string().default(''),
|
||||
|
||||
// --- Synthesis (a local lead reviewer; no frontier model) ---
|
||||
synthesisEnabled: z.boolean().default(true),
|
||||
// Alias of the model that writes the consolidated report. Empty = first model.
|
||||
synthesisModel: z.string().default(''),
|
||||
// Optional persona/instructions for the lead reviewer. Empty = built-in default.
|
||||
synthesisPersona: z.string().default(''),
|
||||
// --- Adjudication (a local lead grader; no frontier model) ---
|
||||
adjudicatorEnabled: z.boolean().default(true),
|
||||
// Alias of the model that reconciles the panel's grades. Empty = first model.
|
||||
adjudicatorModel: z.string().default(''),
|
||||
// Optional persona/instructions for the adjudicator. Empty = built-in default.
|
||||
adjudicatorPersona: z.string().default(''),
|
||||
|
||||
// --- Document handling ---
|
||||
// After a job, wipe the extracted document text from the Sparks. Reports are
|
||||
// After a job, wipe the extracted deck text from the Sparks. Scorecards are
|
||||
// kept on the StartOS box regardless. Default true for confidentiality.
|
||||
wipeRemoteDocs: z.boolean().default(true),
|
||||
// Watch /data/inbox and auto-start a review when files land (debounced).
|
||||
// Default false: you trigger reviews explicitly with "Run Review".
|
||||
// Watch /data/inbox and auto-start grading when decks land (debounced).
|
||||
// Default false: you trigger grading explicitly with "Grade Decks".
|
||||
autoRunOnDrop: z.boolean().default(false),
|
||||
// Name of the per-job Docker network created on the head Spark.
|
||||
networkName: z.string().default('boardroom-net'),
|
||||
|
||||
// --- Portfolio companies ---
|
||||
// The authoritative source of pinned KPI targets and KPI-name aliases. Decks
|
||||
// are dropped into /data/inbox/<slug>/ and each company keeps its own running
|
||||
// scorecard ledger. Mirrors bm_config.py CONFIG_DEFAULTS["companies"].
|
||||
companies: z
|
||||
.array(
|
||||
z.object({
|
||||
// Directory name under /data/inbox and the ledger key. Stable — do not
|
||||
// rename once decks have been graded.
|
||||
slug: z.string(),
|
||||
// Display name for the dashboard. Empty = the slug.
|
||||
name: z.string().default(''),
|
||||
// Newline-separated "canonical=alias1;alias2" lines mapping the names a
|
||||
// deck uses for a KPI onto its canonical name.
|
||||
kpiAliases: z.string().default(''),
|
||||
// Targets the scorer holds the company to even when a deck goes quiet
|
||||
// about them. `profitability: true` marks the KPI as part of the
|
||||
// heavier profitability bucket.
|
||||
pinnedTargets: z
|
||||
.array(
|
||||
z.object({
|
||||
kpi: z.string(),
|
||||
target: z.number(),
|
||||
unit: z.string().default(''),
|
||||
direction: z.enum(['gte', 'lte']).default('gte'),
|
||||
profitability: z.boolean().default(false),
|
||||
}),
|
||||
)
|
||||
.default([]),
|
||||
}),
|
||||
)
|
||||
.default([]),
|
||||
|
||||
// --- Auth flags (the secret itself lives in secrets.ts) ---
|
||||
hfTokenSet: z.boolean().default(false),
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user