Implement BDEF v1.1 grading: scoring core, per-deck pipeline, ledger, dashboard, StartOS layer
- Deterministic scoring.py (quant 60 / qual 40 / flags -15, profitability heaviest) - Per-company JSON ledger with forecast-target chaining deck N-1 -> N - Single-shot sandbox agent with guided-JSON fallback ladder (no tool loop) - Portfolio dashboard with sparklines, KPI hit rates, BDEF category bars - 48 unit tests green; endpoints smoke-tested; npm check+build green Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jonathan Kirkwood
co-authored by
Claude Fable 5
Claude Fable 5
parent
1dde915540
commit
b1d7aed9f4
+3
-3
@@ -1,17 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the Boardroom Map reviewer image ON THE HEAD SPARK (aarch64/GB10).
|
||||
# Build the Boardroom Map grader image ON THE HEAD SPARK (aarch64/GB10).
|
||||
# The orchestrator does this automatically over SSH, but you can also run it by
|
||||
# hand: copy this sandbox/ directory to the Spark and run:
|
||||
#
|
||||
# IMAGE=boardroom-grader:latest bash build.sh
|
||||
#
|
||||
# The tag must match the service's "Reviewer Image Tag" (Configure Sparks).
|
||||
# The tag must match the service's "Grader Image Tag" (Configure Sparks).
|
||||
set -euo pipefail
|
||||
|
||||
IMAGE="${IMAGE:-boardroom-grader:latest}"
|
||||
DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
|
||||
echo ">> Building $IMAGE from $DIR"
|
||||
docker build -t "$IMAGE" -f "$DIR/reviewer.Dockerfile" "$DIR"
|
||||
docker build -t "$IMAGE" -f "$DIR/grader.Dockerfile" "$DIR"
|
||||
echo ">> Done. Image: $IMAGE"
|
||||
docker image inspect "$IMAGE" >/dev/null && echo ">> OK"
|
||||
|
||||
@@ -1,15 +1,17 @@
|
||||
# Boardroom Map reviewer image — BUILT ON THE HEAD SPARK (aarch64), not packed into the
|
||||
# Boardroom Map grader image — BUILT ON THE HEAD SPARK (aarch64), not packed into the
|
||||
# s9pk (the orchestrator ships this build context and builds it on the Spark; see
|
||||
# reviewers.ensure_reviewer_image).
|
||||
# graders.ensure_grader_image).
|
||||
#
|
||||
# One-shot, read-only document reviewer (grader_agent.py) speaking the
|
||||
# OpenAI-compatible API directly — a lean pure-Python image that builds fast.
|
||||
# One-shot, read-only role agent (grader_agent.py; BM_ROLE = extractor | grader |
|
||||
# adjudicator) speaking the OpenAI-compatible API directly — a lean pure-Python
|
||||
# image that builds fast.
|
||||
#
|
||||
# At RUN time the orchestrator launches this HARDENED (non-root, --cap-drop ALL,
|
||||
# --security-opt no-new-privileges, read-only rootfs, no docker socket, only
|
||||
# /docs (ro), /persona (ro), /RUBRIC.md (ro) and /out (rw) mounted, cpu/mem/pid
|
||||
# caps) and attached to the per-job network. In air-gapped mode that network is
|
||||
# --internal, so the container can reach ONLY the on-Spark model proxy.
|
||||
# /docs (ro), /BDEF.md (ro), /schema.json (ro), /persona (ro) and /out (rw)
|
||||
# mounted — the adjudicator instead gets /grades (ro) + /extraction.json (ro) —
|
||||
# cpu/mem/pid caps) and attached to the per-job network. In air-gapped mode that
|
||||
# network is --internal, so the container can reach ONLY the on-Spark model proxy.
|
||||
FROM python:3.11-slim
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
|
||||
+371
-262
@@ -1,43 +1,56 @@
|
||||
"""Boardroom Map reviewer — a sandboxed, ONE-SHOT agent that reads confidential
|
||||
documents through a LOCAL model and writes a single report, then exits.
|
||||
"""Boardroom Map role agent — a sandboxed, ONE-SHOT, single-completion agent.
|
||||
|
||||
Unlike a swarm worker, this never loops forever and never writes to a shared
|
||||
workspace. It mounts the documents read-only at /docs, runs one model (through the
|
||||
on-Spark proxy) under its PERSONA + the shared RUBRIC, and writes exactly one
|
||||
file to /out:
|
||||
role=reviewer -> /out/<BM_REVIEWER_ID>.md
|
||||
role=synthesizer -> /out/CONSOLIDATED_REPORT.md (also reads /reports)
|
||||
One container = one role = one model call (plus a bounded reliability ladder /
|
||||
repair round-trip). No tools, no loops, no shared writable workspace. The
|
||||
orchestrator (orchestrator/graders.py + adjudicator.py) launches this hardened
|
||||
(non-root, read-only rootfs, per-job network) with BM_ROLE set to:
|
||||
|
||||
It speaks the OpenAI-compatible /v1/chat/completions API directly (no Claude CLI,
|
||||
no Anthropic translation — small local models handle this far better). The whole
|
||||
document set is pre-loaded into the prompt up to a budget; for anything larger the
|
||||
model can pull more with read_file. Native tool_calls are used when available,
|
||||
with a JSON-action text fallback for models without a vLLM tool parser.
|
||||
extractor reads /docs (ro) + the BDEF red-flag taxonomy, emits STRUCTURED
|
||||
JSON per /schema.json (extraction schema) -> /out/extraction.json
|
||||
grader reads /docs (ro) + the full /BDEF.md rubric, scores categories
|
||||
A-H per /schema.json (grades schema) -> /out/<BM_GRADER_ID>.json
|
||||
adjudicator reads /extraction.json (ro) + the panel's /grades/*.json (ro),
|
||||
writes a MARKDOWN adjudication (no scores) -> /out/ADJUDICATION.md
|
||||
|
||||
In air-gapped mode the container is on an --internal Docker network: the only
|
||||
thing reachable is the model proxy. web_search is offered ONLY when BM_SEARXNG_URL
|
||||
is set (local-services mode).
|
||||
Env (set by the orchestrator):
|
||||
BM_ROLE, BM_GRADER_ID, BM_GRADER_NAME, BM_MODEL,
|
||||
BM_LLM_BASE (http://boardroom-proxy:4000/v1), BM_LLM_KEY,
|
||||
BM_TEMPERATURE (extractor forced to 0.0), BM_MAX_MODEL_LEN
|
||||
|
||||
Mounts: /docs (ro), /BDEF.md (ro), /schema.json (ro; role-appropriate),
|
||||
/persona/PERSONA.md (ro, optional), /out (rw); adjudicator additionally
|
||||
/grades (ro) and /extraction.json (ro).
|
||||
|
||||
JSON reliability ladder (extractor + grader):
|
||||
1. response_format = {"type":"json_schema", ..., "strict": true}
|
||||
2. on HTTP 4xx: retry with top-level {"guided_json": <schema>} (vLLM ext.)
|
||||
3. on another 4xx: retry plain
|
||||
Parse whole-reply JSON, else the first brace-balanced {...} block. If invalid,
|
||||
ONE repair round-trip; if still bad, write the raw text to the output path plus
|
||||
a sibling <output>.invalid marker containing the error. Full jsonschema
|
||||
validation runs orchestrator-side; only lightweight structural checks here.
|
||||
|
||||
Air-gapped: the per-job Docker network is --internal, so the only reachable
|
||||
endpoint is the model proxy. Pure Python stdlib (urllib) — no pip deps.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import ssl
|
||||
import time
|
||||
import traceback
|
||||
import urllib.parse
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
_SSL_CTX = ssl._create_unverified_context() # LAN self-signed (SearXNG)
|
||||
_SSL_CTX = ssl._create_unverified_context() # LAN self-signed certs
|
||||
|
||||
RID = os.environ.get("BM_REVIEWER_ID", "reviewer")
|
||||
NAME = os.environ.get("BM_REVIEWER_NAME", RID)
|
||||
ROLE = os.environ.get("BM_ROLE", "reviewer") # reviewer | synthesizer
|
||||
MODEL = os.environ.get("BM_MODEL", "reviewer-a")
|
||||
ROLE = os.environ.get("BM_ROLE", "grader") # extractor | grader | adjudicator
|
||||
GID = os.environ.get("BM_GRADER_ID", ROLE)
|
||||
NAME = os.environ.get("BM_GRADER_NAME", GID)
|
||||
MODEL = os.environ.get("BM_MODEL", "grader-a")
|
||||
LLM_BASE = os.environ.get("BM_LLM_BASE", "http://boardroom-proxy:4000/v1").rstrip("/")
|
||||
LLM_KEY = os.environ.get("BM_LLM_KEY", "sk-local")
|
||||
SEARXNG_URL = os.environ.get("BM_SEARXNG_URL", "").rstrip("/")
|
||||
try:
|
||||
TEMPERATURE = float(os.environ.get("BM_TEMPERATURE", "") or "0.3")
|
||||
except ValueError:
|
||||
@@ -48,156 +61,44 @@ except ValueError:
|
||||
MAX_MODEL_LEN = 32768
|
||||
|
||||
DOCS = "/docs"
|
||||
REPORTS = "/reports"
|
||||
OUT_DIR = "/out"
|
||||
BDEF_PATH = "/BDEF.md"
|
||||
SCHEMA_PATH = "/schema.json"
|
||||
PERSONA_PATH = "/persona/PERSONA.md"
|
||||
RUBRIC_PATH = "/RUBRIC.md"
|
||||
GRADES_DIR = "/grades"
|
||||
EXTRACTION_PATH = "/extraction.json"
|
||||
OUT_DIR = "/out"
|
||||
|
||||
# Leave headroom for the system/rubric/persona + the model's output; spend the
|
||||
# rest on document text (~3 chars/token is a safe rough estimate).
|
||||
DOC_BUDGET = max(8000, (MAX_MODEL_LEN - 3500) * 3)
|
||||
MAX_STEPS = 8
|
||||
MAX_OUTPUT_TOKENS = 2048
|
||||
MAX_OUTPUT_TOKENS = 3072
|
||||
TRANSPORT_RETRIES = 4
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
print(f"[{GID}] {msg}", flush=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- io helpers
|
||||
def read_text(path: str, limit: int = 1_000_000) -> str:
|
||||
def read_text(path: str, limit: int = 2_000_000) -> str:
|
||||
try:
|
||||
with open(path, errors="replace") as f:
|
||||
return f.read()[:limit]
|
||||
except FileNotFoundError:
|
||||
except (FileNotFoundError, IsADirectoryError):
|
||||
return ""
|
||||
|
||||
|
||||
def _roots() -> list[str]:
|
||||
return [DOCS, REPORTS] if ROLE == "synthesizer" else [DOCS]
|
||||
|
||||
|
||||
def _safe(path: str) -> str:
|
||||
"""Resolve a path inside an allowed read root; refuse escapes."""
|
||||
cand = path or "."
|
||||
for root in _roots():
|
||||
p = os.path.realpath(os.path.join(root, cand) if not os.path.isabs(cand) else cand)
|
||||
if p == root or p.startswith(root + os.sep):
|
||||
return p
|
||||
raise ValueError(f"path outside allowed roots: {path}")
|
||||
|
||||
|
||||
def list_dir(root: str) -> list[str]:
|
||||
out = []
|
||||
if not os.path.isdir(root):
|
||||
return out
|
||||
for r, _dirs, files in os.walk(root):
|
||||
for fn in files:
|
||||
out.append(os.path.relpath(os.path.join(r, fn), root))
|
||||
return sorted(out)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- tools
|
||||
def tool_list_files(args: dict) -> str:
|
||||
lines = []
|
||||
for root in _roots():
|
||||
names = list_dir(root)
|
||||
if names:
|
||||
lines.append(f"{root}:")
|
||||
lines += [f" {n}" for n in names]
|
||||
return "\n".join(lines) or "(no files)"
|
||||
|
||||
|
||||
def tool_read_file(args: dict) -> str:
|
||||
p = _safe(args["path"])
|
||||
try:
|
||||
with open(p, errors="replace") as f:
|
||||
return f.read()[:20000]
|
||||
except FileNotFoundError:
|
||||
return f"(no such file: {args['path']})"
|
||||
except IsADirectoryError:
|
||||
return f"(is a directory: {args['path']})"
|
||||
|
||||
|
||||
def tool_web_search(args: dict) -> str:
|
||||
if not SEARXNG_URL:
|
||||
return "web search unavailable"
|
||||
q = urllib.parse.urlencode({"q": args.get("query", ""), "format": "json"})
|
||||
try:
|
||||
req = urllib.request.Request(f"{SEARXNG_URL}/search?{q}", headers={"User-Agent": "boardroom-grader"})
|
||||
with urllib.request.urlopen(req, timeout=20, context=_SSL_CTX) as resp:
|
||||
data = json.loads(resp.read().decode())
|
||||
lines = [f"- {r.get('title','')}\n {r.get('url','')}\n {r.get('content','')[:300]}"
|
||||
for r in (data.get("results") or [])[:8]]
|
||||
return "\n".join(lines) or "(no results)"
|
||||
except Exception as e:
|
||||
return f"search error: {e}"
|
||||
|
||||
|
||||
DISPATCH = {"list_files": tool_list_files, "read_file": tool_read_file, "web_search": tool_web_search}
|
||||
|
||||
TOOLS = [
|
||||
{"type": "function", "function": {
|
||||
"name": "list_files", "description": "List the available document (and report) files.",
|
||||
"parameters": {"type": "object", "properties": {}}}},
|
||||
{"type": "function", "function": {
|
||||
"name": "read_file", "description": "Read a document or report file by its path (from list_files).",
|
||||
"parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}},
|
||||
]
|
||||
if SEARXNG_URL:
|
||||
TOOLS.append({"type": "function", "function": {
|
||||
"name": "web_search", "description": "Search the web via SearXNG; returns top results.",
|
||||
"parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}})
|
||||
|
||||
|
||||
def run_tool(name: str, args: dict) -> str:
|
||||
fn = DISPATCH.get(name)
|
||||
if not fn:
|
||||
return f"(unknown tool: {name})"
|
||||
try:
|
||||
return fn(args)
|
||||
except Exception as e:
|
||||
return f"(tool error: {e})"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- LLM
|
||||
def chat(messages: list) -> dict:
|
||||
body = {"model": MODEL, "messages": messages, "tools": TOOLS,
|
||||
"tool_choice": "auto", "temperature": TEMPERATURE, "max_tokens": MAX_OUTPUT_TOKENS}
|
||||
req = urllib.request.Request(
|
||||
f"{LLM_BASE}/chat/completions",
|
||||
data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"},
|
||||
method="POST")
|
||||
with urllib.request.urlopen(req, timeout=300, context=_SSL_CTX) as resp:
|
||||
data = json.loads(resp.read().decode())
|
||||
return data["choices"][0]["message"]
|
||||
|
||||
|
||||
_JSON_ACTION = re.compile(r"```(?:json)?\s*(\{.*?\})\s*```", re.DOTALL)
|
||||
|
||||
|
||||
def _text_fallback_calls(content: str) -> list:
|
||||
if not content:
|
||||
def _doc_names() -> list[str]:
|
||||
if not os.path.isdir(DOCS):
|
||||
return []
|
||||
m = _JSON_ACTION.search(content)
|
||||
if not m:
|
||||
return []
|
||||
try:
|
||||
obj = json.loads(m.group(1))
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
name = obj.get("tool") or obj.get("name")
|
||||
if name in DISPATCH:
|
||||
return [{"id": "fallback", "function": {"name": name, "arguments": json.dumps(obj.get("args", {}))}}]
|
||||
return []
|
||||
return sorted(fn for fn in os.listdir(DOCS)
|
||||
if fn.endswith(".txt") and os.path.isfile(os.path.join(DOCS, fn)))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- prompts
|
||||
def _preload_docs() -> tuple[str, bool]:
|
||||
"""Concatenate the document text up to DOC_BUDGET. Returns (text, truncated)."""
|
||||
names = list_dir(DOCS)
|
||||
def preload_docs(budget: int) -> tuple[str, bool]:
|
||||
"""Concatenate /docs/*.txt up to `budget` chars. Returns (text, truncated)."""
|
||||
chunks, used, truncated = [], 0, False
|
||||
for n in names:
|
||||
for n in _doc_names():
|
||||
body = read_text(os.path.join(DOCS, n))
|
||||
header = f"\n\n========== DOCUMENT: {n} ==========\n"
|
||||
room = DOC_BUDGET - used
|
||||
room = budget - used
|
||||
if room <= 0:
|
||||
truncated = True
|
||||
break
|
||||
@@ -209,129 +110,337 @@ def _preload_docs() -> tuple[str, bool]:
|
||||
return "".join(chunks), truncated
|
||||
|
||||
|
||||
def _preload_reports() -> str:
|
||||
names = list_dir(REPORTS)
|
||||
def out_path() -> str:
|
||||
if ROLE == "adjudicator":
|
||||
return os.path.join(OUT_DIR, "ADJUDICATION.md")
|
||||
if ROLE == "extractor":
|
||||
return os.path.join(OUT_DIR, "extraction.json")
|
||||
return os.path.join(OUT_DIR, f"{GID}.json")
|
||||
|
||||
|
||||
def write_out(text: str) -> None:
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
with open(out_path(), "w") as f:
|
||||
f.write(text.rstrip() + "\n")
|
||||
|
||||
|
||||
def write_invalid_marker(err: str) -> None:
|
||||
try:
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
with open(out_path() + ".invalid", "w") as f:
|
||||
f.write(err.strip() + "\n")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- LLM client
|
||||
def _post(payload: dict, timeout: int = 600) -> dict:
|
||||
req = urllib.request.Request(
|
||||
f"{LLM_BASE}/chat/completions",
|
||||
data=json.dumps(payload).encode(),
|
||||
headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"},
|
||||
method="POST")
|
||||
with urllib.request.urlopen(req, timeout=timeout, context=_SSL_CTX) as resp:
|
||||
return json.loads(resp.read().decode())
|
||||
|
||||
|
||||
def chat(messages: list, extra: dict | None = None) -> str:
|
||||
"""One completion. Retries with backoff on 5xx/connection errors; raises
|
||||
HTTPError immediately on 4xx so the caller can walk the reliability ladder."""
|
||||
payload = {"model": MODEL, "messages": messages,
|
||||
"temperature": TEMPERATURE, "max_tokens": MAX_OUTPUT_TOKENS}
|
||||
if extra:
|
||||
payload.update(extra)
|
||||
last: Exception | None = None
|
||||
for attempt in range(1, TRANSPORT_RETRIES + 1):
|
||||
try:
|
||||
data = _post(payload)
|
||||
return (data["choices"][0]["message"].get("content") or "").strip()
|
||||
except urllib.error.HTTPError as e:
|
||||
if 400 <= e.code < 500:
|
||||
raise
|
||||
last = e
|
||||
except Exception as e: # URLError, timeout, bad JSON envelope...
|
||||
last = e
|
||||
sleep = 5 * attempt
|
||||
log(f"transport error ({last}); retry {attempt}/{TRANSPORT_RETRIES} in {sleep}s")
|
||||
time.sleep(sleep)
|
||||
raise RuntimeError(f"model endpoint unreachable after {TRANSPORT_RETRIES} attempts: {last}")
|
||||
|
||||
|
||||
def chat_json(messages: list, schema: dict) -> str:
|
||||
"""The JSON reliability ladder: strict json_schema -> guided_json -> plain."""
|
||||
ladder = [
|
||||
("json_schema", {"response_format": {"type": "json_schema", "json_schema": {
|
||||
"name": schema.get("title") or "output", "schema": schema, "strict": True}}}),
|
||||
("guided_json", {"guided_json": schema}),
|
||||
("plain", None),
|
||||
]
|
||||
last: Exception | None = None
|
||||
for mode, extra in ladder:
|
||||
try:
|
||||
return chat(messages, extra=extra)
|
||||
except urllib.error.HTTPError as e:
|
||||
if 400 <= e.code < 500:
|
||||
log(f"{mode} request rejected (HTTP {e.code}); trying next mode")
|
||||
last = e
|
||||
continue
|
||||
raise
|
||||
raise RuntimeError(f"all completion modes were rejected by the endpoint: {last}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- JSON parse
|
||||
def parse_json(text: str):
|
||||
"""Whole-reply json.loads, else the first brace-balanced {...} block."""
|
||||
if not text:
|
||||
return None
|
||||
try:
|
||||
return json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
start = text.find("{")
|
||||
while start != -1:
|
||||
depth, in_str, esc = 0, False, False
|
||||
for i in range(start, len(text)):
|
||||
c = text[i]
|
||||
if in_str:
|
||||
if esc:
|
||||
esc = False
|
||||
elif c == "\\":
|
||||
esc = True
|
||||
elif c == '"':
|
||||
in_str = False
|
||||
elif c == '"':
|
||||
in_str = True
|
||||
elif c == "{":
|
||||
depth += 1
|
||||
elif c == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
try:
|
||||
return json.loads(text[start:i + 1])
|
||||
except json.JSONDecodeError:
|
||||
break
|
||||
start = text.find("{", start + 1)
|
||||
return None
|
||||
|
||||
|
||||
def structural_error(obj) -> str | None:
|
||||
"""Lightweight in-agent checks; the full jsonschema pass is orchestrator-side."""
|
||||
if not isinstance(obj, dict):
|
||||
return "top-level value is not a JSON object"
|
||||
if ROLE == "extractor":
|
||||
for k in ("schema_version", "deck", "kpis", "forward_targets",
|
||||
"red_flag_candidates", "narrative"):
|
||||
if k not in obj:
|
||||
return f"missing required key: {k}"
|
||||
if not isinstance(obj.get("deck"), dict) or "period" not in obj["deck"]:
|
||||
return "deck.period is missing"
|
||||
else: # grader
|
||||
for k in ("schema_version", "grader", "categories", "red_flags", "overall_comment"):
|
||||
if k not in obj:
|
||||
return f"missing required key: {k}"
|
||||
cats = obj.get("categories")
|
||||
if not isinstance(cats, list) or len(cats) != 8:
|
||||
return "categories must contain exactly 8 entries (A-H)"
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- prompts
|
||||
def _persona() -> str:
|
||||
return read_text(PERSONA_PATH).strip()
|
||||
|
||||
|
||||
def _bdef() -> str:
|
||||
return read_text(BDEF_PATH).strip()
|
||||
|
||||
|
||||
def _red_flag_taxonomy() -> str:
|
||||
"""The '## Red-flag taxonomy' section of BDEF.md (falls back to the whole rubric)."""
|
||||
bdef = _bdef()
|
||||
low = bdef.lower()
|
||||
i = low.find("## red-flag taxonomy")
|
||||
return bdef[i:].strip() if i != -1 else bdef
|
||||
|
||||
|
||||
def build_extractor_prompt(schema: dict) -> tuple[str, str, bool]:
|
||||
schema_text = json.dumps(schema, indent=2)
|
||||
system = (
|
||||
"You are the structured-data EXTRACTOR for a board-deck grading pipeline. "
|
||||
"You turn the deck text into machine-readable JSON; you do not grade.\n\n"
|
||||
"HARD RULES:\n"
|
||||
"- Reply with ONLY one JSON object conforming exactly to the schema below. "
|
||||
"No prose, no markdown fences, no comments.\n"
|
||||
"- NEVER invent numbers. Every actual/target must appear in the deck text; "
|
||||
"record where in \"source\".\n"
|
||||
"- canonical_name is lower_snake_case, generic, and stable across quarters "
|
||||
"(arr, ebitda_margin, churn_rate...).\n"
|
||||
"- deck.period is the reporting period as printed on the deck "
|
||||
"(e.g. 2026-Q2, 2026-H1, FY2026, 2026-05); null if truly absent.\n"
|
||||
"- If the deck text you were given was truncated, set deck.truncated = true.\n\n"
|
||||
"# OUTPUT SCHEMA (JSON Schema)\n" + schema_text
|
||||
)
|
||||
persona = _persona()
|
||||
if persona:
|
||||
system = persona + "\n\n" + system
|
||||
taxonomy = _red_flag_taxonomy()
|
||||
prefix = ("# RED-FLAG TAXONOMY\nUse ONLY these codes when populating "
|
||||
"red_flag_candidates:\n\n" + taxonomy + "\n\n# BOARD DECK TEXT\n")
|
||||
suffix = ("\n\n# TASK\nExtract the deck metadata, every KPI actual, every "
|
||||
"forward-looking target, red-flag candidates (taxonomy codes above), "
|
||||
"and the narrative summary. Output the JSON object now.")
|
||||
budget = max(8000, (MAX_MODEL_LEN - 3500) * 3 - len(system) - len(prefix) - len(suffix))
|
||||
docs, truncated = preload_docs(budget)
|
||||
note = ("\n\n(NOTE: the deck text above was TRUNCATED to fit the context window — "
|
||||
"set deck.truncated = true.)" if truncated else "")
|
||||
return system, prefix + docs + note + suffix, truncated
|
||||
|
||||
|
||||
def build_grader_prompt(schema: dict) -> tuple[str, str, bool]:
|
||||
schema_text = json.dumps(schema, indent=2)
|
||||
system = (
|
||||
f"You are '{NAME}', one grader on a panel scoring a portfolio-company board "
|
||||
"deck against the BDEF rubric.\n\n"
|
||||
"HARD RULES:\n"
|
||||
"- Score every BDEF category A-H with an integer 1-5.\n"
|
||||
"- Any score ABOVE or BELOW 3 REQUIRES verbatim evidence quotes from the deck, "
|
||||
"each with a location (e.g. 'slide 6'). Unsupported non-3 scores will be "
|
||||
"regressed to 3 by the pipeline.\n"
|
||||
"- Do NOT compute totals or composite scores; numbers are computed elsewhere.\n"
|
||||
f"- Set \"grader\" to exactly \"{GID}\".\n"
|
||||
"- Use only the red-flag taxonomy codes defined in the rubric.\n"
|
||||
"- Reply with ONLY one JSON object conforming exactly to the provided schema. "
|
||||
"No prose, no markdown fences."
|
||||
)
|
||||
persona = _persona()
|
||||
if persona:
|
||||
system += "\n\n# YOUR LENS — how YOU specifically read this deck\n" + persona
|
||||
bdef = _bdef()
|
||||
prefix = "# BDEF RUBRIC\n" + bdef + "\n\n# BOARD DECK TEXT\n"
|
||||
suffix = ("\n\n# OUTPUT SCHEMA (JSON Schema)\n" + schema_text +
|
||||
"\n\n# TASK\nGrade the deck per the rubric and your lens. "
|
||||
"Output the JSON object now.")
|
||||
budget = max(8000, (MAX_MODEL_LEN - 3500) * 3 - len(system) - len(prefix) - len(suffix))
|
||||
docs, truncated = preload_docs(budget)
|
||||
note = ("\n\n(NOTE: the deck text above was TRUNCATED to fit the context window — "
|
||||
"grade what is shown and mention the truncation in overall_comment.)"
|
||||
if truncated else "")
|
||||
return system, prefix + docs + note + suffix, truncated
|
||||
|
||||
|
||||
def _load_panel_grades() -> str:
|
||||
"""All /grades/*.json panel reports, skipping extraction.json and anything
|
||||
flagged invalid by the agent that produced it (sibling .invalid marker)."""
|
||||
parts = []
|
||||
budget = DOC_BUDGET
|
||||
used = 0
|
||||
for n in names:
|
||||
body = read_text(os.path.join(REPORTS, n))
|
||||
header = f"\n\n========== REVIEWER REPORT: {n} ==========\n"
|
||||
seg = (header + body)[: max(0, budget - used)]
|
||||
parts.append(seg)
|
||||
used += len(seg)
|
||||
if not os.path.isdir(GRADES_DIR):
|
||||
return ""
|
||||
for fn in sorted(os.listdir(GRADES_DIR)):
|
||||
p = os.path.join(GRADES_DIR, fn)
|
||||
if not fn.endswith(".json") or not os.path.isfile(p):
|
||||
continue
|
||||
if fn == "extraction.json" or os.path.exists(p + ".invalid"):
|
||||
continue
|
||||
parts.append(f"\n\n========== GRADER REPORT: {fn} ==========\n" + read_text(p))
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def system_prompt() -> str:
|
||||
persona = read_text(PERSONA_PATH).strip()
|
||||
if ROLE == "synthesizer":
|
||||
base = (f"You are '{NAME}', the lead reviewer chairing a document-review panel. "
|
||||
"You are given the panel members' individual reports (and the source "
|
||||
"documents for reference). Produce ONE consolidated report in Markdown.")
|
||||
else:
|
||||
base = (f"You are '{NAME}', an expert confidential-document reviewer. Read the "
|
||||
"document(s) provided and produce ONE written report in Markdown. Base every "
|
||||
"statement on the documents; never invent facts. Be specific and cite the "
|
||||
"document/section for each point.")
|
||||
if persona:
|
||||
base += "\n\n# YOUR LENS — how YOU specifically read this\n" + persona
|
||||
tools_note = (
|
||||
"\n\nYou can call list_files and read_file to pull more content if what was "
|
||||
"pre-loaded is truncated"
|
||||
+ (", and web_search for external context" if SEARXNG_URL else "")
|
||||
+ ". When done, reply with the FINAL report only — no tool call. If your client "
|
||||
'cannot emit tool calls, reply with a single fenced block: '
|
||||
'```json\\n{"tool":"read_file","args":{"path":"..."}}\\n``` and nothing else.'
|
||||
def build_adjudicator_prompt() -> tuple[str, str]:
|
||||
system = _persona() or (
|
||||
"You are the adjudicator chairing a panel of board-deck graders. You did not "
|
||||
"read the deck first-hand for a fresh opinion — you weigh the panel's evidence."
|
||||
)
|
||||
return base + tools_note
|
||||
|
||||
|
||||
def first_user_message() -> str:
|
||||
rubric = read_text(RUBRIC_PATH).strip() or "Produce a thorough review report."
|
||||
if ROLE == "synthesizer":
|
||||
reports = _preload_reports()
|
||||
docs, truncated = _preload_docs()
|
||||
return (f"# REVIEW RUBRIC\n{rubric}\n\n# PANEL REPORTS\n{reports}\n\n"
|
||||
f"# SOURCE DOCUMENTS (for reference){' (truncated)' if truncated else ''}\n{docs}\n\n"
|
||||
"# YOUR TASK\nConsolidate the panel's reports into one authoritative report per the "
|
||||
"rubric: shared findings, conflicts (and your adjudication), anything only one "
|
||||
"reviewer caught, and a prioritized overall recommendation. Attribute points to "
|
||||
"reviewers. Output the final consolidated report now.")
|
||||
docs, truncated = _preload_docs()
|
||||
note = ("\n\n(Note: the documents were truncated to fit context — use read_file to pull any "
|
||||
"section you need in full.)" if truncated else "")
|
||||
return (f"# REVIEW RUBRIC\n{rubric}\n\n# DOCUMENT(S)\n{docs}{note}\n\n"
|
||||
"# YOUR TASK\nReview the document(s) above per the rubric and your lens. Output your "
|
||||
"final report now.")
|
||||
extraction_txt = read_text(EXTRACTION_PATH)
|
||||
grades_txt = _load_panel_grades()
|
||||
body = ("# STRUCTURED EXTRACTION (ground truth pulled from the deck)\n" +
|
||||
extraction_txt + "\n\n# PANEL GRADE REPORTS\n" + grades_txt)
|
||||
budget = max(8000, (MAX_MODEL_LEN - 3500) * 3 - len(system) - 1500)
|
||||
if len(body) > budget:
|
||||
body = body[:budget] + "\n\n(NOTE: input truncated to fit the context window.)"
|
||||
task = (
|
||||
"\n\n# TASK\nWrite a MARKDOWN adjudication of the panel:\n"
|
||||
"- Consensus per BDEF category A-H (one line each).\n"
|
||||
"- Material disagreements: where graders diverge, what each cites, and whose "
|
||||
"evidence is stronger (verbatim deck quotes beat assertions).\n"
|
||||
"- Red flags: which are CONFIRMED and which are DISMISSED, and why.\n"
|
||||
"- Exactly 3 questions the board should ask management next quarter.\n"
|
||||
"Do NOT output numeric scores, totals, or JSON — narrative Markdown only."
|
||||
)
|
||||
return system, body + task
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- run
|
||||
def out_path() -> str:
|
||||
name = "CONSOLIDATED_REPORT.md" if ROLE == "synthesizer" else f"{RID}.md"
|
||||
return os.path.join(OUT_DIR, name)
|
||||
def finalize(obj: dict, truncated: bool) -> dict:
|
||||
if ROLE == "grader":
|
||||
obj["grader"] = GID
|
||||
elif ROLE == "extractor" and truncated:
|
||||
deck = obj.get("deck")
|
||||
if isinstance(deck, dict):
|
||||
deck["truncated"] = True
|
||||
return obj
|
||||
|
||||
|
||||
def write_report(text: str) -> None:
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
with open(out_path(), "w") as f:
|
||||
f.write(text.strip() + "\n")
|
||||
def run_json_role() -> None:
|
||||
schema = json.loads(read_text(SCHEMA_PATH) or "{}")
|
||||
if ROLE == "extractor":
|
||||
system, user, truncated = build_extractor_prompt(schema)
|
||||
else:
|
||||
system, user, truncated = build_grader_prompt(schema)
|
||||
if truncated:
|
||||
log("document text truncated to fit the context window")
|
||||
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
|
||||
|
||||
text = chat_json(messages, schema)
|
||||
obj = parse_json(text)
|
||||
err = structural_error(obj) if obj is not None else "reply was not parseable JSON"
|
||||
if obj is not None and err is None:
|
||||
write_out(json.dumps(finalize(obj, truncated), indent=2))
|
||||
log(f"wrote {out_path()}")
|
||||
return
|
||||
|
||||
# ONE repair round-trip.
|
||||
log(f"invalid reply ({err}); attempting one repair round-trip")
|
||||
repair = messages + [
|
||||
{"role": "assistant", "content": text or "(empty reply)"},
|
||||
{"role": "user", "content": (
|
||||
f"Your previous reply was not valid JSON or failed validation: {err}. "
|
||||
"Reply with ONLY the corrected JSON.")},
|
||||
]
|
||||
text2 = chat_json(repair, schema)
|
||||
obj2 = parse_json(text2)
|
||||
err2 = structural_error(obj2) if obj2 is not None else "reply was not parseable JSON"
|
||||
if obj2 is not None and err2 is None:
|
||||
write_out(json.dumps(finalize(obj2, truncated), indent=2))
|
||||
log(f"wrote {out_path()} (after repair)")
|
||||
return
|
||||
|
||||
# Still bad: leave the raw text + an .invalid marker for the orchestrator.
|
||||
write_out(text2 or text or "")
|
||||
write_invalid_marker(f"invalid after repair round-trip: {err2}")
|
||||
log(f"FAILED to produce valid JSON: {err2} (raw text + .invalid marker written)")
|
||||
|
||||
|
||||
def run() -> str:
|
||||
messages = [{"role": "system", "content": system_prompt()},
|
||||
{"role": "user", "content": first_user_message()}]
|
||||
last_text = ""
|
||||
for _ in range(MAX_STEPS):
|
||||
msg = chat(messages)
|
||||
content = msg.get("content") or ""
|
||||
calls = msg.get("tool_calls") or []
|
||||
if content.strip():
|
||||
last_text = content.strip()
|
||||
if not calls:
|
||||
calls = _text_fallback_calls(content)
|
||||
if not calls:
|
||||
break # final report
|
||||
messages.append({"role": "assistant", "content": content})
|
||||
for c in calls:
|
||||
args = json.loads(c["function"]["arguments"] or "{}")
|
||||
res = run_tool(c["function"]["name"], args)
|
||||
messages.append({"role": "user", "content": f"[tool {c['function']['name']} result]\n{res[:20000]}"})
|
||||
continue
|
||||
messages.append({"role": "assistant", "content": content or None, "tool_calls": calls})
|
||||
for c in calls:
|
||||
try:
|
||||
args = json.loads(c["function"]["arguments"] or "{}")
|
||||
except json.JSONDecodeError:
|
||||
args = {}
|
||||
res = run_tool(c["function"]["name"], args)
|
||||
messages.append({"role": "tool", "tool_call_id": c.get("id", ""), "content": res[:20000]})
|
||||
|
||||
# If the model ended on a tool turn with no report text, ask once more plainly.
|
||||
if not last_text.strip():
|
||||
messages.append({"role": "user", "content": "Now output your final report in Markdown."})
|
||||
try:
|
||||
last_text = (chat(messages).get("content") or "").strip()
|
||||
except Exception:
|
||||
pass
|
||||
return last_text
|
||||
def run_adjudicator() -> None:
|
||||
system, user = build_adjudicator_prompt()
|
||||
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
|
||||
text = chat(messages)
|
||||
if not text.strip():
|
||||
raise RuntimeError("model returned an empty adjudication")
|
||||
write_out(text)
|
||||
log(f"wrote {out_path()} ({len(text)} chars)")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
print(f"[{RID}] reviewer starting (role={ROLE} model={MODEL} base={LLM_BASE})", flush=True)
|
||||
log(f"starting (role={ROLE} model={MODEL} base={LLM_BASE} temp={TEMPERATURE})")
|
||||
try:
|
||||
report = run()
|
||||
if not report.strip():
|
||||
report = f"# {NAME}\n\n(The model returned no report text.)"
|
||||
write_report(report)
|
||||
print(f"[{RID}] report written to {out_path()} ({len(report)} chars)", flush=True)
|
||||
if ROLE == "adjudicator":
|
||||
run_adjudicator()
|
||||
else:
|
||||
run_json_role()
|
||||
except Exception as e:
|
||||
print(traceback.format_exc(), flush=True)
|
||||
# Always leave a file so the orchestrator can see this reviewer ran.
|
||||
try:
|
||||
write_report(f"# {NAME} — ERROR\n\nThis reviewer failed: {e}\n")
|
||||
except Exception:
|
||||
pass
|
||||
if ROLE != "adjudicator":
|
||||
# Leave a marker so the orchestrator sees this agent ran and failed.
|
||||
write_invalid_marker(f"agent error: {e}")
|
||||
raise
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user