# Full evaluation — orchestration guide

*Substance file per the portability protocol. Vendor wrappers (e.g.
`adapters/claude/commands/full-eval.md`) point here; this guide is self-contained
and written as plain prose any orchestrating agent could follow.*

You are the orchestrator of a full independent evaluation of one repository. Your job
is fan-out, then synthesis. You do not perform the evaluations yourself, and you do not
fix anything you learn about.

## Phase 1 — Orient (2 minutes, no deep reading)
Read README and AGENTS.md/CLAUDE.md to capture the project's stated intent in one
sentence. Check the file listing for StartOS-wrapper markers (manifest.yaml, *.s9pk,
startos/, start-sdk usage). Check version-control status for uncommitted changes.

## Phase 2 — Fan out
Delegate to these role agents — in parallel where your tooling allows, otherwise
sequentially — each with a task prompt containing: the repo path, the one-sentence
intent, and any focus area you were given.

1. **evaluator** — full six-lens assessment of this repo.
2. **security-auditor** — adversarial audit of this repo.
3. **exerciser** — build, run, and black-box test this repo.
4. **doc-auditor** — claim-by-claim documentation drift audit (READMEs, instruction
   files, public-facing HTML) against what the code actually does.
5. **start9-spec-checker** — only if Phase 1 found StartOS-wrapper markers.
6. **reviewer** — only if there are uncommitted changes; scope = the working diff.

Do not relay agents' reports to the user as they arrive; wait for all of them.

## Phase 3 — Synthesize
Produce ONE report, written to `EVALUATION.md` at the repo root (this file is your only
write), then show the user just the Verdict and Priority queue sections.

This path is stable and overwritten in place on each run, so if `EVALUATION.md` is
committed to version control, a subsequent run produces a reviewable diff of how the
evaluation changed over time. Whether to commit it is the user's call — don't commit it
yourself.

EVALUATION.md structure:

```
# Evaluation — <repo> — <date>
Intent: <the one sentence>
Agents run: <list, with any that failed/were skipped and why>

## Verdict
≤5 sentences synthesizing all reports: overall state, the headline risk, readiness.

## Cross-referenced findings
Where agents corroborate, merge into one finding and say so — e.g. a crash the
exerciser reproduced at the same code path the auditor flagged is ONE P0 with two
kinds of evidence. Deduplicate aggressively.

## Priority queue
P0 → P3, every finding from every agent exactly once, each one line:
[Px] finding — evidence pointer — source agent(s)

## Scorecard
The evaluator's lens table, adjusted only if another agent's evidence contradicts it —
e.g. the doc-auditor's drift findings sharpen the Documentation lens (note any adjustment).

## Disagreements & gaps
Where agents conflict, or where every agent's Coverage section shows the same blind
spot.

## Suggested order of work
3–7 steps, dependencies respected (e.g. "fix the build before trusting test results").
```

## Rules
- Preserve each agent's severity unless evidence from another agent changes it, and
  say so when you do.
- Carry every agent's Surprises forward — don't drop them.
- If an agent failed or returned nothing useful, report that honestly rather than
  papering over it.
- If blocked at any point, report exactly what blocked you — never guess or fabricate
  findings.