ten31-transcripts/docs/04_BUILD_PLAN.md

# Build Plan — Ten31 Transcripts

> **Status: COMPLETE (historical).** Phases 0–6 shipped and the app is in daily
> use; a recap phase (transcript + HTML recap via the backend LLM) was added after
> this plan was written. Kept as the original build log and as the map for the
> "Phase N" references in the code comments. Forward-looking work lives in
> `ROADMAP.md`; current status in `AGENTS.md`.

Companion to docs 01–03. Phased plan for the Claude Code session, each phase with
a demoable milestone. Build in order; the risky/novel work (visual adapters) is
isolated for independent tuning. The SparkControl contract is now known
(`03_DATA_CONTRACTS.md`), so Phase 5 wires the real endpoints.

---

## Ground rules
- **Native Swift + SwiftUI**, menu-bar app (`LSUIElement = true`). macOS 13.0+
  (ScreenCaptureKit system audio).
- **Never write video frames to disk.** Process in-memory, release immediately;
  assert this in `VisualObserver` review.
- **One shared monotonic `t0`** for audio + visual timestamps — wire first.
- **Backend deliverable = `mixed_mono_16k.wav`.** Keep `mic.wav`/`system.wav`
  locally (mic = the user, and the VAD source for self-labeling).
- **SparkControl calls are SEQUENTIAL** (one audio request in flight) over
  self-signed TLS, **no auth**, primary endpoint `POST /api/audio/label-merge`.
- Every adapter must be testable offline against image fixtures.

---

## Phase 0 — Scaffold, permissions, backend ping (milestone: launches + green)
- Menu-bar shell; permissions onboarding (Screen Recording, Microphone,
  Accessibility) with status + deep links.
- Settings: output folder, adapter toggles, **SparkControl base URL** + TLS-skip
  toggle.
- `GET /api/status` health check shown in the menu.
- **Milestone:** app sits in the menu bar; permissions green; backend reachable.

## Phase 1 — Audio capture + mix (milestone: clean mixed-mono WAV)
- `AudioRecorder`: SCK system audio → `system.wav`; AVAudioEngine mic → `mic.wav`;
  shared `t0`; produce **`mixed_mono_16k.wav`**.
- Mic VAD → user "self" speaking spans (held for the timeline).
- Manual start/stop for now.
- **Milestone:** record any call manually → tracks line up, mixed-mono plays back
  clean, mic-VAD spans look right.

## Phase 2 — Call detection (milestone: hands-free start/stop)
- `CallDetector`: CoreAudio mic-running listener + `NSWorkspace` app/frontmost +
  Meet active-tab-URL check; debounce.
- Wire detector → `SessionController` → `AudioRecorder`.
- **Milestone:** joining a call in any of the four apps auto-starts; leaving
  auto-stops; a quick mic test does not false-trigger.

## Phase 3 — VisualObserver + Zoom adapter (milestone: visual_timeline.json)
- `VisualObserver`: window-scoped `SCStream` @3 fps → frames → adapter → release.
  Background/occluded windows capture fine; detect **minimized** (and, for Meet,
  **tab-switched**) → pause analysis + record a `visual_gap`, never emit stale
  frames. Audio keeps recording through gaps.
- `AppAdapter` protocol + `ZoomAdapter` (tile-border cue + OCR names; speaker &
  gallery layouts).
- `TimelineBuilder`: hysteresis, overlap-allowed, merge mic-VAD self spans, alias
  normalization → `visual_timeline.json` (+ the flat `timeline` array form).
- **Offline fixtures:** ~30 sample Zoom frames; unit-test/tune thresholds without
  a live call.
- **Milestone:** a real Zoom call yields a `visual_timeline.json` whose segments
  visibly match who spoke (incl. the user from mic VAD).

## Phase 4 — Remaining adapters (milestone: all four produce timelines)
- `TeamsAdapter`, `SignalAdapter` (a11y names first), `MeetAdapter` (hybrid: a11y
  names + Vision cue). Fixtures per app; per-adapter fps/threshold config.
- **Milestone:** each app yields a sensible timeline; adapters independently
  toggleable.

## Phase 5 — SparkControl hand-off via label-merge (milestone: end-to-end named transcript)
- `SessionPackager`: write session folder; if call > ~3 min, build the chunk plan
  (~2–3 min windows) with **timeline sliced + rebased to chunk-local seconds**.
- `SparkControlClient`:
  - Self-signed TLS handling; multipart `file`; **sequential** requests.
  - Per chunk → `POST /api/audio/label-merge` with `file`, chunk `timeline`,
    `known_voiceprints` (from store), `transcribe=true`.
  - Handle `503 + Retry-After` (wait, retry), `413`, `{"detail":...}` errors.
  - Offset returned `start_ms/end_ms` back to global seconds; concatenate.
- Assemble `speakers.json` (named, global, speaker-attributed transcript).
- Failed sessions stay on disk with a "Resend" affordance.
- **Milestone:** finish a call → `mixed_mono_16k.wav` + timeline delivered → a
  `speakers.json` with real names + accurate boundaries lands locally. Test a
  >3-min call to exercise chunking + cross-chunk name unification.

## Phase 6 — Voiceprint library (milestone: identity that compounds)
- `VoiceprintStore` (`voiceprints.json`): load → `known_voiceprints`; on response,
  store/refresh `fingerprint` for `visual`/high-confidence speakers; skip
  `Unknown_N`.
- Verify the three resolution paths against the backend: (a) **visual** name wins;
  (b) **voiceprint** recovery for a camera-off / OCR-missed speaker; (c)
  **`Unknown_N`** when neither matches.
- Voiceprint manager in the UI (rename, delete, reset).
- **Milestone:** a regular contact gets named on a later call **with their camera
  off**, purely from a stored voiceprint.

## Phase 7 — Polish (milestone: daily-driver quality)
- Recent-sessions list (open/resend/delete); alias-table editor.
- Optional on-screen "recording" indicator toggle.
- Adapter health: warn when detection confidence drops sharply (catches UI
  redesigns early); keep fixtures for quick re-tuning.
- Login-item / launch-at-login.

---

## Risk register
1. **UI fragility (highest ongoing cost):** apps redesign speaking cues / name
   placement. Isolate all pixel/color/label logic in adapters; Phase-7
   confidence-drop warning + fixtures keep re-tuning fast.
2. **Active-speaker lag / crosstalk:** UI cue trails speech and flickers. Mitigated
   by treating visual as a *prior* (the backend votes), hysteresis, and the
   voiceprint fallback.
3. **Meet-in-browser:** canvas video + DOM names; the hybrid adapter needs the most
   iteration — budget extra time.
4. **4-speaker-per-chunk cap (Sortformer):** within a single chunk, >4 simultaneous
   speakers can't all be separated. Chunking + voiceprints + the visual timeline
   mitigate across chunks; for rare large calls, note it as a known limit.
5. **Sequential GPU constraint:** never fire parallel audio requests; queue them.
6. **Permission re-prompts on rebuild:** stable signing identity.

## Definition of done (v1)
Join a call in any of the four apps → app auto-records dual-track audio, mixes to
mono, and builds a visual speaker timeline (incl. mic-VAD self spans) → on
hang-up the audio + timeline go to `label-merge` (chunked + sequential as needed)
→ a `speakers.json` with **real names and accurate boundaries** is produced, the
**voiceprint library is updated**, and **no video is ever written to disk** and
**no transcription/diarization/merge logic lives in this app.**