ten31-transcripts/docs/01_PROJECT_BRIEF.md

Project Brief — Ten31 Transcripts

Local macOS app that auto-detects conference calls, records local audio, and produces a visual-derived, timestamped speaker timeline — then hands the mixed audio + that timeline to the operator's SparkControl backend, which diarizes, names the speakers (majority-overlap vote against the timeline), and returns named transcript segments. A growing voiceprint library recovers speakers even when the visual cue is missing.

Master context document. Read this first, then 02_ARCHITECTURE.md and 03_DATA_CONTRACTS.md. The SparkControl API is fully specified in 03_DATA_CONTRACTS.md.

---

1. What we are building

A lightweight, always-running menu-bar app on macOS that:

1. Detects when the user joins a call in Google Meet, Zoom, Microsoft Teams, or Signal.
2. Records two local audio tracks — system audio (everyone else) and the user's microphone (the user). It sends the backend dual-channel (mic_file + system_file) when the system track is healthy, falling back to a mixed-mono 16 kHz WAV otherwise.
3. Watches the call window at ~2–4 fps and, per app, reads participant names and the active-speaker cue, producing a (start, end, name, confidence) visual timeline — its best guess at who was talking when.
4. Discards every video frame after extraction. No video is ever written to disk. Only audio + the derived timeline persist locally.
5. On call end, POSTs the audio + the visual timeline (+ the known voiceprint library) to POST /api/audio/label-merge on SparkControl, which returns named, speaker-attributed transcript segments and a voiceprint per speaker.
6. Persists the returned voiceprints keyed by name, so the next call can pass them as known_voiceprints and recover a speaker by voice when the visual cue is absent (camera off, a bad OCR frame).
7. Renders the result locally — a readable transcript.md plus an HTML recap.html (topics + meeting extras, generated via the backend's LLM endpoint), with an in-app editor for fixing speaker names after the fact.

The app's job ends at producing the named transcript and recap from SparkControl's segments. All transcription, diarization, name-merge, and LLM analysis happen on the backend. Do not build transcription, diarization, or the merge vote in this app.

2. Why the visual timeline still matters (the core idea)

Audio diarization (NVIDIA Sortformer on the backend) is excellent at segmentation — precise speaker boundaries — but its clusters are anonymous (Speaker_0, Speaker_1…). It cannot name anyone.

The screen already knows the names. Each app visually marks the active speaker (colored tile border, animated audio bars, a ring around an avatar) next to that person's name/initials.

So responsibilities split cleanly:

• Audio (backend) owns segmentation — the exact when.
• Visual capture (this app) owns identity — the who.
• label-merge (backend) fuses them: it diarizes, then assigns each cluster the timeline name with the most temporal overlap. The visual track needn't be perfect — it only needs to win the per-cluster vote.

New compounding layer — the voiceprint library. Every named cluster comes back from the backend with a 192-dim TitaNet voiceprint. The app persists these keyed by name and replays them as known_voiceprints next time. Resolution order per cluster becomes: visual overlap → voiceprint match → Unknown_N (never mislabeled). So the screen capture enrolls a voice library for free, and over a few calls the system can name regulars even with cameras off.

3. Hard scope boundaries

In scope (this app):

• Call detection for Meet / Zoom / Teams / Signal.
• Dual-track local audio capture; dual-channel send (mic + system) with a mix-to-mono fallback for the backend.
• Low-fps window capture → OCR (names) + active-speaker cue detection.
• Per-app "adapter" modules encapsulating each app's UI quirks.
• Building the visual timeline; mic-VAD self-labeling (the mic track is the user, so hot-mic spans pre-seed the user's name into the timeline).
• Chunking long calls (~2–3 min) and calling label-merge sequentially.
• A local voiceprint store (persist + replay named voiceprints).
• Storing the backend's named segments and rendering them — transcript.md plus an HTML recap.html (recap analysis via the backend LLM) — with an in-app speaker-name editor.
• A minimal menu-bar UI: status, manual start/stop, the last session (reveal, resend, open recap, edit speakers), adapter toggles, backend host/health, output folder.

Out of scope (owned by the backend):

• Transcription, diarization, the name-merge vote, and LLM summarization — these run on the backend; the app only orchestrates the recap call and renders the result.

Explicitly not doing: saving video; cloud anything. Everything stays on the operator's LAN.

4. Key decisions (now resolved against the real contract)

Decision	Choice	Why
Language / framework	Native Swift + SwiftUI menu-bar app (LSUIElement)	System audio, window capture, Vision all native; one codebase.
Audio capture	ScreenCaptureKit (system audio) + AVFoundation (mic)	No virtual audio device; works with headphones; macOS 13+.
Backend audio format	Dual-channel (mic + system) when the system track is healthy, else mixed-mono 16 kHz WAV	Separate tracks let the backend attribute the user's mic channel directly; the diarizer can still split the mono fallback.
Call detection	CoreAudio "mic running somewhere" + known-app / Meet-tab heuristic	Clean live-mic signal + app disambiguation.
Speaker naming	Backend, via POST /api/audio/label-merge	One call does diarize + overlap-vote naming + transcription. No client merge.
Identity recovery	Local voiceprint library replayed as known_voiceprints	Recovers camera-off / OCR-missed speakers by voice; compounds over calls.
Self-identity	mic-VAD → pre-seed user's name in timeline	The mic track is the user; gives the backend a strong prior + enrolls the user's voiceprint immediately.
Requests	Sequential, one audio request in flight	Parallel audio requests trip a backend GPU race (503 + Retry-After).
Long calls	Chunk ~2–3 min, sequential, stitch via names+voiceprints	Diarizer caps at 4 speakers/chunk; voiceprints + names unify across chunks.
Transport / TLS	multipart/form-data, file field file (mono) or mic_file + system_file (dual-channel); self-signed Start9 cert (trust the Root CA — supported default; host-scoped skip-verify is an off-by-default escape hatch); no auth on LAN	Matches every other SparkControl endpoint.
Timing	Batch after call (sync endpoints, no polling)	Endpoints are synchronous; no job/poll machinery needed.

On forking Hyprnote

Unchanged recommendation: the audio capture is the trivial part (~200 lines of native Swift) and the rest (Vision screen-reading) is native too. Build native; use Hyprnote's capture/detection only as reference. Fork remains an override.

5. Target apps & identifiers

App	Join via	Bundle ID(s)	Speaking cue / names
Zoom	Native	us.zoom.xos	Colored tile border; name label in tile.
Microsoft Teams	Native (new)	com.microsoft.teams2 (new), com.microsoft.teams (classic)	Colored ring/border; labeled.
Signal	Native (Electron)	org.whispersystems.signal-desktop	Ring around avatar/initials; try Accessibility names first.
Google Meet	Browser tab	com.google.Chrome, com.apple.Safari, company.thebrowser.Browser (Arc)…	Canvas video → Vision for the cue; DOM names → Accessibility/AppleScript; confirm via active-tab URL meet.google.com.

Four required adapters; adding a 5th must be one new file conforming to the AppAdapter protocol.

6. The backend (do not rebuild) — now concrete

SparkControl, on the operator's Start9 LAN, fronting two DGX Sparks:

• STT: NVIDIA Parakeet TDT 0.6B — POST /v1/audio/transcriptions (OpenAI-compatible).
• Diarization: NVIDIA Sortformer 4spk-v1 — POST /api/audio/diarize-chunk (anonymous clusters + voiceprints) and POST /api/audio/transcribe-with-speakers.
• Embeddings: NVIDIA TitaNet (192-dim voiceprints).
• ★ Primary endpoint for this app: POST /api/audio/label-merge — diarize + name from the visual timeline (+ voiceprint fallback), optionally transcribe, in one synchronous call.
• LLM (recap): Qwen3 via OpenAI-compatible POST /v1/chat/completions — generates the readable recap (topics + meeting extras) from the transcript.
• Health/discovery: GET /api/status, GET /api/endpoints, GET /v1/models.

Full request/response shapes, curl examples, limits, and error formats are in 03_DATA_CONTRACTS.md.

7. Settled decisions (were open at brief time)

1. Base URL. A private LAN host — a .local mDNS name (preferred over a raw IP, since it survives IP changes) — configured in Settings or via the SPARK_BACKEND_URL env var, never committed. A neutral placeholder ships as the default and stays editable in Settings. Service-discovery at GET /api/endpoints.
2. Send trigger. Auto-send on call end is a setting (autoSendOnStop), off by default — the user reviews the session and sends manually unless they opt in.
3. Retention. The session folder is kept after a successful hand-off (output location is configurable); nothing is pruned automatically.
4. Voiceprint update policy. Store/refresh the latest high-confidence vector per name (02_ARCHITECTURE.md §2.9); a per-name running average is a possible later refinement.
5. Signing. A stable identity via Config/Signing.xcconfig (gitignored) keeps macOS from re-prompting for permissions on each rebuild.