grant/ten31-transcripts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: grant/ten31-transcripts:main
Branches Tags
grant/ten31-transcripts:main
..
compare: grant/ten31-transcripts:3dd02f8ce6b146c5d3ddfe396e7ba3a91126459a
Branches Tags
grant/ten31-transcripts:main

32 Commits
main .. 3dd02f8ce6

Author SHA1 Message Date
Grant Gilliam 3dd02f8ce6 Add agent instructions; extract signing/backend secrets from source 
- Add AGENTS.md (canonical) + CLAUDE.md symlink + ROADMAP.md
- Move Apple Team ID from project.yml into a gitignored
  Config/Signing.xcconfig via configFiles; commit the .example template
- Replace hardcoded backend host in AppSettings with a neutral
  placeholder + SPARK_BACKEND_URL env-var fallback
- Scrub the Team ID, .local host, and raw LAN IP from README/docs
- Ignore Config/Signing.xcconfig and .env
 2026-06-13 12:23:54 -05:00
Grant Gilliam b0a4b50dac Make diarization chunk length configurable (Auto + presets) 
Chunk size was hardcoded at 2.5-min bodies. Add a Settings control:
Auto / Standard 2.5min / Large group 60s / Fine 90s. Shorter chunks keep fewer
simultaneous speakers per window (Sortformer resolves ~4/chunk), useful for large
calls, at some cost to speed and cross-chunk voice matching.

- ChunkMode (new, pure/testable): mode → body seconds; Auto picks 60s when >4
  participants were detected, else 150s; overlap + single-chunk threshold scale
  with the body length.
- AppSettings.chunkMode (+ typed `chunk`); SettingsView picker with explanation.
- TranscriptPipeline.process gains chunkSeconds; derives overlap/threshold from it.
- SessionController resolves the body from the setting + the session's detected
  participant count (visual_timeline participants) for both send + re-process.
- Participant roster now counts EVERY tile OCR'd, not just who spoke
  (TimelineBuilder.observedNames → VisualObserver → VisualCapture), so the Auto
  call-size signal is meaningful even though speaking-detection is sparse.

Tests: ChunkMode resolution, overlap scaling, short-body re-chunking. 69 pass.
 2026-06-09 10:15:16 -05:00
Grant Gilliam 9a80c7c96e Restyle recap.html to match recap-relay 
The recap output looked notably different from recap-relay (bigger 15px font,
different palette, single-column cards). Match recap-relay's job-output view:
slate/indigo palette (--bg #0a0e1a, --accent #818cf8), 13px base type with the
Helvetica/Arial stack, monospace accent-soft timestamps, and the two-pane layout
— topic list on the left, full diarized transcript on the right, click a topic to
scroll + highlight its range (inline JS, data baked in; no backend fetch). The
summary/takeaways render as recap-relay-style cards in a band above the split.
markdown() output unchanged. 66 tests pass.
 2026-06-08 21:00:56 -05:00
Grant Gilliam 18af17f26c Drop stuck whole-call visual spans at processing time 
Defense-in-depth + salvage for sessions captured before the adapter fix: drop any
vision-source span whose single unbroken duration covers ≥60% of the call. No one
speaks that long without a break, so it's a stuck/false active-speaker cue that
would dominate backend name attribution. Self (mic_vad) spans are never dropped.
Applied to both the live and re-process paths. Test added; 66 pass.
 2026-06-08 16:21:45 -05:00
Grant Gilliam 19ca85abd5 Fix Meet visual: reject solid avatar tiles + screen-share OCR 
Root cause of the "4 people → 2 speakers" Meet call: the colored-border detector
read solid camera-off avatar tiles (orange "J", magenta "G") as active speakers
for the ENTIRE call. Those whole-call phantom spans dominated backend name
attribution, collapsing every remote voice onto one name — and the giant filled
bbox also swallowed screen-share text (WERUNBTC.COM ×49) as a speaker.

Validated against 9 real fixtures (harness over the real MeetAdapter):

Detection:
- FrameSampler.thinColoredPoints: coloured counterpart of thinWhitePoints — keeps
  thin border/ring/pill edges, drops solid colour fills.
- GridCallAnalyzer.isHollow: reject a highlight component whose interior is filled
  (a solid tile) vs a hollow ring (a real border). Config.maxInteriorFill (0.2 default).
- MeetAdapter: detect thin BLUE edges only (hue 180–240°, measured from the
  fixtures), maxInteriorFill 0.3 (real Meet rings ≈0.2–0.3, solid tiles ≈0.36).
- Result on fixtures: John Arnold/Grant Gilliam (solid tiles) now NEVER detected;
  Matt Odell/Mark detected when their blue cue is present. Sparse but never wrong —
  correct for a naming hint over audio diarization.

OCR name hygiene:
- isLikelyName rejects domain-like screen-share text ("WERUNBTC.COM", OCR'd ".GOM").
- cleaned() strips trailing punctuation ("Mark." → "Mark").
- TimelineBuilder.canonicalizeByFrequency folds rare OCR misspellings into a
  dominant near-twin name ("Matt Odel"/"MattOdell" → "Matt Odell", "Mare" → "Mark").

Tests: hollow-ring, extended OCR filter, fuzzy-merge. 65 pass.
 2026-06-08 16:18:52 -05:00
Grant Gilliam 98a198471c Revert adjacent same-speaker segment collapse 
User found the merged transcript lines harder to read — too many sentences
joined into one statement. Remove SpeakerReconciler.mergeAdjacent, its wiring in
finishBackend (restore the no-LLM early return), and its tests. Back to one
segment per diarized utterance.
 2026-06-08 15:52:27 -05:00
Grant Gilliam a273e768dc Open Settings in its own roomy window, not the cramped popover 
Settings was a NavigationLink pushed inside the 320pt menu-bar popover, so the
grouped form was cramped and most controls sat below a non-obvious scroll (and
showed a confusing "< Settings" back arrow). Add SettingsWindow (same standalone
NSWindow pattern as the Editor/Templates windows) and open it from the menu-bar
"Settings…" button. Drop the now-unused NavigationStack and the 320pt cap so the
form uses real window width with normal macOS spacing; window is resizable.
 2026-06-08 13:57:24 -05:00
Grant Gilliam c81bdc4cba Make adapter toggles actually gate screen-reading 
The Settings "Adapters" toggles wrote adapterEnabled but nothing in the capture
path ever read it, so flipping one off did nothing — and the caption still said
"Inert in Phase 0". The adapters (Zoom/Teams/Signal/Meet) are all live now.

SessionController.startVisual now skips visual capture when the detected app's
adapter is toggled off (records audio-only; transcription still runs). Update the
section caption to describe the real behavior.
 2026-06-08 13:30:31 -05:00
Grant Gilliam 836b930083 Surface "Your name" as its own top Settings section 
The name field was the first row of the third "Transcription" section, below
the fold — users couldn't find where to set their name (it's the setting that
labels the mic channel and reserves the name so the LLM never assigns it to
another speaker). Move it into a dedicated "Your name" section at the top of
Settings, and show an orange nudge while it's still the placeholder "Me"/empty.
 2026-06-08 13:25:33 -05:00
Grant Gilliam 217639f12e Collapse adjacent same-speaker segments after reconciliation 
Fragments reabsorbed by smoothFragments (e.g. "I" then "need to switch it
back") were left as separate transcript lines. Add SpeakerReconciler.mergeAdjacent
to join consecutive same-speaker segments within 2s, concatenating their text.

Wire it into SessionController.finishBackend AFTER reconcile/LLM naming. The
collapse needs no LLM, so finishBackend no longer early-returns when the gateway
has no chat model — it runs the collapse and re-persists speakers.json
unconditionally, gating only the reconcile and recap passes on the model.
 2026-06-08 13:19:05 -05:00
Grant Gilliam b4bf88ed2c Chunk overlap + overlap-aware stitching 
Chunks were contiguous (start = prev end) with a naïve offset-concat stitch — no
overlap. That cut sentences at boundaries, denied the diarizer context at edges, and
let one voice split across chunks (the MH/Unknown_0 problem). Now each ~150s body is
sliced with a 15s margin on both sides ([bodyStart-15, bodyEnd+15]); the stitcher
keeps a segment only in the chunk that owns its MIDPOINT (body region) and drops it
from the neighbour's margin — so boundary-spanning speech is seen whole by the
backend and kept exactly once.

- SessionPackager.PlannedChunk gains bodyStart/bodyEnd; planChunks adds overlapSeconds.
- TranscriptAssembler.ChunkResult carries body bounds (defaults keep-all → no-overlap
  behaviour preserved for existing callers); assemble dedups by midpoint-in-body.
- TranscriptPipeline passes body bounds through.

Complements (doesn't replace) the fragment-smoothing + reconciliation safety nets;
this is the upstream fix. ~+20% backend audio per interior chunk. 63/63 XCTest
(new: overlap window layout + boundary-segment dedup).
 2026-06-08 13:03:56 -05:00
Grant Gilliam a25ef16a2a Fix mis-attributed fragments + LLM naming guardrails + re-process saved sessions 
Investigating Grant's real 38-min group call: 'Marty' was a GARBAGE cluster (192
segs, 0.37s mean, 186 ≤2 words, 125 single words flanked by the same other speaker —
diarization micro-fragments split mid-sentence, then LLM-named 'Marty'). Same for
'Message'/'HI'.

- SpeakerReconciler.smoothFragments: dissolve non-self clusters whose MEDIAN segment
  duration ≤ 1s (≥3 segs) — reassign each fragment to the temporally-nearest real
  speaker. (Median, not max, so one stray long segment can't rescue a fragment
  cluster — the bug in the first cut.) On the real call: 7 speakers (3 junk) → 4 real
  (Marty/Message/HI absorbed into Grant/Jonathan/Me/MH). Runs before LLM naming.
- LLM naming guardrails: forbid assigning the self name or ANY already-taken name to
  another voice (fixes 'Grant' = the user's name pinned on a remote speaker); prompt
  demands self-intro / direct-address evidence (mention ≠ presence), 'precision over
  coverage', one name per speaker.
- Open saved session now offers Open Editor vs Re-process, so newer logic can be
  applied to past calls (+ always-visible progress from the prior fix).

NOTE: the self-name guardrail needs the app to KNOW the user's name — selfName is still
'Me', so set it in Settings (e.g. 'Grant') so the LLM can't reuse it. 62/62 XCTest.
 2026-06-08 12:45:17 -05:00
Grant Gilliam c8e7a27150 Open saved session: visible progress + clear errors (no silent no-op) 
The status line only rendered inside the last-in-memory-session block, so 'Open
saved session' processed invisibly — looked like nothing happened. Now: the
transcript status (with a spinner) is always shown, the processing(0,0) reconcile
phase reads 'Working… (this can take a few minutes)', and invalid picks surface an
alert (not a recorded session / already processing / unreadable transcript) instead
of doing nothing.
 2026-06-08 12:16:52 -05:00
Grant Gilliam 3e3a1dea2e Ring-based speaker attribution (fixes real large-tile detection) 
Real Teams/Signal frames exposed a geometry bug: estimating a tile's SIZE from its
name width (×3) produces a tiny box on big real tiles, so the speaking border ring
fell entirely outside it → zero points → 'not speaking' (Joe Payne's clear blue
border went undetected). Pure nearest-name fails too (the top edge of a lower tile
is closer to the upper tile's bottom-anchored name).

Fix: cluster the highlight pixels into connected RINGS (GridCallAnalyzer.connectedComponents,
spatial-hashed union-find), then attribute each ring to the OCR'd name inside its
bounding box. The ring *is* the tile, so detection is independent of tile-size
estimation, and multiple simultaneous borders (lag/persist/crosstalk) become separate
rings naturally — exactly the multi-ring case Grant flagged. minRingSpan rejects specks.

Validated on real frames: Teams now detects 'Joe Payne' (was empty); Signal detects
'JA' in the group grid. (Signal _002 has a border but no rendered name that frame —
inherent Signal intermittency; voice + reconciliation cover it.) 59/59 synthetic
XCTest still green (white + coloured, single + crosstalk).
 2026-06-08 12:03:36 -05:00
Grant Gilliam 1e4ab30ab7 Speaker reconciliation + open/re-process any saved session 
Reconciliation (the marry-the-signals layer): after transcription, before the recap,
SpeakerReconciler (1) MERGES non-self clusters whose voiceprints are highly similar
(cosine >= 0.82) — fixes a person split across chunks (the real 1-on-1 failure: one
remote came back as 'MH' + 'Unknown_0'); and (2) NAMES remaining non-self clusters
from transcript CONTENT via the gateway LLM (people addressed by name / self-intros),
conservative + confidence-gated, keeping the placeholder when unrevealed. The
mic-channel self is protected and never reassigned. Voice does the segmentation; the
fingerprint-merge fixes splits; the LLM adds the content signal visual/voiceprint lack.

- SpeakerReconciler: pure cosine merge (tested) + LLM content-naming pass; rewrites
  speakers.json before recap. SessionController.finishBackend shares one model lookup
  for reconcile + recap. Gated by settings.reconcileSpeakers (default on).
- Open saved session: menu 'Open saved session…' → folder picker. Edits it if already
  transcribed, else reconstructs inputs from disk (visual_timeline vision segs +
  channel self-spans) and runs transcribe → reconcile → recap, then opens the editor.
  Lets you evaluate/correct ANY past call, not just the in-memory last one.

Note (from real Signal data): visual naming is unreliable on Signal (sparse, misread
initials, lowercase/center names) — so reconciliation + the editor (which teaches
voiceprints on confirm) carry it; the editor remains the human arbiter. 59/59 XCTest.
 2026-06-08 11:54:41 -05:00
Grant Gilliam 625c25cfc8 gitignore example-screenshots/ (personal call frames) 2026-06-08 11:01:34 -05:00
Grant Gilliam eb8a15b82e Configurable recap templates (categories per meeting type, in Settings) 
Takeaways categories are no longer hardcoded — they're editable templates. A
template = the always-on TLDR + an ordered list of sections, each with a title, a
type (attributed items / bulleted list / paragraph), and an instruction (the prompt
text for that category). The analyzer assembles the LLM prompt FROM the template
and parses generically, so adding/removing/renaming a category needs zero code and
the output always renders.

- RecapTemplate / TemplateSection / SectionKind + TopicGranularity; built-in
  defaults (Internal Meeting, 1:1, Company/Sales Call), all editable.
- Generic extras: RecapExtras{tldr, primarySpeakers, sections:[RenderedSection]} +
  RecapItem{text,who,when,note} replaces the fixed MeetingExtras. Analyzer builds
  per-section sec_N fields + parses by kind; renderer + remap are generic.
- Topic granularity (coarse/auto/fine) answers 'should chunking be configurable' —
  it scales the target topic count; raw window sizes stay as tuned defaults.
- AppSettings persists templates + defaultTemplateId (seeded once). Settings gets a
  default-template picker + 'Manage…' → TemplatesView (CRUD, edit sections/
  instructions, set default, **Preview prompt** for full transparency).
- Recap editor gains a template picker; Regenerate uses the chosen template. Auto
  recap uses the default template.

54/54 XCTest (template prompt build, generic parse/remap/render updated).
 2026-06-06 19:26:03 -05:00
Grant Gilliam 174fa91e48 Recap editor: Regenerate recap (re-run LLM on corrected transcript) 
Adds a 'Regenerate recap' action so corrected speaker names flow into freshly
written summaries/extras (not just find-replaced). regenerate() commits the
corrections (rewrite speakers.json + reconcile voiceprints), re-runs RecapAnalyzer
on the corrected transcript via the gateway LLM, and rewrites recap.json +
transcript.md + recap.html. save() and regenerate() share commitCorrections();
both rebaseline the speaker set afterward so further edits map cleanly. Editor view
gains the button + progress spinner; RecapEditModel takes the gateway baseURL/skipTLS.

52/52 XCTest; builds clean.
 2026-06-06 16:48:18 -05:00
Grant Gilliam c481c0103a Speaker corrections: rename / merge / reassign + voice learning 
Native editor to fix speaker-ID errors after transcription (modeled on recap-relay's
correction UX): rename a speaker in the legend, merge two speakers, or reassign an
individual transcript line. Saving rewrites speakers.json, re-renders transcript.md +
recap.html, and updates the voiceprint memory — so a correction compounds: naming an
"Unknown" speaker teaches that voice for future calls.

- SpeakerEditing (pure, tested): replaceSpeaker (rename = merge-onto-existing),
  reassign, netNameMap (compose ops), and remap (apply a name map to a recap's
  structured fields + whole-word free text, so summaries/extras update without re-LLM).
- RecapEditModel (@MainActor): loads speakers.json (+ optional recap.json +
  cluster_fingerprints.json); on save writes the resolved speakers.json, re-renders,
  and reconciles voiceprints — merge keeps the survivor's print; rename/name-an-Unknown
  enrolls the cluster's fingerprint under the new name.
- TranscriptEditorView (SwiftUI) + EditorWindow (AppKit window for the LSUIElement app);
  menu gains "Edit speakers".
- Pipeline now persists cluster_fingerprints.json (every cluster incl. Unknown) and
  recap.json (RecapFile) so the editor can learn voices + re-render offline.
- RecapModels made Codable; TranscriptAssembler exposes allFingerprints;
  VoiceprintStore gains enroll() + merge().

52/52 XCTest (6 new, incl. a full rename→artifacts→voiceprint round-trip on disk).
 2026-06-06 15:12:23 -05:00
Grant Gilliam 9240718e5e Recap: readable transcript + topic sections + meeting extras (gateway LLM) 
New 'Recap' phase — turns speakers.json into a human-readable recap, leveraging
recap-relay's proven logic/prompts but calling the Spark gateway's OpenAI-compatible
/v1/chat/completions directly (same host/TLS as label-merge; Qwen3-35B). We start
from already-named speakers (label-merge), so recap-relay's speaker clustering +
name-inference are skipped entirely.

- GatewayLLMClient: /v1/chat/completions (JSON mode), model discovery via
  /api/endpoints, TLS-skip reuse, 503 retry, sequential.
- RecapAnalyzer: speakers.json → numbered [N] (MM:SS) Name: text transcript →
  time-windowed analyze (single window for short calls, 18min/2min overlap for long)
  → stitch/dedup topic sections → meeting extras (TLDR/decisions/action_items/
  open_questions/key_quotes). Defensive JSON parsing of LLM output.
- RecapRenderer: writes transcript.md + a self-contained dark-theme recap.html
  (topic sections w/ collapsible transcripts, extras panels, speaker color chips,
  full timestamped speaker-attributed transcript, print styles).
- SessionController.buildRecap: best-effort after speakers.json (gated by
  settings.recapEnabled); surfaces recapURL → menu 'Open recap'. Skips silently if
  the gateway has no LLM. Settings toggle added.

Validated END-TO-END on the real Meet session against the live gateway: dual-channel
transcription → 3 topic sections + accurate TLDR + key quotes; 'Go Bitcoin'
correctly attributed to the remote speaker. 46/46 XCTest (10 new).
 2026-06-06 14:36:18 -05:00
Grant Gilliam cecc853dd8 Client: dual-channel label-merge (mic_file + system_file) 
The backend shipped dual-channel mode; wire the client to it. We already capture
mic (you) and system (others) separately, so send them as two files instead of the
mono mix — fixing the misattribution at the source.

- SparkControlClient: labelMergeDual(mic_file, system_file, self_name, self_vad);
  multipart generalized to N files; shared POST/retry/decode extracted.
- SessionPackager.rebasedSelfVadData: chunk-local [{start,end}] for self_vad;
  sliceAudio reused for both tracks.
- TranscriptPipeline.process: dual-channel chunking (slice mic+system, rebase
  timeline + self_vad per chunk) when system audio is healthy; mono mixed-file
  fallback (self folded into the timeline) otherwise.
- VisualCapture.finish: write the full visual_timeline.json (remote + self merged)
  but return REMOTE (vision) segments only — self travels via the mic channel.
- TranscriptAssembler: rank mic_channel highest (the user's own track wins).
- VoiceprintStore: store the clean mic_channel self voiceprint.
- SessionController: pass mic/system URLs + remote timeline + channel self-spans +
  self_name + systemHealthy; self_vad.json now reflects the channel-verified spans.

Validated END-TO-END against the live backend on the real misattributing session:
'Go Bitcoin' (remote) is now attributed to Unknown_0, NOT the user; the user's own
lines come back source=mic_channel; per-channel ASR recovered fuller remote text.
36/36 XCTest (4 new: self_vad rebase, mic_channel ranking + voiceprint storage).
 2026-06-06 13:15:29 -05:00
Grant Gilliam 71f0a3b74e Channel-verified self identity: the mic track is you 
Grant's insight + proven on real session audio: we capture self (mic) and others
(system) as separate tracks, then throw the separation away by mixing to mono — so
the backend has to re-guess who's who. Analysis of a real call showed the channels
are cleanly separated (envelope corr 0.015, NO echo); Caitlyn's 'Go Bitcoin' was
11.8x louder in system than mic, yet the mono mix + noisy visual named it 'Grant'.

ChannelSelfVAD marks self-speech as windows where the mic is active AND louder than
system (mic > system x1.5). Benefits: (1) self is identified by CHANNEL, not by the
on-screen name — set one name in Settings, no per-platform matching; (2) a remote
speaker (or room echo) can never be mislabeled as self. Computed at finalize from
the two finished WAVs; the live capture path is untouched. Falls back to mic-VAD if
tracks can't be read. SessionController feeds these spans to the backend timeline.

Validated on the real session: 16 self spans; 'Go Bitcoin' (72-74s) correctly
EXCLUDED, Grant's 49.9-53.3s / 62.6-64s correctly INCLUDED. 33/33 XCTest (5 new).
 2026-06-06 12:24:29 -05:00
Grant Gilliam e6955425dc Filter OCR to participant-name labels (kill visual-timeline noise) 
Real Meet capture revealed the visual pipeline was treating ALL on-screen text as
participant names: meeting URL, clock, 'Add others' button, lobby 'Your meeting's
ready' dialog, 'Joined as …@gmail.com', etc. 46 of 52 'visual segments' in a real
session were phantom speakers. (The backend was unaffected — it diarizes from audio
and ignores names that match no voice cluster — but the visual_timeline.json and the
segment count were junk.)

GridCallAnalyzer.isLikelyName now gates OCR strings to things shaped like a name:
2–30 chars, 1–3 Title-Cased alphabetic words, no digits/URL/email/glyph punctuation.
Errs toward dropping (a missed name just loses a hint; audio diarization still runs).
Unit-tested against the EXACT 19 OCR strings from the real session: keeps the 5
real names, drops all 14 chrome strings. 28/28 XCTest.
 2026-06-06 12:01:57 -05:00
Grant Gilliam dba306c900 Fix Signal (Electron) call detection: resolve mic-using helper to its app 
Signal 1:1 (and group) calls didn't auto-record. Root cause confirmed on-device:
Signal is Electron and holds the mic in a HELPER process
(org.whispersystems.signal-desktop.helper.Renderer, a child of the main app).
detectViaMicAttribution only matched PIDs listed in NSWorkspace.runningApplications
against the main bundle ID, so the helper's mic use was never attributed to Signal.
(Zoom worked = single native process; Meet worked = browser resolved.)

Fix: iterate the mic-using PIDs and resolve each to its owning app by walking the
parent-process chain (sysctl KERN_PROC_PID → ppid) until an NSRunningApplication is
found. Helper PIDs that return nil directly now resolve to the main app. Validated
against the live Signal helpers: pids 2383/2372 → org.whispersystems.signal-desktop.
Superset of the old behavior, so Zoom/Meet detection is preserved (browser case now
also more robust); our own recording is still skipped (selfPID).
 2026-06-06 11:50:58 -05:00
Grant Gilliam 11b50907e3 Per-platform colour-border sensitivity (Teams violet, Meet glow) 
Cross-platform research (Grant) flagged that the colour-border cue differs by app;
checking the real brand colours against the detector found a concrete bug: the
global 0.5 saturation threshold MISSES Teams' violet ring (#6264A7 ≈ 0.41, light
variants ~0.27) entirely and Meet's lighter blue glow (#8ab4f8 ≈ 0.44). Those
adapters would have detected nothing.

- FrameSampler.saturatedPoints: add a tunable threshold + optional hue-band gate
  (degrees) so a lowered threshold doesn't pick up warm video.
- GridCallAnalyzer.Config: colorSaturation / colorMinBrightness / colorHueRange,
  plumbed to the colour-border path (defaults preserve prior behaviour).
- MeetAdapter sat→0.35 (catch the glow); TeamsAdapter sat→0.22 + hue 215–275°
  (catch the faint violet, reject other colours); ZoomAdapter sat 0.45 + hue
  40–150° (vivid green/yellow). Values are first-pass pending real-fixture
  calibration; the hue gate is the main calibration lever.

Tests: Teams now detects the faint violet ring and rejects a green one; Meet/Zoom
vivid cases still pass. 27/27 XCTest.
 2026-06-06 10:51:12 -05:00
Grant Gilliam 9d789645b6 Surface whether visual capture ran on the last session 
Visual capture falls back to audio-only silently, so the user couldn't tell if
it attached on a real call. SessionInfo now carries visualSegmentCount (nil =
audio-only; a count = visual ran, with that many vision-detected speaker
segments), shown in the menu as '… · N visual segments' or '… · audio-only'.
Makes the pending live-call validation unambiguous.
 2026-06-06 10:21:44 -05:00
Grant Gilliam a52a296122 Wire visual capture into the recording lifecycle (failure-isolated) 
Visual capture now runs alongside audio: on call start the session picks the
app's adapter, captures the call window on the SAME monotonic clock as the audio
(AudioRecorder.sharedT0Host), and on stop writes visual_timeline.json and hands
the backend the visual segments with mic-VAD self-spans merged. Any visual
failure (no adapter, no window, Screen Recording denied) leaves the session
recording audio-only — the proven path is never blocked or broken.

- CallDetector now emits DetectedCall{app, bundleID, windowID}: the exact
  CGWindowID of the matched Meet browser window (native apps → nil → largest).
- VisualCapture wraps VisualObserver + AdapterRegistry, writes visual_timeline.json.
- AudioRecorder.sharedT0Host() exposes the shared t0 for frame alignment.

Hardened per a 3-lens adversarial review (concurrency / failure-isolation /
data-flow), all 6 confirmed findings fixed:
- P0 (critical): startVisual could adopt a stale capture into a DIFFERENT session
  (cross-session SCStream leak + visual_timeline.json written to the wrong
  folder). Now gated on session identity — generation + recorder ===, still
  .recording — with fail-closed adoption; otherwise the stream is cancelled.
- P1: observer captured the browser's largest window, not the detected Meet
  window. Now targets the exact CGWindowID (pickWindowIndex, unit-tested),
  largest-area only as fallback.
- P2: a startVisual orphaned by a concurrent stop could leak a stream on quit.
  inFlightVisual is registered before the await and drained in prepareForTermination.
- P3: trailing visual gap/segment ends could exceed duration_sec. Clamped in
  VisualCapture (clampSegments/clampGaps, unit-tested).
- P4: capture pixel size used NSScreen.main scale; now uses the scale of the
  display actually hosting the window (OCR clarity on secondary displays).
- VisualObserver.stop() bounds stopCapture() with a 3s timeout (mirrors audio) so
  a wedged stream can't hang finalization.

25/25 XCTest pass. Live validation on real calls still pending.
 2026-06-06 10:18:52 -05:00
Grant Gilliam dd3a5ce33d Adapters: add Meet, Zoom, Teams (coloured border) + adapter registry 
Front-loads the remaining visual adapters per the Signal→Meet→Zoom priority.
All three reuse GridCallAnalyzer's coloured-border (saturated) detection path
and share the new bottom-left name anchor:

- GridCallAnalyzer: generalise nameAtBottom:Bool into a NameAnchor enum
  (.bottomCenter for Signal's centered footer, .bottomLeft for Meet/Zoom/Teams
  where the name hugs the tile's bottom-left corner, .center for completeness).
  tileRect estimates the tile up-and-right of a bottom-left name.
- MeetAdapter (Google-blue ring, browser-hosted), ZoomAdapter (green/yellow
  border), TeamsAdapter (violet ring): coloured-border on, white-border off,
  bottom-left names. Geometry constants are first-pass pending real fixtures.
- AdapterRegistry.adapter(for:) maps CallDetector.DetectedApp → AppAdapter so
  VisualObserver can be constructed when live visual capture is wired in;
  unmapped apps degrade to audio-only.

Synthetic 4-tile tests: Meet picks each blue-bordered speaker with no
adjacent-tile bleed, Zoom picks the green-bordered speaker, and Signal's
white-only detector correctly ignores a coloured border. 18/18 XCTest pass.
 2026-06-06 09:57:53 -05:00
Grant Gilliam 06bb8233a1 Signal: detect the white speaking border (not a coloured one) 
Signal's active-speaker cue is a 3px #ffffff rounded border (saturation ≈ 0),
which the saturation-based highlight detector could never see. Per the
Signal-Desktop source review:

- FrameSampler.thinWhitePoints: grid-sample near-white pixels that sit on a
  THIN structure (a non-white pixel within edgeGap on some axis) so a border/
  ring counts but a solid white blob (face, bright video) does not.
- GridCallAnalyzer: combine coloured (saturated) + white (thin) highlight
  pixels; exclude name-text regions so the white footer name can't be mistaken
  for the border; estimate the tile UP from the name footer (nameAtBottom);
  attribute each highlight pixel to exactly one tile by containment (nearest
  centre as tiebreak) so a border can't bleed into an adjacent tile.
- SignalAdapter: white border on, coloured off, name-at-bottom geometry.

Synthetic 4-tile harness now isolates each speaker with no adjacent-tile bleed;
all 15 XCTest cases pass. Real-screenshot geometry calibration still pending.
 2026-06-06 09:52:10 -05:00
Grant Gilliam c46d7f81d8 Phases 2-6: detection, visual timeline, backend hand-off, voiceprints 
Phase 2 (call detection): CallDetector using CoreAudio per-process mic
attribution (anarlog technique) — robust start+stop for Zoom/Teams/Signal/Meet,
ignoring our own recording; auto-record toggle. Built; pending live multi-app
confirmation by the user.

Phase 3 (visual timeline foundation): AppAdapter protocol + SpeakerObservation,
TimelineBuilder (hysteresis/overlap/self-merge/aliases), VisualTimeline (schema
1.1), TextRecognizer (Vision OCR), FrameSampler + GridCallAnalyzer (name OCR +
saturated-highlight active-speaker attribution), SignalAdapter, VisualObserver
(window capture; frames released, never saved; minimized->visual_gap, idle != gap).
Synthetic-frame tested; adapter geometry pending real Signal fixtures + live
VisualObserver validation.

Phase 5 (backend hand-off): SparkControlClient (multipart label-merge, sequential,
TLS-skip, 503 Retry-After/413), SessionPackager (chunk plan + WAV slice + timeline
slice/rebase), TranscriptAssembler + SpeakersFile, TranscriptPipeline. Validated
END-TO-END against the live backend (chunk -> label-merge -> speakers.json).

Phase 6 (voiceprints): VoiceprintStore (known_voiceprints, persist named
fingerprints, skip Unknown). Wired: 'Send to backend' button + transcript status,
auto-send toggle (default off) + self-name setting.

All adversarial-review findings fixed. App + XCTest suite build; tests pass.
 2026-06-06 00:15:49 -05:00
Grant Gilliam 9218af0c45 Phase 1: dual-track audio capture → mixed-mono 16 kHz WAV + mic VAD 
AudioRecorder captures system audio (ScreenCaptureKit) + mic (AVAudioEngine) on a
single serial ioQueue, one shared monotonic t0, time-driven writers (pad gaps /
trim overlaps) so tracks stay aligned, and an energy mic-VAD for 'self' spans.
AudioMixer sums the aligned tracks into mixed_mono_16k.wav. SessionController
drives a serialized start/stop state machine, writes the session folder +
self_vad.json, exposes live level meters, and finalizes on quit.

Hardening from review: ioQueue single-domain (no races), stop() never hangs
(mic-first teardown + bounded stopCapture), layout-agnostic mic deep-copy,
discard-only video output to keep SCStream alive, VAD lockstep on committed
frames, stable signing team in project.yml, single-instance enforcement.
 2026-06-05 21:30:11 -05:00
Grant Gilliam 22447627c6 Phase 0: menu-bar scaffold, permissions, backend health check 
Native SwiftUI menu-bar app (LSUIElement, macOS 13+), generated from project.yml
via XcodeGen. Includes:
- PermissionsManager (Microphone / Screen Recording / Accessibility) + UI
- SparkControlHealth: GET /api/status over self-signed TLS (InsecureTrustDelegate)
- AppSettings persistence (host, TLS-skip, output folder, adapter toggles)
- Menu-bar panel + Settings, app sandbox & hardened runtime off (LAN tool)
 2026-06-05 19:33:53 -05:00
26 changed files with 150 additions and 661 deletions
Expand all files Collapse all files
.claude/settings.json
-1
View File
@@ -1 +0,0 @@
{}
.gitignore
+1 -12
View File
@@ -23,15 +23,4 @@ Config/Signing.xcconfig
# Local env files (e.g. SPARK_BACKEND_URL for dev/harness runs) — never commit
.env
.env.*
!.env.example
# Claude Code — deny by default, allow-list shared wiring.
# .claude/ also accumulates worktrees, editor configs, and OS cruft; commit
# only the shared parts so new local scratch (or a stray secret) stays out.
.claude/*
!.claude/rules/
!.claude/agents/
!.claude/commands/
!.claude/skills/
!.claude/settings.json
.env.local
AGENTS.md
+14 -21
View File
@@ -2,14 +2,12 @@
Native macOS **menu-bar app** that detects video calls, records dual-track audio + watches the call window for active-speaker cues, and sends audio + a visual timeline to a self-hosted **SparkControl** backend that does transcription/diarization/naming — producing named transcripts and recaps.
> **Inbox check:** At session start, if `~/Projects/standards/INBOX.md` exists, scan it for items tagged `(ten31-transcripts)` and surface them before proposing next steps; triage with `/triage`.
## Stack (versions that matter)
- **Swift 5.0**, **SwiftUI** + AppKit, macOS **13.0** deployment target. `LSUIElement` (menu-bar only, no Dock icon).
- Project is generated by **XcodeGen** from `project.yml` (`brew install xcodegen`). `*.xcodeproj` is **gitignored** — regenerate, don't edit.
- Full Xcode lives at `/Applications/Xcode.app`, but `xcode-select` points at CommandLineTools → **set `DEVELOPER_DIR` for every `xcodebuild`**.
- Bundle id `xyz.ten31.transcripts`; `DEVELOPMENT_TEAM` (Apple Team ID) is set in a **gitignored `Config/Signing.xcconfig`** (copy `Config/Signing.xcconfig.example` and set your team). Keep it stable — a constant signing identity is what preserves TCC grants across rebuilds.
- Backend: SparkControl gateway at `$SPARK_BACKEND_URL` (a private LAN backend — IP or `.local` host; Start9 self-signed cert. Install the StartOS Root CA in the System keychain so normal TLS validation succeeds; skip-TLS is an opt-in, **host-scoped** escape hatch, **off by default** — see `InsecureTrustDelegate`). Resolution order: a value saved in **Settings → SparkControl backend** (UserDefaults) wins, else the `SPARK_BACKEND_URL` env var, else the placeholder default in `AppSettings.swift`. Diarization = Sortformer/TitaNet (**mono-only**, ~4 speakers/chunk); LLM = Qwen3 via OpenAI-compatible `/v1/chat/completions`; audio via `/api/audio/label-merge`.
- Backend: SparkControl gateway at `$SPARK_BACKEND_URL` (a private LAN `.local` host; self-signed cert, so TLS-skip is intentional). Resolution order: a value saved in **Settings → SparkControl backend** (UserDefaults) wins, else the `SPARK_BACKEND_URL` env var, else the placeholder default in `AppSettings.swift`. Diarization = Sortformer/TitaNet (**mono-only**, ~4 speakers/chunk); LLM = Qwen3 via OpenAI-compatible `/v1/chat/completions`; audio via `/api/audio/label-merge`.
## Commands
First time on a machine — create the local signing config (else `xcodegen generate`/signing won't find a team):
@@ -46,24 +44,22 @@ open /Applications/Ten31Transcripts.app
## Layout (day one)
- `Ten31Transcripts/App/` — `@main` entry + `AppDelegate`.
- `Ten31Transcripts/Session/` — `SessionController` (state machine), `TranscriptPipeline`, `SessionPackager` (chunking), `TranscriptAssembler`, `SpeakerReconciler`, `ChunkPlan` (`ChunkMode`), `SpeakersFile`, `SessionNaming` (pure folder-name + recap-title logic).
- `Ten31Transcripts/Session/` — `SessionController` (state machine), `TranscriptPipeline`, `SessionPackager` (chunking), `TranscriptAssembler`, `SpeakerReconciler`, `ChunkPlan` (`ChunkMode`), `SpeakersFile`.
- `Ten31Transcripts/Visual/` — `VisualCapture`/`VisualObserver` (ScreenCaptureKit, ~3fps), `GridCallAnalyzer` (+ `FrameSampler`, `TextRecognizer`, `TimelineBuilder`, `VisualTimeline`, `SpeakerObservation`).
- `Ten31Transcripts/Adapters/` — per-app screen-readers (`MeetAdapter`, `ZoomAdapter`, `TeamsAdapter`, `SignalAdapter`) + `AdapterRegistry`.
- `Ten31Transcripts/Audio/` — `AudioRecorder`, `MicVAD`, `ChannelSelfVAD`, `AudioMixer`, `MonoTrackWriter`, `Resampler`.
- `Ten31Transcripts/Audio/` — `AudioRecorder`, `MicVAD`, `ChannelSelfVAD`.
- `Ten31Transcripts/Backend/` — `SparkControlClient`, `GatewayLLMClient`, `VoiceprintStore`, `SparkControlHealth`, `InsecureTrustDelegate` (TLS skip).
- `Ten31Transcripts/Recap/` — `RecapAnalyzer`, `RecapRenderer` (writes `transcript.md` + `recap.html`), `RecapModels`, `RecapTemplate`, `SpeakerEditing`, `RecapEditModel`.
- `Ten31Transcripts/{Detection,Permissions,Settings,UI,Support}/` — `CallDetector`/`AudioInputProcesses`/`MicActivityMonitor`; `PermissionsManager`; `AppSettings` (UserDefaults); SwiftUI views + AppKit window hosts; `Info.plist` + entitlements.
- `Ten31Transcripts/{Detection,Permissions,Settings,UI,Support}/` — `CallDetector`; `PermissionsManager`; `AppSettings` (UserDefaults); SwiftUI views + AppKit window hosts; `Info.plist` + entitlements.
- `Ten31TranscriptsTests/` — XCTest. `example-screenshots/` — real fixtures (gitignored). `docs/`, `README.md`.
- **Runtime output** (default `~/Ten31Transcripts/sessions/<ts>_<app>/`, configurable in Settings): `mic.wav`, `system.wav`, `mixed_mono_16k.wav`, `self_vad.json`, `visual_timeline.json`, `speakers.json` (output), `cluster_fingerprints.json`, `recap.{html,json}`, `transcript.md`. The folder is created at session start as `<yyyy-MM-dd'T'HH-mm-ss>_<app>`; on stop the user can name the meeting and it's renamed to `<date>_<name>_<app>` (skipping keeps the auto stamp).
- **Runtime output** (default `~/Ten31Transcripts/sessions/<ts>_<app>/`, configurable in Settings): `mic.wav`, `system.wav`, `mixed_mono_16k.wav`, `self_vad.json`, `visual_timeline.json`, `speakers.json` (output), `cluster_fingerprints.json`, `recap.{html,json}`, `transcript.md`.
## Conventions
- Match the surrounding file's style; small reviewable diffs; comments explain **why**, not what.
- Write/extend XCTest alongside non-trivial changes; pure logic (chunking, reconciliation, analyzer math) is unit-tested offline.
- Commits: imperative mood, concise; authored by Grant. Push to the self-hosted Gitea remote `origin` (branch `main`, over SSH) after committing, with my approval; the remote URL lives in `.git/config`, kept out of source. Work on `main` — don't create feature branches unless I ask.
- **Gitea push gotcha:** `origin`'s URL uses a raw `.local` mDNS host that intermittently fails to resolve (`Could not resolve hostname`, or a push that connects then stalls). The `gitea-home` SSH alias (in `~/.ssh/config`) points at the **same** Gitea server (port 59916, user `git`) via a reliable HostName — the sibling `standards` repo uses it. Reliable fallback: `git push gitea-home:grant/ten31-transcripts.git main` then `git update-ref refs/remotes/origin/main main`. Repointing `origin` to the alias would make this permanent (not yet done).
- Commits: imperative mood, concise; authored by Grant. **No remote is configured** — confirm where to push (choosing one is tracked in `ROADMAP.md`). Branch before committing; never commit to `main` without asking.
- Never commit recordings, transcripts, screenshots, or the generated `*.xcodeproj`.
- No API keys/tokens/passwords in the repo. The backend host (`$SPARK_BACKEND_URL`) and the Apple Team ID (`Config/Signing.xcconfig`, gitignored) are kept out of source — real values live in Settings/UserDefaults and the local xcconfig. Build env vars: `DEVELOPER_DIR` (required) and optional `SPARK_BACKEND_URL`.
- **Git history scrubbed (2026-06-13):** the private backend host + LAN IP were purged from all commits via `git filter-repo` (replaced with the `your-spark-backend.local` placeholder) and force-pushed; 0 hits across refs. Pre-rewrite backup bundle: `../ten31-transcripts-prehistory-rewrite.bundle`. A **second rewrite the same day** purged two backend LAN IPs that had slipped into a docs/test commit, replacing them with RFC 5737 documentation IPs (`192.0.2.1`/`192.0.2.2`) and force-pushing; 0 hits across refs; backup bundle `../ten31-transcripts-pre-ip-scrub.bundle`. The Apple Team ID was intentionally **not** scrubbed (it's public in every signed binary) — don't re-flag it.
## Always
- Set `DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer` on every `xcodebuild`.
@@ -81,16 +77,13 @@ open /Applications/Ten31Transcripts.app
- Never do per-platform display-name matching for self (Zoom/Meet/Signal names differ) — channel + one canonical name only.
- Never treat a solid camera-off avatar tile (Meet's orange/magenta fill) as an active speaker — the real cue is a thin **hollow** coloured ring; require thin-edge + hue gate (see `GridCallAnalyzer.isHollow`, `FrameSampler.thinColoredPoints`).
- Never collapse adjacent same-speaker transcript segments (reverted by request) — one line per diarized utterance.
- Never let a session-folder name put the meeting name where the app label is parsed from: the app must stay the **last** `_`-segment (`SessionController.appLabel(from:)` reads `.split("_").last`; `SessionNaming` enforces this and disambiguates collisions on the name segment). Renames happen at `finish()`-time after files are closed — re-derive track URLs from the (possibly moved) folder, never from `RecordingResult`'s start-time paths.
- Never send call audio to a raw IP the user didn't configure. Offline backend checks: a `.local` mDNS host can't be resolved by a plain `swiftc`/URLSession binary (`-1009`) — use the **real app** or `curl`; but a **configured raw IP _is_ reachable from a plain swiftc URLSession binary** (that's how the TLS fix was verified offline).
- Never force-push a shared branch, and never push without my approval. (Work on `main` — don't create feature branches unless I ask.)
- Never send call audio to a raw IP the user didn't configure. The backend host (`$SPARK_BACKEND_URL`) is a private `.local` mDNS name a plain `swiftc` binary can't resolve via URLSession (`-1009`) — use the **real app** for backend runs (or `curl` for health checks).
- Never commit to `main` or force-push a shared branch; branch first and ask.
## Current state
Present tense; overwritten each session. `main` clean and pushed (HEAD `a5c227e`, pushed via the `gitea-home` alias — origin's `.local` host wouldn't resolve); `/Applications/Ten31Transcripts.app` rebuilt + installed from HEAD. **Full suite re-run: 91 pass** (was 73; +18 `SessionNamingTests`).
- **This session (2026-06-17) — meeting-name prompt + folder rename:** on stop, an NSAlert asks for a meeting name (Save/Skip) and the session folder is renamed `<ts>_<app>` → `<date>_<name>_<app>` (HH-MM-SS dropped; Skip/blank keeps the stamp). Pure logic in `SessionNaming` (sanitize, leaf compose, `recapTitle` for both forms); app label stays the last `_`-segment; collisions disambiguate on the name segment; `finish()` re-derives track URLs post-rename; quit never prompts and aborts an open prompt. Reviewer-reviewed; its P1 (quit-during-modal) + two P2s fixed.
- **Backend connected end-to-end:** real LAN URL saved in Settings → SparkControl backend (off-repo: `defaults read xyz.ten31.transcripts backendBaseURL`); committed default stays the placeholder.
- **Working:** backend hand-off (live), call detection (Meet/Zoom/Teams/Signal), dual-track capture, dual-channel + chunked send, speaker reconciliation, recap, speaker editor, configurable chunk length, standalone Settings, meeting-name prompt + readable folders.
- **Verify next (real app):** the naming prompt + rename is unit-tested + builds but **not yet exercised on a live stop** — run a real recording, stop, name it, confirm the folder renames and backend output lands in the renamed folder.
- **Next up:** (a) repoint `origin` to `gitea-home` so pushes stop hitting the flaky `.local` host (see Conventions); (b) **backend URL primary→fallback** + the `mmss()` NaN/∞ guard freebie (sketch first; keep real IPs out of source — use `192.0.2.x`).
- **In progress / unverified:** the Meet visual fix (reject solid camera-off tiles) still has no clean end-to-end run — re-process the saved Meet session + a fresh Meet call (needs real app + backend).
- **Known bugs / loose end:** sparse Meet speaking-detection (faint blue border); sub-second junk "self" mic fragments; desktop-mic vs phone doesn't unify by voiceprint. Doc loose end: `docs/01 §5`/`docs/02 §2.4` still list "AppleScript" as a Meet name source though the code uses window titles.
Present tense; overwritten each session. 69 tests pass; `/Applications/Ten31Transcripts.app` matches HEAD and runs.
- **Working:** call detection (Meet/Zoom/Teams/Signal), dual-track capture, dual-channel + chunked backend hand-off, speaker reconciliation, recap (`transcript.md` + recap-relay-styled `recap.html`), speaker editor, configurable chunk length, standalone Settings window.
- **In progress:** the Meet visual fix (reject solid camera-off tiles) is unverified end-to-end — no clean run exists yet; the saved Meet session's `visual_timeline.json` predates the fix.
- **Decided but not implemented:** none open (deferred items live in `ROADMAP.md`).
- **Known bugs:** Meet speaking-detection is sparse (faint blue border); the mic channel emits some sub-second junk "self" fragments; the same person on desktop-mic vs phone-speakerphone does not unify by voiceprint.
- **Next:** (1) re-process the saved Meet session in the app, then read its `speakers.json` + `cluster_fingerprints.json` to confirm ~4 speakers recover; (2) confirm Settings → Your name = "Grant"; (3) record a fresh Meet call to validate the fix on a clean capture; (4) decide a git remote and push.
EVALUATION.md
-57
View File
@@ -1,57 +0,0 @@
# Evaluation — ten31-transcripts — 2026-06-13
Intent: A native macOS menu-bar app (Swift/SwiftUI/AppKit, macOS 13+, generated by XcodeGen) that auto-detects conference calls (Meet/Zoom/Teams/Signal), records dual-track audio while watching the call window via ScreenCaptureKit for active-speaker cues, and hands audio + a visual speaker timeline to a self-hosted SparkControl backend that performs transcription/diarization/speaker-naming — producing named transcripts and recaps.
Agents run: evaluator, security-auditor, exerciser, doc-auditor. Skipped: start9-spec-checker (no StartOS-wrapper markers found), reviewer (working tree clean — no diff to review).
## Verdict
This is a genuinely well-engineered personal tool: it builds cleanly with the documented `DEVELOPER_DIR` toolchain, all 69 tests pass in ~1s (claim verified empirically), the architecture is disciplined (the app records/watches/packages/reconciles and correctly delegates all ML to the backend), and secrets hygiene is verifiably clean — the documented 2026-06-13 history scrub survives a full-ref grep with zero leaked hosts or IPs. The headline risk is the TLS trust model: certificate validation is bypassed **globally for any host, on by default**, so anyone on the LAN can MITM the full upload of call audio, the visual timeline, and stored voiceprints — and the same bypass makes a reproducible recap-time crash (`mmss()` on a malformed `Double`) attacker-reachable. The second-largest issue is documentation: the README still describes "Phase 0 (scaffold)" for an app that has shipped through Phase 6, and the `docs/` specs have diverged from the dual-channel API and the recap phase. Code-wise this is close to ready for its single-user, LAN-only purpose; the fixes are well-scoped and mostly small. Fix the TLS model first — it gates the safety of every backend-integration test that follows.
## Cross-referenced findings
- **TLS bypass scope — contradiction resolved against the evaluator.** The evaluator rated Security 4 partly on the basis that the TLS-skip is "intentional/scoped" (`InsecureTrustDelegate.swift:9`). The security-auditor read the implementation and found it is **not** scoped: `URLCredential(trust:)` is returned for any host without a host/fingerprint/CA check (`InsecureTrustDelegate.swift:22`), and it is default-on (`AppSettings.swift:109`). The auditor's direct evidence wins; the Security lens is adjusted down accordingly (see Scorecard).
- **One attack chain, two agents.** The exerciser independently reproduced (twice) a fatal crash in `RecapAnalyzer.mmss()` on `Double.nan`/`Double.infinity` (`RecapAnalyzer.swift:137`), reachable when the backend returns e.g. `"duration": 1e400`. The security-auditor's P1 global TLS bypass is exactly what lets an on-LAN attacker *be* that backend. These are not two unrelated findings — the P1 bypass converts the P2 crash from "trust the backend" to "any LAN attacker can crash the app at recap time." Listed once each below, but they share an exploit path.
- **README staleness — corroborated by two agents.** Both the evaluator (P2) and the doc-auditor (multiple lines) independently flagged that `README.md` describes Phase 0 while the code is at Phase 6+, and both flagged the matching stale source comment at `AppSettings.swift:7`. Merged into one finding; the doc-auditor adds that the drift extends into the `docs/` design specs.
- **Test count — claim verified, not just asserted.** The evaluator and exerciser both built and ran the suite; "69 tests pass" (AGENTS.md) is confirmed by execution, not by counting `func test` declarations.
## Priority queue
- [P1] Global, unscoped TLS bypass trusts any certificate from any host (default-on) — anyone on the LAN can ARP/DNS-spoof the unauthenticated `.local` mDNS name and receive the full mic+system audio, visual timeline, and voiceprints, then return attacker-chosen transcripts — `InsecureTrustDelegate.swift:22`, wired at `SparkControlClient.swift:85`/`GatewayLLMClient.swift:36`/`SparkControlHealth.swift:35` — security-auditor
- [P2] Skip-TLS defaults to ON, so the P1 MITM window is open from first launch before any user choice — `AppSettings.swift:109` (`... as? Bool ?? true`) — security-auditor
- [P2] `RecapAnalyzer.mmss()` fatally crashes on NaN/±Infinity (reproduced twice); a malformed/MITM'd backend `duration` decodes to `Double.infinity` and aborts the app at recap-render time — `RecapAnalyzer.swift:137` (`Int(sec.rounded())`) — exerciser (exploit path opened by the P1 finding)
- [P2] README is stale by six phases — claims "Phase 0 (scaffold)… no audio capture, call detection, screen reading, or backend hand-off yet" for an app that has all of it; the same lie is in source comment `AppSettings.swift:7` — `README.md:7,49,51,56-66` vs. `Ten31Transcripts/{Audio,Detection,Visual,Session,Recap}/` — evaluator + doc-auditor
- [P2] `SessionController` (670 lines, the most concurrency-dense file: generations, in-flight task adoption, pending-auto-stop) has zero unit tests, while comparable pure logic is well covered — `SessionController.swift:256-282` — evaluator
- [P3] `docs/` design specs drifted from the implemented backend path: the dual-channel fields (`mic_file`/`system_file`/`self_name`/`self_vad`) are undocumented and the recap/LLM phase is absent — `docs/03_DATA_CONTRACTS.md:109-116`, `docs/02_ARCHITECTURE.md:51,197`, `docs/01_PROJECT_BRIEF.md:31,83,94`, `docs/04_BUILD_PLAN.md` (no recap phase) vs. `SparkControlClient.swift:106-130` / `RecapAnalyzer.swift:8-12` — doc-auditor
- [P3] `docs/01_PROJECT_BRIEF.md:142-153` §7 lists open items 2–5 (send trigger, retention, voiceprint-update policy, signing) that are already resolved in code — `AppSettings.swift:46`, `VoiceprintStore.swift:25`, `Config/Signing.xcconfig` — doc-auditor
- [P3] `docs/02_ARCHITECTURE.md:214-216` §2.10 claims MenuBarUI features (recent-sessions list with resend/delete, voiceprint manager) that are absent from the actual UI (`MenuBarView` surfaces only the single last session) — doc-auditor
- [P3] AGENTS.md Layout listings are incomplete: `Audio/` omits `AudioMixer`/`MonoTrackWriter`/`Resampler`, `Detection/` omits `AudioInputProcesses`/`MicActivityMonitor` — `AGENTS.md:50,53` — doc-auditor
- [P3] The `manifest.json` per-file `sha256` integrity contract is specified but never written by the pipeline — spec-vs-reality gap — `docs/03_DATA_CONTRACTS.md:61-63` — evaluator
- [P3] Env-var precedence footgun: a saved UserDefaults backend URL permanently shadows `SPARK_BACKEND_URL`, so the env var silently has no effect once Settings is touched (already noted in ROADMAP) — `AppSettings.swift:105-107`, `ROADMAP.md:23` — evaluator
- [P3] `SessionController` owns three jobs — recording state machine, backend-processing orchestration, and the saved-session/NSOpenPanel UI flow; extract the open/reprocess UI before the file grows — `SessionController.swift:467-535` — evaluator
- [P3] Unused, scary-looking `NSAppleEventsUsageDescription` entitlement string ("reads the active browser tab's URL") with no AppleEvents code path (Meet detection uses `CGWindowListCopyWindowInfo` titles only) — drop it — `Info.plist:33` — security-auditor
- [P3] Backend is unauthenticated by design — any LAN device that reaches it can drive transcription; consider a shared bearer token even on LAN — `docs/03_DATA_CONTRACTS.md:89` — security-auditor
- [P3] App Sandbox OFF + Hardened Runtime OFF (intentional, required for cross-app observation) leaves the app unconfined; keep the zero-dependency posture as a deliberate compensating control and document it as such — `project.yml:38` + entitlements — security-auditor
## Scorecard
The evaluator's six-lens table, with two lenses adjusted where another agent's evidence contradicts the evaluator's stated basis (adjustments noted):
| Lens | Score /5 | Notes |
|---|---|---|
| Architecture | 5 | Clean layering; ML delegated to backend per intent; pure/testable seams split from I/O. The single 670-line `SessionController` is the only concentration (P3 to extract). |
| Security | **3** (was 4) | **Adjusted down.** The evaluator's "TLS-skip is intentional/scoped" basis is contradicted by the security-auditor's read: the bypass is global/any-host (`InsecureTrustDelegate.swift:22`) and default-on. Otherwise strong — zero deps, no shell-out, verified-clean secrets, the "never write frames" privacy claim holds in code. |
| Performance | 5 | Idles near-zero; frames released immediately; grid-sampled vision with reused `CIContext`; sequential backend calls honor the single-GPU constraint. |
| Testing | 4 | 69 tests pass (verified by execution); they target the real load-bearing logic. Gap: the `SessionController` concurrency state machine is untested. |
| Code quality | 5 | Consistent style, comments explain *why*, zero warnings, no `try!`. One latent robustness ding: the `mmss()` NaN/∞ fatal (P2). |
| Documentation | **3** (was 4) | **Adjusted down.** The evaluator scored 4 calling `docs/` "excellent and true," but the doc-auditor's claim-by-claim pass found drift well beyond the README — the dual-channel API and the entire recap phase are undocumented across `docs/01-04`, and the build plan never mentions recap. |
## Disagreements & gaps
- **TLS scope (resolved).** Evaluator said "scoped" and scored Security 4; security-auditor read `InsecureTrustDelegate.swift:22` and found it global + default-on (P1). Resolved in favor of the auditor's direct evidence; Security adjusted to 3.
- **Documentation breadth (resolved).** Evaluator sampled `docs/` and judged them accurate (lens 4); doc-auditor did a claim-by-claim pass and found material drift in the specs, not just the README. Resolved in favor of the doc-auditor for the lens; adjusted to 3.
- **Shared blind spot (all runtime-capable agents).** None could exercise live end-to-end behavior — the SparkControl `.local` backend is unreachable from any of these environments by design, and the real on-call visual-cue accuracy needs the gitignored `example-screenshots/`. The Meet visual fix (reject solid camera-off tiles) therefore remains **unverified end-to-end**, which AGENTS.md "Current state" itself acknowledges. No agent could close this; it requires a real call on the user's machine.
## Suggested order of work
1. **Fix the TLS trust model first** — scope the override to the configured backend host and pin the Start9 root CA (or the leaf SPKI hash); default skip-TLS to `false`. This is the P1, and it is the precondition that makes any later backend-integration test trustworthy (it currently gates the P2 crash's reachability).
2. **Harden `Double`→`Int` conversions on backend-decoded values** — give `mmss()` a finite-guard fallback and audit sibling call sites; closes the recap-time crash chain that step 1 also narrows.
3. **Rewrite `README.md` to match the shipped app** and fix the `AppSettings.swift:7` "Phase 0" comment — the single highest-leverage doc change (first thing any newcomer reads).
4. **Reconcile the `docs/` specs** — document the dual-channel fields in `docs/03` §4 and `docs/02`, add the recap phase to `docs/01/02/04`, and close the already-resolved §7 open items.
5. **Add `SessionController` state-machine tests** (auto-start-then-immediate-call-end via `pendingAutoStop`; the visual-adoption generation guard) — do this *before* the next refactor so it has a safety net.
6. **Then extract the saved-session/open-panel UI** out of `SessionController` into a small coordinator.
7. **Run one real call end-to-end** on the user's machine to validate the unverified Meet visual fix and confirm `speakers.json` + `transcript.md` + `recap.html` are written correctly — only meaningful after step 1 makes that path safe.
README.md
+41 -113
View File
@@ -1,146 +1,74 @@
# Ten31 Transcripts
Native macOS menu-bar app that auto-detects conference calls, records dual-track
audio while watching the call window for active-speaker cues, and hands the audio
plus a visual speaker timeline to a self-hosted **SparkControl** backend that does
the transcription, diarization, and speaker naming — producing named transcripts
and meeting recaps.
Native macOS menu-bar app that auto-detects conference calls, records local audio,
builds a visual-derived speaker timeline, and hands audio + timeline to the
SparkControl backend for naming/transcription. See `docs/` for the full spec.
It runs as a menu-bar-only app (no Dock icon). All machine-learning work lives on
the backend; the app only records, watches, packages, and reconciles hints.
## How it works
1. **Detect** — a call in Google Meet, Zoom, Teams, or Signal starts; `CallDetector`
notices and (optionally) auto-starts a session.
2. **Record + watch** — dual-track audio (your mic + system output) is captured while
`ScreenCaptureKit` samples the call window (~3 fps) to read names and spot the
active speaker. Video frames are analyzed in memory and released immediately —
**never written to disk**.
3. **Package + send** — audio is chunked and sent to the backend, dual-channel
(`mic_file` + `system_file`) when the system track is healthy, else a mono mix.
The visual timeline rides along as naming hints. Backend calls are sequential
(one in flight) to respect the single-GPU backend.
4. **Transcribe + name** — the backend diarizes (Sortformer/TitaNet) and an LLM
(Qwen3, via an OpenAI-compatible endpoint) assigns names, helped by the visual
hints and your stored voiceprints.
5. **Reconcile + recap** — the app reconciles speaker hints, then writes a readable
`transcript.md` and an HTML `recap.html`. A built-in speaker editor lets you fix
names after the fact.
**You** are identified by the mic channel plus the single name in *Settings → Your
name* — that name is reserved so the LLM never assigns it to anyone else. (There's
no per-platform display-name matching; your Zoom/Meet/Signal names can all differ.)
This repo is at **Phase 0** (scaffold, permissions, backend health check).
## One-time setup
1. **Install Xcode** from the Mac App Store (free; large download). Open it once and
1. **Install Xcode** from the Mac App Store (free; ~40 GB). Open it once and
accept the license prompt.
2. **Install XcodeGen** (generates the Xcode project from `project.yml`):
```sh
brew install xcodegen
```
3. **Set your signing team.** The Apple Team ID is kept out of source in a gitignored
`Config/Signing.xcconfig`. Copy the template and set your team:
3. **Set your signing team.** The Apple Team ID is kept out of source in a
gitignored `Config/Signing.xcconfig`. Copy the template and set your team:
```sh
cp Config/Signing.xcconfig.example Config/Signing.xcconfig # then set DEVELOPMENT_TEAM
```
`xcodegen` wires it in via `configFiles`, so **Signing & Capabilities** shows the
team automatically. Keep the value stable so macOS preserves the app's permission
(TCC) grants across rebuilds. Edit the xcconfig, not Xcode — `xcodegen generate`
overwrites Xcode-side changes.
4. **Generate the project** (re-run any time you add/remove/rename a source file):
team automatically — no manual selection. Keep the value stable so macOS
preserves the app's permission (TCC) grants across rebuilds. Edit the xcconfig,
not Xcode — `xcodegen generate` overwrites Xcode-side changes.
4. **Generate the project:**
```sh
xcodegen generate
```
This creates `Ten31Transcripts.xcodeproj` (gitignored — regenerate, don't edit).
This creates `Ten31Transcripts.xcodeproj` (git-ignored — regenerate any time).
5. **Open it:**
```sh
open Ten31Transcripts.xcodeproj
```
6. Press **Run** (⌘R).
## Build & run
> **Note:** after adding files in a new phase, re-run `xcodegen generate` and let
> Xcode reload the project. The signing team persists because it lives in
> `Config/Signing.xcconfig` (gitignored), so macOS permissions stay granted across
> rebuilds.
The simplest path is to open `Ten31Transcripts.xcodeproj` and press **Run** (⌘R).
## What Phase 0 does
To build a standalone app and install it (Xcode doesn't need to stay open) — note the
`DEVELOPER_DIR` prefix: full Xcode lives at `/Applications/Xcode.app` but
`xcode-select` may point at the Command Line Tools, so set it on **every**
`xcodebuild`:
- Launches as a menu-bar-only app (no Dock icon).
- Menu panel shows live status for the three permissions it needs — **Microphone**,
**Screen Recording**, **Accessibility** — with Grant / Open Settings buttons.
- Shows a **backend health check** (`GET /api/status`) against the configured host.
- **Settings:** backend base URL, skip-TLS toggle (on by default for the
self-signed cert), output folder, and adapter toggles (inert this phase).
```sh
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcodebuild \
-project Ten31Transcripts.xcodeproj -scheme Ten31Transcripts \
-configuration Release -derivedDataPath /tmp/ten31-release build
ditto /tmp/ten31-release/Build/Products/Release/Ten31Transcripts.app /Applications/Ten31Transcripts.app
open /Applications/Ten31Transcripts.app
```
The installed copy does **not** auto-update — rebuild and `ditto` again after changes.
Run the test suite:
```sh
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcodebuild test \
-project Ten31Transcripts.xcodeproj -scheme Ten31Transcripts \
-destination 'platform=macOS' -derivedDataPath /tmp/ten31-dd
```
## Permissions
The menu panel shows live status for the three permissions the app needs, each with
Grant / Open Settings buttons:
- **Microphone** — to record your side of the call.
- **Screen Recording** — to capture system audio and watch the call window.
- **Accessibility** — to read window/participant information.
## Backend setup
Point the app at your SparkControl backend in **Settings → SparkControl backend**.
The resolution order is: the value saved in Settings (UserDefaults) wins, else the
`SPARK_BACKEND_URL` env var, else a neutral placeholder default. The committed
default is only a placeholder (`https://your-spark-backend.local`) — your real LAN
URL lives in Settings and never touches source.
The backend sits behind a Start9 self-signed Root CA. The supported path is to
**install the StartOS Root CA in your System keychain**, after which normal TLS
validation succeeds. *Skip TLS verification* is an opt-in escape hatch, **off by
default** and **scoped to the configured backend host** — it never becomes
"trust any server."
## Output
Each session writes to `~/Ten31Transcripts/sessions/<timestamp>_<app>/` (configurable
in Settings):
```
mic.wav system.wav mixed_mono_16k.wav # audio (dual-track + mono mix)
self_vad.json visual_timeline.json # self voice-activity + visual hints
speakers.json cluster_fingerprints.json # reconciled speakers + voiceprints
transcript.md recap.html recap.json # final outputs
```
No audio capture, call detection, screen reading, or backend hand-off yet — those
arrive in Phases 1–6 (`docs/04_BUILD_PLAN.md`).
## Project layout
```
project.yml # XcodeGen recipe → generates the .xcodeproj
project.yml # XcodeGen recipe → generates the .xcodeproj
Ten31Transcripts/
App/ @main entry + AppDelegate
Detection/ CallDetector — which app is in a call
Audio/ dual-track capture, mixing, resampling, self-VAD
Visual/ ScreenCaptureKit capture + grid analysis → speaker timeline
Adapters/ per-app screen-readers (Meet, Zoom, Teams, Signal) + registry
Session/ SessionController state machine, packaging, reconciliation
Backend/ SparkControl + LLM clients, voiceprint store, TLS handling
Recap/ transcript.md + recap.html rendering, speaker editor
Permissions/ Settings/ UI/ Support/ (permissions, AppSettings, views, Info.plist)
Ten31TranscriptsTests/ # XCTest — pure logic (chunking, reconciliation, analyzer math)
docs/ # architecture & data-contract design notes
App/ Ten31TranscriptsApp.swift, AppDelegate.swift
UI/ MenuBarView, SettingsView, PermissionRow
Permissions/PermissionsManager.swift
Backend/ SparkControlHealth.swift, InsecureTrustDelegate.swift
Settings/ AppSettings.swift
Support/ Info.plist, Ten31Transcripts.entitlements
Ten31TranscriptsTests/ # placeholder; real tests land in Phase 3
```
## Notes
- **App Sandbox is off** and **Hardened Runtime is off** — this is a personal,
LAN-only tool that must observe other apps. Revisit only if distributing.
- **Privacy:** video frames are never written to disk; recordings, transcripts, and
screenshots are gitignored and never committed.
- `AGENTS.md` is the canonical reference for build commands, conventions, and current
state; `ROADMAP.md` holds the backlog; `docs/` holds the architecture and
data-contract design notes.
- The backend host is a private LAN address — set it in **Settings**, or seed it
from the `SPARK_BACKEND_URL` env var; the committed default is only a neutral
placeholder (`https://your-spark-backend.local`).
ROADMAP.md
+1 -8
View File
@@ -10,9 +10,6 @@ Longer-term backlog and deferred decisions. Near-term status + the next few step
- 1:1 Signal: audio-pill fallback (no active border ever appears in 1:1).
- Accessibility-tree name source for Electron/Meet (cleaner than OCR); `AppAdapter.namesFromAccessibility` hook exists but returns nil.
## Platform support
- Jitsi: add call detection + a `JitsiAdapter` (Jitsi Meet is browser-based like Google Meet — needs `CallDetector` title recognition, an adapter for participant-name reading, and active-speaker visual cues). New platform alongside Meet/Zoom/Teams/Signal.
## Audio / speakers
- Self mic-channel cleanup: tighten self-VAD / smooth self so sub-second junk "self" fragments stop surviving (self is currently protected from fragment-smoothing).
- Adaptive chunk sizing from the backend's first-chunk speaker count, instead of the visual participant estimate.
@@ -22,13 +19,9 @@ Longer-term backlog and deferred decisions. Near-term status + the next few step
- Constrain recap reading width on very wide windows (long line length in the summary band).
## Tooling / repo
- Decide and configure a git remote (none set); then push.
- Decide whether to add a linter/formatter (SwiftLint/SwiftFormat) — none configured today.
- `SPARK_BACKEND_URL` is read only at `AppSettings.init` and is shadowed by any value already saved in Settings (UserDefaults wins). So once a backend URL has been saved, the env var has no effect — a stale stored value can override it in dev/CI/harness runs. If that bites, treat an empty/placeholder stored URL as absent so the env var can still win.
## Quality / debt (from the 2026-06-13 independent eval — full queue + evidence in `EVALUATION.md`)
- Guard `RecapAnalyzer.mmss()` (`:137`) against NaN/∞ — a malformed backend `duration` aborts the app at recap render (eval P2). Cheap; fold into the next backend change.
- Add `SessionController` state-machine tests (`pendingAutoStop`, visual-adoption generation guard) before refactoring; then extract its saved-session / open-panel UI (eval P2/P3).
- Smaller P3s in `EVALUATION.md`: whether to actually emit the `manifest.json` per-file `sha256` (now documented as not-emitted in `docs/03` §2); unauthenticated LAN backend (consider a bearer token).
## Deferred decisions
- Cross-device self unification (same person, desktop mic vs phone speakerphone) does not work by voiceprint and is treated as a separate identity; revisit only if a reliable signal emerges (mic-channel-as-self remains the robust path).
Ten31Transcripts/App/Ten31TranscriptsApp.swift
+3 -2
View File
@@ -3,8 +3,9 @@ import SwiftUI
/// Menu-bar-only app entry point.
///
/// `LSUIElement` (set in Info.plist) keeps the app out of the Dock; the
/// `MenuBarExtra` scene provides the status-bar item and its panel, which wires
/// up permissions, settings, recording control, and the backend health check.
/// `MenuBarExtra` scene provides the status-bar item and its panel. Phase 0 only
/// wires up permissions, settings, and a backend health check — no audio,
/// capture, or call detection yet.
@main
struct Ten31TranscriptsApp: App {
@NSApplicationDelegateAdaptor(AppDelegate.self) private var appDelegate
Ten31Transcripts/Audio/AudioRecorder.swift
+1 -1
View File
@@ -14,7 +14,7 @@ struct RecordingResult {
let systemNote: String?
}
/// Dual-track local audio capture.
/// Dual-track local audio capture for Phase 1.
///
/// - System audio via `SCStream` (`capturesAudio`); its audio handler runs on
/// `ioQueue`. A discard-only video output runs on `screenQueue` purely to keep
Ten31Transcripts/Audio/MicVAD.swift
+2 -2
View File
@@ -13,8 +13,8 @@ struct VADSpan: Equatable {
/// internal sample cursor always equals the mic file position, and span times
/// land on the same instants as `mixed_mono_16k.wav`.
///
/// `TimelineBuilder` folds these in as high-confidence pre-seeded "self"
/// segments. Thresholds are intentionally simple.
/// Phase 3's `TimelineBuilder` will fold these in as high-confidence pre-seeded
/// "self" segments. Thresholds are intentionally simple and will be tuned later.
///
/// Single-threaded: all calls happen on `AudioRecorder.ioQueue`.
final class MicVAD {
Ten31Transcripts/Backend/GatewayLLMClient.swift
+1 -3
View File
@@ -33,9 +33,7 @@ final class GatewayLLMClient {
config.timeoutIntervalForRequest = 600
config.timeoutIntervalForResource = 900
config.waitsForConnectivity = false
let delegate: URLSessionDelegate? = skipTLS
? InsecureTrustDelegate(allowedHost: URL(string: self.baseURL)?.host)
: nil
let delegate: URLSessionDelegate? = skipTLS ? InsecureTrustDelegate() : nil
self.urlSession = URLSession(configuration: config, delegate: delegate, delegateQueue: nil)
}
Ten31Transcripts/Backend/InsecureTrustDelegate.swift
+6 -29
View File
@@ -1,42 +1,19 @@
import Foundation
/// URLSession delegate that bypasses certificate validation for **one host only**
/// — the configured SparkControl backend.
/// URLSession delegate that trusts the server certificate without validation.
///
/// SparkControl sits behind a Start9 self-signed Root CA on the LAN. The supported
/// path is to install that CA in the System keychain; default trust evaluation then
/// succeeds and this delegate is never used. It exists only as an opt-in escape
/// hatch (the "Skip TLS verification" setting, off by default) for a machine where
/// the CA isn't installed. Even then it trusts a certificate only when the challenge
/// host equals `allowedHost` — a server-trust challenge from any other host falls
/// back to default validation, so the bypass can never become "trust any server".
/// SparkControl sits behind a Start9 self-signed Root CA on the LAN, so default
/// trust evaluation rejects it. This delegate is used **only** when the
/// "Skip TLS verification" setting is on. It trusts any server certificate —
/// acceptable for a personal tool on a trusted local network and nothing else.
final class InsecureTrustDelegate: NSObject, URLSessionDelegate {
/// The single host the bypass is scoped to (the configured backend host). When
/// nil — only reachable via a malformed base URL — the gate never fires and every
/// challenge falls back to default validation: the safe degenerate case.
private let allowedHost: String?
init(allowedHost: String?) {
self.allowedHost = allowedHost
}
/// The security gate: the trust override may fire only for a server-trust
/// challenge whose host matches `allowedHost`. Pure and synchronous so the
/// host-scoping can be unit-tested without fabricating a `SecTrust`; the
/// credential itself is built only when this is true *and* a serverTrust exists.
func allowsTrustOverride(for space: URLProtectionSpace) -> Bool {
guard let allowedHost else { return false }
return space.authenticationMethod == NSURLAuthenticationMethodServerTrust
&& space.host == allowedHost
}
func urlSession(
_ session: URLSession,
didReceive challenge: URLAuthenticationChallenge,
completionHandler: @escaping (URLSession.AuthChallengeDisposition, URLCredential?) -> Void
) {
guard
allowsTrustOverride(for: challenge.protectionSpace),
challenge.protectionSpace.authenticationMethod == NSURLAuthenticationMethodServerTrust,
let serverTrust = challenge.protectionSpace.serverTrust
else {
completionHandler(.performDefaultHandling, nil)
Ten31Transcripts/Backend/SparkControlClient.swift
+1 -3
View File
@@ -82,9 +82,7 @@ final class SparkControlClient {
config.timeoutIntervalForRequest = 600 // diarization can take up to ~600s
config.timeoutIntervalForResource = 900
config.waitsForConnectivity = false
let delegate: URLSessionDelegate? = skipTLS
? InsecureTrustDelegate(allowedHost: URL(string: self.baseURL)?.host)
: nil
let delegate: URLSessionDelegate? = skipTLS ? InsecureTrustDelegate() : nil
self.urlSession = URLSession(configuration: config, delegate: delegate, delegateQueue: nil)
}
Ten31Transcripts/Backend/SparkControlHealth.swift
+4 -6
View File
@@ -1,10 +1,10 @@
import Foundation
import Combine
/// Performs the backend reachability check: `GET {baseURL}/api/status`.
/// Performs the Phase 0 backend reachability check: `GET {baseURL}/api/status`.
///
/// This is a thin slice; the full upload path (label-merge, multipart, sequential
/// queueing, retries) lives in `SparkControlClient`.
/// This is a thin slice — the full `SparkControlClient` (label-merge, multipart,
/// sequential queueing, retries) arrives in Phase 5.
@MainActor
final class SparkControlHealth: ObservableObject {
@@ -32,9 +32,7 @@ final class SparkControlHealth: ObservableObject {
config.timeoutIntervalForRequest = 8
config.waitsForConnectivity = false
let delegate: URLSessionDelegate? = skipTLS
? InsecureTrustDelegate(allowedHost: url.host)
: nil
let delegate: URLSessionDelegate? = skipTLS ? InsecureTrustDelegate() : nil
let session = URLSession(configuration: config, delegate: delegate, delegateQueue: nil)
defer { session.finishTasksAndInvalidate() }
Ten31Transcripts/Session/SessionController.swift
+14 -74
View File
@@ -99,11 +99,6 @@ final class SessionController: ObservableObject {
/// Bumped each time a start/stop Task is spawned (Task is a value type, so this
/// is how `prepareForTermination` detects a newly-spawned transition).
private var lifecycleGeneration = 0
/// The meeting-name prompt currently on screen, if any, so a quit can end it
/// instead of blocking termination on user input (set in `askMeetingName`).
private weak var activeNamingAlert: NSAlert?
/// Set once `prepareForTermination` begins, so we skip the post-stop naming prompt.
private var isTerminating = false
init(settings: AppSettings) {
self.settings = settings
@@ -329,9 +324,6 @@ final class SessionController: ObservableObject {
lifecycleTask = Task {
let result = await recorder.stop()
let visual = await self.stopVisualAndTimeline(result, folder: folder)
// Interactive stop only: ask for a meeting name and give the folder a
// readable name before `finish()` captures it for backend processing.
self.promptMeetingNameAndRename()
self.finish(result, timeline: visual.timeline, selfSpans: visual.selfSpans, visualRan: visual.visualRan)
}
}
@@ -346,18 +338,13 @@ final class SessionController: ObservableObject {
if let folder = currentFolder {
writeSelfSpans(spans: selfSpans, result: result, to: folder)
let visualCount = visualRan ? timeline.count : nil // `timeline` is the remote vision segments
// Re-derive the track URLs from `folder`: a meeting-name rename may have
// moved the session after `result` captured its original paths.
let micURL = folder.appendingPathComponent("mic.wav")
let systemURL = folder.appendingPathComponent("system.wav")
let mixedURL = folder.appendingPathComponent("mixed_mono_16k.wav")
lastSession = SessionInfo(
folder: folder, mixedURL: mixedURL,
folder: folder, mixedURL: result.mixedURL,
duration: result.duration, selfSpanCount: selfSpans.count,
visualSegmentCount: visualCount)
lastProcess = ProcessInputs(
folder: folder, sessionId: folder.lastPathComponent, app: currentLabel,
micURL: micURL, systemURL: systemURL, mixedURL: mixedURL,
micURL: result.micURL, systemURL: result.systemURL, mixedURL: result.mixedURL,
timeline: timeline, selfSpans: selfSpans, selfName: settings.selfName,
systemHealthy: result.systemNote == nil)
}
@@ -432,13 +419,24 @@ final class SessionController: ObservableObject {
guard settings.recapEnabled, !resolved.segments.isEmpty else { return }
let analyzer = RecapAnalyzer(llm: llm, model: model)
guard let result = try? await analyzer.recap(file: resolved, template: settings.defaultTemplate) else { return }
let title = SessionNaming.recapTitle(app: inputs.app, sessionId: inputs.sessionId)
let title = Self.recapTitle(app: inputs.app, sessionId: inputs.sessionId)
try? RecapRenderer.write(file: resolved, result: result, title: title, to: inputs.folder)
try? RecapFile(title: title, result: result).write(to: inputs.folder.appendingPathComponent("recap.json"))
let url = inputs.folder.appendingPathComponent("recap.html")
if FileManager.default.fileExists(atPath: url.path) { self.recapURL = url }
}
/// Friendly recap title, e.g. "Google Meet call — 2026-06-06 11:43".
private static func recapTitle(app: String, sessionId: String) -> String {
let appName = CallDetector.DetectedApp(rawValue: app)?.display ?? app.capitalized
let stamp = sessionId.split(separator: "_").first.map(String.init) ?? sessionId
let parts = stamp.split(separator: "T")
let date = parts.first.map(String.init) ?? ""
let timeBits = parts.count > 1 ? parts[1].split(separator: "-") : []
let time = timeBits.count >= 2 ? "\(timeBits[0]):\(timeBits[1])" : ""
return "\(appName) call — \(date) \(time)".trimmingCharacters(in: .whitespaces)
}
// MARK: - Speaker corrections
/// True once the last session has a transcribed `speakers.json` to correct.
@@ -586,11 +584,6 @@ final class SessionController: ObservableObject {
/// its WAV headers are finalized before the process exits. Handles quit while
/// `.starting` and `.finishing`, not just `.recording`.
func prepareForTermination() async {
isTerminating = true
// If the meeting-name prompt is open, end its modal loop so quit isn't blocked
// waiting on the user — the session keeps its auto timestamped name. (Falls
// back to the user answering the on-screen dialog if the abort isn't serviced.)
if activeNamingAlert != nil { NSApp.abortModal() }
// Cancel any in-flight backend transcription (audio is already saved; the
// user can resend). The pipeline's checkCancellation + defer clean up chunks.
processTask?.cancel()
@@ -656,59 +649,6 @@ final class SessionController: ObservableObject {
return f.string(from: Date())
}
/// Ask the user to name the just-finished recording, then rename its folder to
/// a readable `<date>_<name>_<app>` (dropping the HH-MM-SS auto stamp). Skipping
/// or leaving it blank keeps the timestamped name. Must run BEFORE `finish()` so
/// the renamed folder is what flows to backend processing. The recorder and
/// visual capture have both finished by now, so every session file is closed and
/// the move is safe. Never called from the quit path — we don't block a quit on
/// a prompt.
private func promptMeetingNameAndRename() {
// A quit can begin while we're finishing — don't put a blocking prompt in its
// way; keep the auto timestamped name and let termination drain.
guard !isTerminating, let folder = currentFolder,
let name = askMeetingName() else { return } // nil = skipped / blank
let base = folder.deletingLastPathComponent()
let date = SessionNaming.datePrefix(ofSessionNamed: folder.lastPathComponent)
let fm = FileManager.default
var counter = 0
while counter < 100 {
guard let leaf = SessionNaming.renamedLeaf(
date: date, app: currentLabel, meetingName: name, counter: counter) else { return }
let target = base.appendingPathComponent(leaf, isDirectory: true)
if fm.fileExists(atPath: target.path) { counter += 1; continue } // disambiguate
do {
try fm.moveItem(at: folder, to: target)
currentFolder = target
} catch {
NSLog("Session rename to “\(leaf)” failed: \(error.localizedDescription)") // keep the original folder
}
return
}
NSLog("Session rename: kept “\(folder.lastPathComponent)” — 100 name collisions")
}
/// Modal prompt for a meeting name. Registers the alert so `prepareForTermination`
/// can end it on quit. Returns the trimmed name, or nil if the user skipped, left
/// it empty, or a quit aborted the prompt (caller keeps the auto folder name).
private func askMeetingName() -> String? {
let alert = NSAlert()
alert.messageText = "Name this recording"
alert.informativeText = "Give the meeting a name so its folder is easy to find in your sessions. Leave blank to keep the timestamped name."
alert.addButton(withTitle: "Save") // .alertFirstButtonReturn
alert.addButton(withTitle: "Skip") // .alertSecondButtonReturn
let field = NSTextField(frame: NSRect(x: 0, y: 0, width: 240, height: 24))
field.placeholderString = "Meeting name"
alert.accessoryView = field
alert.window.initialFirstResponder = field
NSApp.activate(ignoringOtherApps: true)
activeNamingAlert = alert
defer { activeNamingAlert = nil }
guard alert.runModal() == .alertFirstButtonReturn else { return nil }
let text = field.stringValue.trimmingCharacters(in: .whitespacesAndNewlines)
return text.isEmpty ? nil : text
}
/// Debug artifact: the channel-verified "self" spans actually sent to the backend
/// as `self_vad` (mic active AND louder than system). Lets us eyeball self detection.
private func writeSelfSpans(spans: [VADSpan], result: RecordingResult, to folder: URL) {
Ten31Transcripts/Session/SessionNaming.swift
-71
View File
@@ -1,71 +0,0 @@
import Foundation
/// Pure helpers for session-folder names. A session folder is created at start
/// with an auto name `<yyyy-MM-dd'T'HH-mm-ss>_<app>`; when the user names the
/// recording on stop it's renamed to `<yyyy-MM-dd>_<name>_<app>` (no HH-MM-SS),
/// which is far easier to scan in `sessions/`. The app label always stays the
/// LAST `_`-separated segment so `SessionController.appLabel(from:)` keeps working
/// even when the meeting name itself contains spaces or underscores.
enum SessionNaming {
/// Filesystem- and parse-safe meeting name: trims, turns path separators into
/// dashes, drops control characters, collapses whitespace runs, removes leading
/// dots (no hidden/`.`/`..` folders), and caps the length. Returns "" if nothing
/// usable is left, which callers treat as "skip the rename".
static func sanitize(_ raw: String) -> String {
var s = raw.trimmingCharacters(in: .whitespacesAndNewlines)
// Path-hostile separators (`/` and the classic Mac `:`, plus `\`) → dash.
s = s.components(separatedBy: CharacterSet(charactersIn: "/:\\")).joined(separator: "-")
// Strip control characters outright.
s = s.components(separatedBy: .controlCharacters).joined()
// Collapse internal whitespace runs to single spaces.
s = s.split(whereSeparator: { $0 == " " || $0 == "\t" }).joined(separator: " ")
while s.hasPrefix(".") { s.removeFirst() }
s = s.trimmingCharacters(in: .whitespaces)
if s.count > 60 { s = String(s.prefix(60)).trimmingCharacters(in: .whitespaces) }
return s
}
/// The date prefix of a session leaf name, e.g. `2026-06-17T09-59-48_signal`
/// → `2026-06-17`. Already-renamed leaves (`2026-06-17_name_signal`) return the
/// same date, so this is safe to call on either form.
static func datePrefix(ofSessionNamed leaf: String) -> String {
let head = leaf.split(separator: "_").first.map(String.init) ?? leaf
return head.split(separator: "T").first.map(String.init) ?? head
}
/// Compose the renamed leaf `<date>_<name>_<app>`. A positive `counter`
/// disambiguates a collision by suffixing the NAME segment (`<name>-2`) so the
/// trailing `_<app>` stays parseable. Returns nil when the name sanitizes to
/// empty (the caller keeps the auto timestamped name).
static func renamedLeaf(date: String, app: String, meetingName: String, counter: Int = 0) -> String? {
let clean = sanitize(meetingName)
guard !clean.isEmpty else { return nil }
let suffix = counter > 0 ? "-\(counter + 1)" : ""
return "\(date)_\(clean)\(suffix)_\(app)"
}
/// Friendly recap title from a session id, understanding both folder forms:
/// `2026-06-06T11-43-02_meet` → "Google Meet call — 2026-06-06 11:43"
/// `2026-06-06_Weekly sync_meet` → "Weekly sync — Google Meet (2026-06-06)"
static func recapTitle(app: String, sessionId: String) -> String {
let appName = CallDetector.DetectedApp(rawValue: app)?.display ?? app.capitalized
var parts = sessionId.split(separator: "_").map(String.init)
if parts.count > 1 { parts.removeLast() } // drop the trailing "_<app>"
let head = parts.first ?? sessionId
let tBits = head.split(separator: "T").map(String.init)
let date = tBits.first ?? head
let time: String = {
guard tBits.count > 1 else { return "" }
let b = tBits[1].split(separator: "-")
return b.count >= 2 ? "\(b[0]):\(b[1])" : ""
}()
let when = [date, time].filter { !$0.isEmpty }.joined(separator: " ")
// Rejoin with "_" — the faithful inverse of split("_") — so a name that
// itself contained underscores survives the round-trip through the folder name.
let name = parts.count > 1 ? parts[1...].joined(separator: "_") : ""
if name.isEmpty {
return "\(appName) call — \(when)".trimmingCharacters(in: .whitespaces)
}
return "\(name) — \(appName) (\(when))".trimmingCharacters(in: .whitespaces)
}
}
Ten31Transcripts/Session/TranscriptPipeline.swift
+2 -2
View File
@@ -121,8 +121,8 @@ final class TranscriptPipeline {
return assembled.speakersFile
}
/// Build the `label-merge` timeline from mic-VAD self spans; the visual
/// adapters' segments are merged in alongside these.
/// Build the `label-merge` timeline from mic-VAD self spans (Phase 1/2). Once
/// the visual adapters land (Phase 3–4), their segments are merged in too.
static func timeline(fromSelfSpans spans: [VADSpan], selfName: String) -> [VisualTimeline.Segment] {
spans.map { .init(start: $0.start, end: $0.end, name: selfName, confidence: $0.confidence, source: "mic_vad") }
}
Ten31Transcripts/Settings/AppSettings.swift
+3 -6
View File
@@ -3,8 +3,8 @@ import Combine
/// User-facing settings, persisted to `UserDefaults`.
///
/// Covers the backend host + TLS handling, output folder, your name, chunk
/// length, per-app adapter toggles, and the auto-record/auto-send/recap flags.
/// Phase 0 scope: backend host + TLS-skip, output folder, and adapter toggles.
/// The adapter toggles persist but do nothing yet (adapters arrive in Phase 3–4).
@MainActor
final class AppSettings: ObservableObject {
@@ -106,10 +106,7 @@ final class AppSettings: ObservableObject {
?? ProcessInfo.processInfo.environment["SPARK_BACKEND_URL"]
?? Self.defaultBackendURL
// Off by default: install the Start9 Root CA in the System keychain and the
// backend's cert validates normally. The bypass is an opt-in escape hatch and,
// when on, is scoped to the configured host (see `InsecureTrustDelegate`).
self.skipTLSVerification = defaults.object(forKey: Keys.skipTLS) as? Bool ?? false
self.skipTLSVerification = defaults.object(forKey: Keys.skipTLS) as? Bool ?? true
self.outputFolderPath = defaults.string(forKey: Keys.outputFolder)
?? "~/Ten31Transcripts"
Ten31Transcripts/Support/Info.plist
+2
View File
@@ -30,6 +30,8 @@
<string>Ten31</string>
<key>NSMicrophoneUsageDescription</key>
<string>Ten31 Transcripts records your microphone during calls to build the local audio track.</string>
<key>NSAppleEventsUsageDescription</key>
<string>Ten31 Transcripts reads the active browser tab's URL to detect Google Meet calls.</string>
<key>NSLocalNetworkUsageDescription</key>
<string>Ten31 Transcripts connects to your SparkControl server on the local network.</string>
<key>NSAppTransportSecurity</key>
Ten31Transcripts/UI/MenuBarView.swift
+1 -1
View File
@@ -173,7 +173,7 @@ struct MenuBarView: View {
private var header: some View {
VStack(alignment: .leading, spacing: 2) {
Text("Ten31 Transcripts").font(.headline)
Text("Setup & status")
Text("Phase 0 · setup & status")
.font(.caption)
.foregroundStyle(.secondary)
}
Ten31Transcripts/Visual/VisualTimeline.swift
+1 -1
View File
@@ -62,7 +62,7 @@ struct VisualTimeline: Codable {
}
/// The flat array `label-merge` wants: `[{start,end,name,confidence}]`,
/// dropping `source`. Slice/rebase to chunk-local seconds happens at chunking time.
/// dropping `source`. Slice/rebase to chunk-local seconds happens in Phase 5.
func flatTimelineData() throws -> Data {
let flat = segments.map { seg -> [String: Any] in
["start": seg.start, "end": seg.end, "name": seg.name, "confidence": seg.confidence]
Ten31TranscriptsTests/InsecureTrustDelegateTests.swift
-35
View File
@@ -1,35 +0,0 @@
import XCTest
@testable import Ten31Transcripts
/// The TLS bypass is an opt-in escape hatch scoped to the configured backend host.
/// These cover the security gate (`allowsTrustOverride`) so a regression can't widen
/// it back to "trust any server". The gate is pure, so no network or SecTrust needed.
final class InsecureTrustDelegateTests: XCTestCase {
private func space(host: String,
method: String = NSURLAuthenticationMethodServerTrust) -> URLProtectionSpace {
URLProtectionSpace(host: host, port: 62419, protocol: "https",
realm: nil, authenticationMethod: method)
}
func testFiresForMatchingHost() {
let d = InsecureTrustDelegate(allowedHost: "192.0.2.1")
XCTAssertTrue(d.allowsTrustOverride(for: space(host: "192.0.2.1")))
}
func testRejectsMismatchedHost() {
let d = InsecureTrustDelegate(allowedHost: "192.0.2.1")
XCTAssertFalse(d.allowsTrustOverride(for: space(host: "evil.example.com")))
}
func testNilAllowedHostNeverFires() {
let d = InsecureTrustDelegate(allowedHost: nil)
XCTAssertFalse(d.allowsTrustOverride(for: space(host: "192.0.2.1")))
}
func testOnlyServerTrustMethodFires() {
// Matching host but a non-server-trust challenge (e.g. HTTP Basic) must not override.
let d = InsecureTrustDelegate(allowedHost: "192.0.2.1")
XCTAssertFalse(d.allowsTrustOverride(
for: space(host: "192.0.2.1", method: NSURLAuthenticationMethodHTTPBasic)))
}
}
Ten31TranscriptsTests/SessionNamingTests.swift
-110
View File
@@ -1,110 +0,0 @@
import XCTest
@testable import Ten31Transcripts
final class SessionNamingTests: XCTestCase {
// MARK: sanitize
func testSanitizeTrimsAndKeepsSpaces() {
XCTAssertEqual(SessionNaming.sanitize(" Weekly Sync "), "Weekly Sync")
}
func testSanitizeReplacesPathSeparators() {
XCTAssertEqual(SessionNaming.sanitize("9/10 standup"), "9-10 standup")
XCTAssertEqual(SessionNaming.sanitize("a:b\\c"), "a-b-c")
}
func testSanitizeCollapsesWhitespaceRuns() {
XCTAssertEqual(SessionNaming.sanitize("board 1:1"), "board 1-1")
}
func testSanitizeStripsLeadingDots() {
XCTAssertEqual(SessionNaming.sanitize("...hidden"), "hidden")
XCTAssertEqual(SessionNaming.sanitize(".."), "")
}
func testSanitizeEmptyForBlankOrWhitespace() {
XCTAssertEqual(SessionNaming.sanitize(""), "")
XCTAssertEqual(SessionNaming.sanitize(" \n\t "), "")
}
func testSanitizeCapsLength() {
let long = String(repeating: "x", count: 200)
XCTAssertEqual(SessionNaming.sanitize(long).count, 60)
}
func testSanitizeStripsControlCharacters() {
XCTAssertEqual(SessionNaming.sanitize("a\u{0000}b\u{001F}c"), "abc")
}
// MARK: datePrefix
func testDatePrefixFromAutoName() {
XCTAssertEqual(SessionNaming.datePrefix(ofSessionNamed: "2026-06-17T09-59-48_signal"), "2026-06-17")
}
func testDatePrefixFromRenamedName() {
XCTAssertEqual(SessionNaming.datePrefix(ofSessionNamed: "2026-06-17_Weekly sync_signal"), "2026-06-17")
}
// MARK: renamedLeaf
func testRenamedLeafBasic() {
XCTAssertEqual(
SessionNaming.renamedLeaf(date: "2026-06-17", app: "signal", meetingName: "Weekly sync"),
"2026-06-17_Weekly sync_signal")
}
func testRenamedLeafAppStaysLastSegment() {
// The meeting name may contain underscores; the app must remain parseable as
// the final "_"-segment (what SessionController.appLabel reads).
let leaf = SessionNaming.renamedLeaf(date: "2026-06-17", app: "meet", meetingName: "q3_planning")
XCTAssertEqual(leaf, "2026-06-17_q3_planning_meet")
XCTAssertEqual(leaf?.split(separator: "_").last.map(String.init), "meet")
}
func testRenamedLeafNilForBlankName() {
XCTAssertNil(SessionNaming.renamedLeaf(date: "2026-06-17", app: "signal", meetingName: " "))
}
func testRenamedLeafCounterDisambiguatesNameSegment() {
// A collision suffixes the NAME, not the whole leaf, so "_app" stays last.
let leaf = SessionNaming.renamedLeaf(date: "2026-06-17", app: "signal", meetingName: "sync", counter: 1)
XCTAssertEqual(leaf, "2026-06-17_sync-2_signal")
XCTAssertEqual(leaf?.split(separator: "_").last.map(String.init), "signal")
}
func testRenamedLeafAppStaysLastAtMaxCollisionDepth() {
// The 100-collision cap is counter 0…99; the app must still parse out last.
let leaf = SessionNaming.renamedLeaf(date: "2026-06-17", app: "signal", meetingName: "q3_sync", counter: 99)
XCTAssertEqual(leaf, "2026-06-17_q3_sync-100_signal")
XCTAssertEqual(leaf?.split(separator: "_").last.map(String.init), "signal")
}
// MARK: recapTitle
func testRecapTitleAutoNamePreservesLegacyFormat() {
XCTAssertEqual(
SessionNaming.recapTitle(app: "meet", sessionId: "2026-06-06T11-43-02_meet"),
"Google Meet call — 2026-06-06 11:43")
}
func testRecapTitleNamedSession() {
XCTAssertEqual(
SessionNaming.recapTitle(app: "meet", sessionId: "2026-06-06_Weekly sync_meet"),
"Weekly sync — Google Meet (2026-06-06)")
}
func testRecapTitleNamePreservesUnderscores() {
// A meeting name with underscores must survive the split/join round-trip.
XCTAssertEqual(
SessionNaming.recapTitle(app: "meet", sessionId: "2026-06-06_q3_planning_meet"),
"q3_planning — Google Meet (2026-06-06)")
}
func testRecapTitleUnknownAppCapitalizes() {
XCTAssertEqual(
SessionNaming.recapTitle(app: "manual", sessionId: "2026-06-06T11-43-02_manual"),
"Manual call — 2026-06-06 11:43")
}
}
docs/01_PROJECT_BRIEF.md
+35 -46
View File
@@ -7,9 +7,9 @@
> returns named transcript segments. A growing **voiceprint library** recovers
> speakers even when the visual cue is missing.
Master context document. Read this first, then `02_ARCHITECTURE.md` and
`03_DATA_CONTRACTS.md`. The SparkControl API is fully specified in
`03_DATA_CONTRACTS.md`.
Master context document. Read this first, then `02_ARCHITECTURE.md`,
`03_DATA_CONTRACTS.md`, `04_BUILD_PLAN.md`. The SparkControl API is now fully
specified — see `03_DATA_CONTRACTS.md` (and the source `AUDIO_API.md`).
---
@@ -20,30 +20,25 @@ A lightweight, always-running **menu-bar app on macOS** that:
1. **Detects** when the user joins a call in Google Meet, Zoom, Microsoft Teams,
or Signal.
2. **Records two local audio tracks** — system audio (everyone else) and the
user's microphone (the user). It sends the backend **dual-channel**
(`mic_file` + `system_file`) when the system track is healthy, falling back to
a **mixed-mono 16 kHz WAV** otherwise.
user's microphone (the user) — and **mixes them to one 16 kHz mono WAV** for
the backend.
3. **Watches the call window** at ~2–4 fps and, per app, reads participant
**names** and the **active-speaker cue**, producing a
`(start, end, name, confidence)` **visual timeline** — its best guess at who
was talking when.
4. **Discards every video frame after extraction.** No video is ever written to
disk. Only audio + the derived timeline persist locally.
5. On call end, **POSTs the audio + the visual timeline (+ the known voiceprint
library) to `POST /api/audio/label-merge`** on SparkControl, which returns
**named, speaker-attributed transcript segments** and a **voiceprint per
speaker**.
5. On call end, **POSTs the mixed audio + the visual timeline (+ the known
voiceprint library) to `POST /api/audio/label-merge`** on SparkControl, which
returns **named, speaker-attributed transcript segments** and a **voiceprint
per speaker**.
6. **Persists the returned voiceprints** keyed by name, so the next call can pass
them as `known_voiceprints` and recover a speaker by voice when the visual cue
is absent (camera off, a bad OCR frame).
7. **Renders the result locally** — a readable `transcript.md` plus an HTML
`recap.html` (topics + meeting extras, generated via the backend's LLM
endpoint), with an in-app editor for fixing speaker names after the fact.
The app's job ends at producing the named transcript and recap from SparkControl's
segments. **All transcription, diarization, name-merge, and LLM analysis happen on
the backend.** Do not build transcription, diarization, or the merge vote in this
app.
The app's job ends at receiving and storing the named segments from SparkControl.
**All transcription, diarization, and the name-merge happen on the backend.** Do
not build transcription, diarization, or the merge vote in this app.
## 2. Why the visual timeline still matters (the core idea)
@@ -73,25 +68,19 @@ few calls the system can name regulars even with cameras off.
**In scope (this app):**
- Call detection for Meet / Zoom / Teams / Signal.
- Dual-track local audio capture; **dual-channel send** (mic + system) with a
mix-to-mono fallback for the backend.
- Dual-track local audio capture + mix-to-mono for the backend.
- Low-fps window capture → OCR (names) + active-speaker cue detection.
- Per-app "adapter" modules encapsulating each app's UI quirks.
- Building the visual timeline; **mic-VAD self-labeling** (the mic track is the
user, so hot-mic spans pre-seed the user's name into the timeline).
- Chunking long calls (~2–3 min) and calling `label-merge` **sequentially**.
- A local **voiceprint store** (persist + replay named voiceprints).
- Storing the backend's named segments and **rendering** them — `transcript.md`
plus an HTML `recap.html` (recap analysis via the backend LLM) — with an in-app
speaker-name editor.
- A minimal menu-bar UI: status, manual start/stop, the last session (reveal,
resend, open recap, edit speakers), adapter toggles, backend host/health,
output folder.
- Storing the backend's named transcript segments locally.
- A minimal menu-bar UI: status, manual start/stop, recent sessions, adapter
toggles, backend host/health, output folder.
**Out of scope (owned by the backend):**
- Transcription, diarization, the name-merge vote, and LLM summarization — these
run on the backend; the app only orchestrates the recap call and renders the
result.
- Transcription, diarization, the name-merge vote, summarization/analysis.
**Explicitly not doing:** saving video; cloud anything. Everything stays on the
operator's LAN.
@@ -102,14 +91,14 @@ operator's LAN.
|---|---|---|
| Language / framework | Native Swift + SwiftUI menu-bar app (`LSUIElement`) | System audio, window capture, Vision all native; one codebase. |
| Audio capture | ScreenCaptureKit (system audio) + AVFoundation (mic) | No virtual audio device; works with headphones; macOS 13+. |
| Backend audio format | **Dual-channel (mic + system)** when the system track is healthy, else **mixed-mono 16 kHz WAV** | Separate tracks let the backend attribute the user's mic channel directly; the diarizer can still split the mono fallback. |
| Backend audio format | **Mixed-mono 16 kHz WAV** | Diarizer separates speakers from one mixed stream; 16 kHz is ideal. |
| Call detection | CoreAudio "mic running somewhere" + known-app / Meet-tab heuristic | Clean live-mic signal + app disambiguation. |
| Speaker naming | **Backend, via `POST /api/audio/label-merge`** | One call does diarize + overlap-vote naming + transcription. No client merge. |
| Identity recovery | **Local voiceprint library** replayed as `known_voiceprints` | Recovers camera-off / OCR-missed speakers by voice; compounds over calls. |
| Self-identity | mic-VAD → pre-seed user's name in timeline | The mic track is the user; gives the backend a strong prior + enrolls the user's voiceprint immediately. |
| Requests | **Sequential, one audio request in flight** | Parallel audio requests trip a backend GPU race (`503 + Retry-After`). |
| Long calls | Chunk ~2–3 min, sequential, stitch via names+voiceprints | Diarizer caps at **4 speakers/chunk**; voiceprints + names unify across chunks. |
| Transport / TLS | `multipart/form-data`, file field `file` (mono) or `mic_file` + `system_file` (dual-channel); self-signed Start9 cert (trust the Root CA — supported default; host-scoped skip-verify is an off-by-default escape hatch); **no auth on LAN** | Matches every other SparkControl endpoint. |
| Transport / TLS | `multipart/form-data`, file field `file`; self-signed Start9 cert (skip verify or trust the Root CA); **no auth on LAN** | Matches every other SparkControl endpoint. |
| Timing | Batch after call (sync endpoints, no polling) | Endpoints are synchronous; no job/poll machinery needed. |
### On forking Hyprnote
@@ -139,25 +128,25 @@ SparkControl, on the operator's Start9 LAN, fronting two DGX Sparks:
- **★ Primary endpoint for this app:** `POST /api/audio/label-merge` — diarize +
name from the visual timeline (+ voiceprint fallback), optionally transcribe,
in one synchronous call.
- **LLM (recap):** Qwen3 via OpenAI-compatible `POST /v1/chat/completions` —
generates the readable recap (topics + meeting extras) from the transcript.
- Health/discovery: `GET /api/status`, `GET /api/endpoints`, `GET /v1/models`.
Full request/response shapes, curl examples, limits, and error formats are in
`03_DATA_CONTRACTS.md`.
## 7. Settled decisions (were open at brief time)
## 7. Remaining open items (small)
1. **Base URL.** A private LAN host — a `.local` mDNS name (preferred over a raw
IP, since it survives IP changes) — configured in Settings or via the
`SPARK_BACKEND_URL` env var, never committed. A neutral placeholder ships as the
default and stays editable in Settings. Service-discovery at `GET /api/endpoints`.
2. **Send trigger.** Auto-send on call end is a setting (`autoSendOnStop`), **off
by default** — the user reviews the session and sends manually unless they opt in.
3. **Retention.** The session folder is kept after a successful hand-off (output
location is configurable); nothing is pruned automatically.
4. **Voiceprint update policy.** Store/refresh the latest high-confidence vector
per name (`02_ARCHITECTURE.md §2.9`); a per-name running average is a possible
later refinement.
5. **Signing.** A stable identity via `Config/Signing.xcconfig` (gitignored) keeps
macOS from re-prompting for permissions on each rebuild.
1. **Base URL — RESOLVED.** A private LAN host — a `.local` mDNS name (preferred
over a raw IP, since it survives IP changes) — configured in Settings or via the
`SPARK_BACKEND_URL` env var, and never committed. Ship a neutral placeholder as
the default; keep it editable in settings. Service-discovery at
`GET /api/endpoints`.
2. **Send trigger** — assume auto-POST on call end; expose a "hold for review"
toggle if the user wants to eyeball the timeline first.
3. **Retention** — keep the session folder after a successful hand-off, or prune
audio and keep only `speakers.json` + voiceprints? Default: keep everything,
user-configurable.
4. **Voiceprint update policy** — overwrite vs running-average a person's stored
voiceprint across calls (see `02_ARCHITECTURE.md §2.9`). Start simple
(store/refresh latest high-confidence), refine later.
5. **Signing** — stable identity so macOS doesn't re-prompt for permissions on
each rebuild.
docs/02_ARCHITECTURE.md
+6 -23
View File
@@ -64,9 +64,6 @@ pattern, the macOS APIs, and the SparkControl integration (now fully specified).
└────────────────┘ └────────────────────┘
```
(After `speakers.json`, a recap phase renders `transcript.md` + `recap.html` via
the backend LLM — see §2.11.)
## 2. Modules
### 2.1 `CallDetector`
@@ -179,10 +176,8 @@ Write the session folder and, if the call is longer than ~3 min, produce a
```
### 2.7 `SparkControlClient`
Deliver to SparkControl. **Primary path = `POST /api/audio/label-merge`**. Sends
**dual-channel** (`mic_file` + `system_file` + `self_name` + `self_vad`) when the
system track is healthy, else the **mono** `file`; always with `timeline`,
`known_voiceprints`, `transcribe=true`.
Deliver to SparkControl. **Primary path = `POST /api/audio/label-merge`** with
`file`, `timeline`, `known_voiceprints`, `transcribe=true`.
- **Sequential only** — one audio request in flight (parallel ⇒ `503 + Retry-After`).
- **Self-signed TLS** — skip verification (`URLSession` delegate trusting the
Start9 cert) or trust the Root CA. **No auth on the LAN.**
@@ -215,22 +210,10 @@ Local persistence of named voiceprints — the compounding-identity layer.
- Editable/clearable from the menu-bar UI (rename, delete a person, reset).
### 2.10 `MenuBarUI` (SwiftUI, `LSUIElement`)
Status (idle / detected / recording / finishing), manual start/stop with live
mic/system level meters, and the **last session** — reveal in Finder, resend
("Send to backend"), open recap, and edit speakers — plus "Open saved session…"
to reprocess an existing folder. Also a **backend host + health check**
(`GET /api/status`), adapter toggles, output folder, and a permissions checklist
(Microphone, Screen Recording, Accessibility). (No multi-session list or
voiceprint-manager UI yet — those are in `ROADMAP.md`.)
### 2.11 Recap (`RecapAnalyzer`, `RecapRenderer`)
After `speakers.json`, the recap phase turns the named transcript into the
human-readable deliverables. `RecapAnalyzer` calls the backend LLM
(`POST /v1/chat/completions`, Qwen3) for topics + meeting extras; `RecapRenderer`
writes `transcript.md` (one line per diarized utterance) and `recap.html` (+ a
`recap.json` sidecar). The in-app speaker editor (`SpeakerEditing` /
`RecapEditModel`) rewrites names across all outputs after the fact. All
language-model work stays on the backend; the app orchestrates and renders.
Status (idle / detected / recording / uploading), manual start/stop, recent
sessions (open folder, resend, delete), adapter toggles, **backend host + a
health check** (`GET /api/status`), output folder, voiceprint manager, and a
permissions checklist (Screen Recording, Microphone, Accessibility).
## 3. macOS frameworks & permissions
docs/03_DATA_CONTRACTS.md
+11 -28
View File
@@ -1,7 +1,7 @@
# Data Contracts — Ten31 Transcripts
Companion to docs 01/02. Defines the files the app produces/stores and the **real
SparkControl contract** (verified against the live backend). The `label-merge`
SparkControl contract** (source of truth: `AUDIO_API.md`). The `label-merge`
endpoint is the app's primary integration point.
---
@@ -69,10 +69,8 @@ When chunking, **slice to the chunk window and rebase to chunk-local seconds**
"app_version": "0.1.0"
}
```
(On the dual-channel path the backend gets `mic.wav` + `system.wav` directly; on
the mono fallback it gets `mixed_mono_16k.wav`. The mic track is the user's known
identity / VAD source. **Note:** the per-file `sha256` fields above are part of the
intended contract but are **not currently emitted** by the pipeline.)
(`mixed_mono_16k.wav` is the one the backend gets; the separate tracks are kept
locally — the mic track is the user's known identity / VAD source.)
---
@@ -85,17 +83,15 @@ intended contract but are **not currently emitted** by the pipeline.)
endpoints in §4–§5 hang off this base. **Make it a setting** so the host can
change, and ship a neutral placeholder (`https://your-spark-backend.local`) as
the default.
- **TLS:** Start9 self-signed Root CA. Supported path: install the Start9 Root CA
into the System keychain (default trust then succeeds). Skip-verification is an
**off-by-default, host-scoped** escape hatch (`InsecureTrustDelegate`, scoped to
the configured backend host), not the default.
- **TLS:** Start9 self-signed Root CA. Either skip verification (`URLSession`
delegate trusting the cert; curl `-k`; `rejectUnauthorized:false`) **or** install
the Start9 Root CA into the trust store.
- **Auth:** **none on the LAN.** No token/key today.
- **Limits:** **200 MB/request** (`413` over); timeouts ~300 s (transcription),
~600 s (diarization). **Send audio requests SEQUENTIALLY** — concurrent audio
trips a GPU FFT race → `503 + Retry-After`.
- **Transport:** `multipart/form-data`. Audio file field is **`file`** on the mono
path, or **`mic_file`** + **`system_file`** on the dual-channel path (bytes, not
base64/path).
- **Transport:** `multipart/form-data`, audio file field name **`file`** (bytes,
not base64/path).
- **All endpoints are synchronous** (no job IDs / polling).
- **Errors:** JSON `{"detail": "..."}`; `400` malformed, `413` too large, `503 +
Retry-After` transient (retry after the interval).
@@ -109,16 +105,11 @@ Diarize + name clusters from the visual timeline (majority temporal overlap),
with voiceprint fallback, optionally transcribed. Synchronous. **Stateless** —
the app owns the timeline and the voiceprint library.
**Multipart fields** — two audio shapes: **mono** (`file`) or **dual-channel**
(`mic_file` + `system_file`, preferred when the system track is healthy):
**Multipart fields:**
| field | required | notes |
|---|---|---|
| `file` | mono path | mixed-mono WAV (the chunk, when chunking) |
| `mic_file` | dual path | the user's mic track (chunk) — attributed to `self_name` |
| `system_file` | dual path | the remote/system track (chunk) |
| `self_name` | dual path | the user's name; the mic channel is attributed to them |
| `self_vad` | no | chunk-local windows where the mic is genuinely the user (active + louder than system) |
| `timeline` | **yes** | flat JSON array `[{"start","end","name","confidence"}]`, chunk-local seconds (§1.1); on the dual path it names only the remote speakers |
| `file` | **yes** | mixed-mono WAV (the chunk, when chunking) |
| `timeline` | **yes** | flat JSON array `[{"start","end","name","confidence"}]`, chunk-local seconds (§1.1) |
| `known_voiceprints` | no | JSON `{"<name>":[192 floats], ...}` from `VoiceprintStore` |
| `transcribe` | no | `"true"` to also return per-segment text (default false) |
| `min_overlap` | no | min fraction of a cluster's time overlapping the winning name (default `0.0`) |
@@ -222,11 +213,3 @@ Loaded → `known_voiceprints` on every `label-merge` call. Updated from respons
`fingerprints` for `visual`/high-confidence `voiceprint` speakers only. Never
stores `Unknown_N`. Update policy (`02 §2.9`): start = store latest with
`overlap_confidence ≥ ~0.8`; consider per-name running mean later.
## 8. Recap outputs (`transcript.md`, `recap.{html,json}`)
After `speakers.json` is assembled, the recap phase renders the human-readable
deliverables: a `transcript.md` (one line per diarized utterance) and an HTML
`recap.html`, backed by a structured `recap.json`. The recap's topic/summary
content is generated by the **backend LLM** (`POST /v1/chat/completions`, Qwen3);
the app owns the rendering and the in-app **speaker-name editor**, which can rewrite
names across `speakers.json`, the transcript, and the recap after the fact.
docs/04_BUILD_PLAN.md
-6
View File
@@ -1,11 +1,5 @@
# Build Plan — Ten31 Transcripts
> **Status: COMPLETE (historical).** Phases 0–6 shipped and the app is in daily
> use; a recap phase (transcript + HTML recap via the backend LLM) was added after
> this plan was written. Kept as the original build log and as the map for the
> "Phase N" references in the code comments. Forward-looking work lives in
> `ROADMAP.md`; current status in `AGENTS.md`.
Companion to docs 01–03. Phased plan for the Claude Code session, each phase with
a demoable milestone. Build in order; the risky/novel work (visual adapters) is
isolated for independent tuning. The SparkControl contract is now known