Phases 2-6: detection, visual timeline, backend hand-off, voiceprints

Phase 2 (call detection): CallDetector using CoreAudio per-process mic attribution (anarlog technique) — robust start+stop for Zoom/Teams/Signal/Meet, ignoring our own recording; auto-record toggle. Built; pending live multi-app confirmation by the user. Phase 3 (visual timeline foundation): AppAdapter protocol + SpeakerObservation, TimelineBuilder (hysteresis/overlap/self-merge/aliases), VisualTimeline (schema 1.1), TextRecognizer (Vision OCR), FrameSampler + GridCallAnalyzer (name OCR + saturated-highlight active-speaker attribution), SignalAdapter, VisualObserver (window capture; frames released, never saved; minimized->visual_gap, idle != gap). Synthetic-frame tested; adapter geometry pending real Signal fixtures + live VisualObserver validation. Phase 5 (backend hand-off): SparkControlClient (multipart label-merge, sequential, TLS-skip, 503 Retry-After/413), SessionPackager (chunk plan + WAV slice + timeline slice/rebase), TranscriptAssembler + SpeakersFile, TranscriptPipeline. Validated END-TO-END against the live backend (chunk -> label-merge -> speakers.json). Phase 6 (voiceprints): VoiceprintStore (known_voiceprints, persist named fingerprints, skip Unknown). Wired: 'Send to backend' button + transcript status, auto-send toggle (default off) + self-name setting. All adversarial-review findings fixed. App + XCTest suite build; tests pass.
2026-06-06 00:15:49 -05:00
parent fd7e1a5907
commit 863136aeec
27 changed files with 2108 additions and 22 deletions
@@ -0,0 +1,72 @@
+import Foundation
+
+/// `visual_timeline.json` (schema 1.1) — the app's primary visual output. Times
+/// are seconds relative to session t0. Segments may overlap (crosstalk).
+struct VisualTimeline: Codable {
+    var schemaVersion = "1.1"
+    let sessionId: String
+    let app: String
+    let adapterVersion: String
+    let t0Unix: Double
+    let durationSec: Double
+    let fpsSampled: Int
+    let selfName: String?
+    let participants: [Participant]
+    let segments: [Segment]
+    let visualGaps: [Gap]
+
+    struct Participant: Codable {
+        let name: String
+        let isSelf: Bool?
+        let aliases: [String]?
+        enum CodingKeys: String, CodingKey {
+            case name
+            case isSelf = "is_self"
+            case aliases
+        }
+    }
+
+    struct Segment: Codable, Equatable {
+        let start: Double
+        let end: Double
+        let name: String
+        let confidence: Double
+        let source: String   // vision | accessibility | fused | mic_vad
+    }
+
+    struct Gap: Codable, Equatable {
+        let start: Double
+        let end: Double
+        let reason: String   // minimized | tab_switched
+    }
+
+    enum CodingKeys: String, CodingKey {
+        case schemaVersion = "schema_version"
+        case sessionId = "session_id"
+        case app
+        case adapterVersion = "adapter_version"
+        case t0Unix = "t0_unix"
+        case durationSec = "duration_sec"
+        case fpsSampled = "fps_sampled"
+        case selfName = "self_name"
+        case participants
+        case segments
+        case visualGaps = "visual_gaps"
+    }
+
+    /// Write the rich `visual_timeline.json`.
+    func write(to url: URL) throws {
+        let encoder = JSONEncoder()
+        encoder.outputFormatting = [.prettyPrinted, .sortedKeys]
+        try encoder.encode(self).write(to: url)
+    }
+
+    /// The flat array `label-merge` wants: `[{start,end,name,confidence}]`,
+    /// dropping `source`. Slice/rebase to chunk-local seconds happens in Phase 5.
+    func flatTimelineData() throws -> Data {
+        let flat = segments.map { seg -> [String: Any] in
+            ["start": seg.start, "end": seg.end, "name": seg.name, "confidence": seg.confidence]
+        }
+        return try JSONSerialization.data(withJSONObject: flat, options: [])
+    }
+}