Phases 2-6: detection, visual timeline, backend hand-off, voiceprints
Phase 2 (call detection): CallDetector using CoreAudio per-process mic attribution (anarlog technique) — robust start+stop for Zoom/Teams/Signal/Meet, ignoring our own recording; auto-record toggle. Built; pending live multi-app confirmation by the user. Phase 3 (visual timeline foundation): AppAdapter protocol + SpeakerObservation, TimelineBuilder (hysteresis/overlap/self-merge/aliases), VisualTimeline (schema 1.1), TextRecognizer (Vision OCR), FrameSampler + GridCallAnalyzer (name OCR + saturated-highlight active-speaker attribution), SignalAdapter, VisualObserver (window capture; frames released, never saved; minimized->visual_gap, idle != gap). Synthetic-frame tested; adapter geometry pending real Signal fixtures + live VisualObserver validation. Phase 5 (backend hand-off): SparkControlClient (multipart label-merge, sequential, TLS-skip, 503 Retry-After/413), SessionPackager (chunk plan + WAV slice + timeline slice/rebase), TranscriptAssembler + SpeakersFile, TranscriptPipeline. Validated END-TO-END against the live backend (chunk -> label-merge -> speakers.json). Phase 6 (voiceprints): VoiceprintStore (known_voiceprints, persist named fingerprints, skip Unknown). Wired: 'Send to backend' button + transcript status, auto-send toggle (default off) + self-name setting. All adversarial-review findings fixed. App + XCTest suite build; tests pass.
This commit is contained in:
@@ -0,0 +1,36 @@
|
||||
import Foundation
|
||||
import CoreGraphics
|
||||
import CoreVideo
|
||||
|
||||
/// One per-frame observation from an app adapter: a participant tile, whether its
|
||||
/// active-speaker cue is showing, and where it is. `name` may be a full name,
|
||||
/// initials (Signal), or "" if unknown. `t` is seconds relative to the session t0.
|
||||
struct SpeakerObservation: Equatable {
|
||||
let name: String
|
||||
let speaking: Bool
|
||||
let bbox: CGRect
|
||||
let confidence: Double // 0…1
|
||||
let t: TimeInterval
|
||||
}
|
||||
|
||||
/// Per-app screen-reading strategy. Each conferencing app gets one implementation
|
||||
/// that knows that app's tile layout, name placement, and active-speaker cue.
|
||||
/// Adapters must be testable offline against still-image fixtures.
|
||||
protocol AppAdapter {
|
||||
static var bundleIDs: [String] { get }
|
||||
var adapterVersion: String { get }
|
||||
var preferredFPS: Int { get }
|
||||
|
||||
/// Analyze one frame; return the speakers visible and whether each is speaking.
|
||||
/// Must process in-memory and never persist the frame.
|
||||
func analyze(frame: CVPixelBuffer, at t: TimeInterval) -> [SpeakerObservation]
|
||||
|
||||
/// Optional: participant names from the app's Accessibility tree (Electron
|
||||
/// apps like Signal expose these), preferred over OCR when available.
|
||||
func namesFromAccessibility() -> [String]?
|
||||
}
|
||||
|
||||
extension AppAdapter {
|
||||
func namesFromAccessibility() -> [String]? { nil }
|
||||
var preferredFPS: Int { 3 }
|
||||
}
|
||||
Reference in New Issue
Block a user