Grant Gilliam
863136aeec
Phase 2 (call detection): CallDetector using CoreAudio per-process mic attribution (anarlog technique) — robust start+stop for Zoom/Teams/Signal/Meet, ignoring our own recording; auto-record toggle. Built; pending live multi-app confirmation by the user. Phase 3 (visual timeline foundation): AppAdapter protocol + SpeakerObservation, TimelineBuilder (hysteresis/overlap/self-merge/aliases), VisualTimeline (schema 1.1), TextRecognizer (Vision OCR), FrameSampler + GridCallAnalyzer (name OCR + saturated-highlight active-speaker attribution), SignalAdapter, VisualObserver (window capture; frames released, never saved; minimized->visual_gap, idle != gap). Synthetic-frame tested; adapter geometry pending real Signal fixtures + live VisualObserver validation. Phase 5 (backend hand-off): SparkControlClient (multipart label-merge, sequential, TLS-skip, 503 Retry-After/413), SessionPackager (chunk plan + WAV slice + timeline slice/rebase), TranscriptAssembler + SpeakersFile, TranscriptPipeline. Validated END-TO-END against the live backend (chunk -> label-merge -> speakers.json). Phase 6 (voiceprints): VoiceprintStore (known_voiceprints, persist named fingerprints, skip Unknown). Wired: 'Send to backend' button + transcript status, auto-send toggle (default off) + self-name setting. All adversarial-review findings fixed. App + XCTest suite build; tests pass.
37 lines
1.4 KiB
Swift
37 lines
1.4 KiB
Swift
import Foundation
|
|
import CoreGraphics
|
|
import CoreVideo
|
|
|
|
/// One per-frame observation from an app adapter: a participant tile, whether its
|
|
/// active-speaker cue is showing, and where it is. `name` may be a full name,
|
|
/// initials (Signal), or "" if unknown. `t` is seconds relative to the session t0.
|
|
struct SpeakerObservation: Equatable {
|
|
let name: String
|
|
let speaking: Bool
|
|
let bbox: CGRect
|
|
let confidence: Double // 0…1
|
|
let t: TimeInterval
|
|
}
|
|
|
|
/// Per-app screen-reading strategy. Each conferencing app gets one implementation
|
|
/// that knows that app's tile layout, name placement, and active-speaker cue.
|
|
/// Adapters must be testable offline against still-image fixtures.
|
|
protocol AppAdapter {
|
|
static var bundleIDs: [String] { get }
|
|
var adapterVersion: String { get }
|
|
var preferredFPS: Int { get }
|
|
|
|
/// Analyze one frame; return the speakers visible and whether each is speaking.
|
|
/// Must process in-memory and never persist the frame.
|
|
func analyze(frame: CVPixelBuffer, at t: TimeInterval) -> [SpeakerObservation]
|
|
|
|
/// Optional: participant names from the app's Accessibility tree (Electron
|
|
/// apps like Signal expose these), preferred over OCR when available.
|
|
func namesFromAccessibility() -> [String]?
|
|
}
|
|
|
|
extension AppAdapter {
|
|
func namesFromAccessibility() -> [String]? { nil }
|
|
var preferredFPS: Int { 3 }
|
|
}
|