Grant Gilliam
a56b47143c
Signal's active-speaker cue is a 3px #ffffff rounded border (saturation ≈ 0), which the saturation-based highlight detector could never see. Per the Signal-Desktop source review: - FrameSampler.thinWhitePoints: grid-sample near-white pixels that sit on a THIN structure (a non-white pixel within edgeGap on some axis) so a border/ ring counts but a solid white blob (face, bright video) does not. - GridCallAnalyzer: combine coloured (saturated) + white (thin) highlight pixels; exclude name-text regions so the white footer name can't be mistaken for the border; estimate the tile UP from the name footer (nameAtBottom); attribute each highlight pixel to exactly one tile by containment (nearest centre as tiebreak) so a border can't bleed into an adjacent tile. - SignalAdapter: white border on, coloured off, name-at-bottom geometry. Synthetic 4-tile harness now isolates each speaker with no adjacent-tile bleed; all 15 XCTest cases pass. Real-screenshot geometry calibration still pending.
39 lines
1.6 KiB
Swift
39 lines
1.6 KiB
Swift
import Foundation
|
|
import CoreVideo
|
|
|
|
/// Signal Desktop adapter. Signal shows avatars/initials with a coloured ring
|
|
/// around the active speaker; names may also be available via the Electron
|
|
/// Accessibility tree (preferred over OCR when we enable it). Geometry/threshold
|
|
/// here are first-pass and will be calibrated against real Signal screenshots.
|
|
struct SignalAdapter: AppAdapter {
|
|
static let bundleIDs = ["org.whispersystems.signal-desktop"]
|
|
let adapterVersion = "signal-0.1.0"
|
|
let preferredFPS = 3
|
|
|
|
private let analyzer: GridCallAnalyzer
|
|
|
|
init() {
|
|
var config = GridCallAnalyzer.Config()
|
|
// Signal's speaking cue is a 3px WHITE rounded border (not coloured); the
|
|
// name is a bottom footer, so the tile extends up from it. Geometry tuned
|
|
// with real fixtures. (Gotchas, per Signal source: NO border in 1:1 calls —
|
|
// fall back to mic-VAD/audio pill — and in Speaker view the large tile is
|
|
// the speaker; both handled at a higher level later.)
|
|
config.nameAtBottom = true
|
|
config.detectWhiteBorder = true
|
|
config.detectColoredBorder = false
|
|
config.tileExpandX = 2.4
|
|
config.tileExpandY = 4.8
|
|
self.analyzer = GridCallAnalyzer(config: config)
|
|
}
|
|
|
|
func analyze(frame: CVPixelBuffer, at t: TimeInterval) -> [SpeakerObservation] {
|
|
analyzer.analyze(pixelBuffer: frame, at: t)
|
|
}
|
|
|
|
// Exposed for fixture/synthetic tests.
|
|
func analyze(cgImage: CGImage, at t: TimeInterval) -> [SpeakerObservation] {
|
|
analyzer.analyze(cgImage: cgImage, at: t)
|
|
}
|
|
}
|