Keysat
8d839e3714
- Add redaction gateway (redaction_gateway.py, redaction/ scrub + tests) - Add embeddings proxy and spark_embed service (Dockerfile + main.py) - Expand audio_proxy with speaker-aware handling; deep_health/health/server updates - Package: configureSparks action + sparkConfig model updates, manifest/main wiring - Docs: AUDIO_API, EMBEDDINGS, REDACTION_GATEWAY; HANDOFF and runbook/known-issues refresh
106 lines
6.0 KiB
Markdown
106 lines
6.0 KiB
Markdown
# Redaction Gateway — `/scrub` + `/rehydrate` (Spark Control v0.16.0)
|
|
|
|
The privacy boundary between sovereign LP data and the Claude API, living at the
|
|
same trusted Spark Control host as `/v1/chat/completions`, `/v1/embeddings`,
|
|
`/v1/rerank`, and `/api/search`. Built to **behavioral parity** with the CRM's
|
|
reference `backend/redaction/scrub.py` — that engine is vendored verbatim into
|
|
Spark Control and its leak test passes here, so `SCRUB_BACKEND=gateway` is a
|
|
drop-in for the in-repo path.
|
|
|
|
## What it is
|
|
|
|
- `POST /scrub` — de-identify an agent's assembled context. Returns placeholder-only
|
|
text (the agent forwards that to Claude) plus an opaque `map_handle`.
|
|
- `POST /rehydrate` — swap the real values back into Claude's placeholder-bearing
|
|
response, locally, for human review.
|
|
|
|
Spark Control does **not** call Claude. It's the scrub/rehydrate transform pair
|
|
plus a server-held pseudonym map.
|
|
|
|
## Contract (matches the handover doc)
|
|
|
|
`POST /scrub`
|
|
```json
|
|
{ "task_id": "...", "actor": "analyst",
|
|
"items": [{"id": "ctx_1", "text": "..."}],
|
|
"known_entities": {"persons": [], "orgs": [], "funds": [], "emails": [], "locations": []},
|
|
"tier1_action": "drop", // or "reject" (fail-closed 422 on any Tier-1)
|
|
"bucket": {"amounts": false, "dates": false},
|
|
"ner": "auto", // "auto" | "rules_only" | "qwen"
|
|
"map_handle": null } // pass to reuse/extend a task's map (stable tokens)
|
|
```
|
|
→ `200 { task_id, map_handle, items:[{id, scrubbed_text, tokens_used}], stats:{tier1_dropped, tier2_tokenized, distinct_entities, descriptive_flags:[{item, span, action}]}, expires_at }`
|
|
- `422 {"error":"tier1_detected","spans":[{item, kinds}]}` when `tier1_action="reject"` and Tier-1 found (kinds only — never the raw value).
|
|
- `422 {"error":"ner_unavailable", ...}` when `ner=auto|qwen` and the local Qwen is unreachable / no model loaded — **fail-closed, emits nothing**.
|
|
- `400` on malformed input.
|
|
|
|
`POST /rehydrate`
|
|
```json
|
|
{ "task_id": "...", "map_handle": "...", "items": [{"id": "out_1", "text": "...[PERSON_1]..."}],
|
|
"actor": "analyst", "strict": true }
|
|
```
|
|
→ `200 { items:[{id, rehydrated_text}], stats:{tokens_substituted, unknown_tokens} }`
|
|
- `409 {"error":"unknown_tokens","tokens":[...]}` when `strict` and a token has no map entry (your tripwire for a Claude-hallucinated/smuggled token).
|
|
- `410 {"error":"map_expired"}` if the map TTL lapsed or the handle is unknown.
|
|
|
|
## The dictionary is caller-supplied — and treated as sensitive
|
|
|
|
You supply `known_entities` (built by your `build_known_entities`, scoped to the LP
|
|
in play) in each `/scrub` call. Spark Control never reads your CRM — keeps the
|
|
package portable and needs no CRM credentials. The gateway treats your dictionary
|
|
as a slice of the LP list: used transiently for the scrub, **never persisted beyond
|
|
the resulting tokens, never logged, never echoed**. Only the resulting
|
|
`{token → real_value}` map is held server-side.
|
|
|
|
## NER backstop is load-bearing, not optional
|
|
|
|
The dictionary is the deterministic floor; the local-Qwen NER pass catches the
|
|
unknown names it can't know (new prospects, an advisor named in passing) and flags
|
|
**descriptive re-identifiers** ("the family that sold the mining company in Texas" →
|
|
redacted). Under `ner=auto` (default) or `ner=qwen`, if the local Qwen is unreachable
|
|
or no model is loaded, `/scrub` **fails closed (422)** rather than passing name-blind
|
|
text to Claude. `ner=rules_only` is the explicit, knowing opt-out — never the silent
|
|
fallback. The NER uses the same local Qwen at `/v1/chat/completions`; the sensitive
|
|
text never reaches a remote model.
|
|
|
|
> Verified live against Qwen3.6: an unknown "Sarah Kim from Atlas Ventures" → `[PERSON_1] from [ORG_1]`; a descriptive re-identifier → `[redacted]` + flagged.
|
|
|
|
## Map-stays-local
|
|
|
|
The pseudonym map (the de-anonymization key) is held only on this box, keyed by
|
|
`map_handle`, in a TTL-swept local store on the StartOS `/data` volume (default 2h;
|
|
survives a Spark Control restart mid-review). Never returned in full, never logged,
|
|
never in a Claude-bound payload. `REDACTION_MAP_TTL` and `REDACTION_MAP_DB` are
|
|
configurable via env if you want a different TTL/path.
|
|
|
|
## Logging stays on your side
|
|
|
|
`/scrub` and `/rehydrate` return counts-only `stats`; **your app writes the
|
|
`interaction_log` row** (you already have `log_scrub`/`log_rehydrate`). Spark Control
|
|
does not write to your DB and keeps no audit log of its own that contains real values.
|
|
The `descriptive_flags` span text is in the `/scrub` *response* (to you, the local
|
|
caller) — strip it before you persist, per your own logging rule (payload = counts only).
|
|
|
|
## Acceptance — what passed
|
|
|
|
1. **Parity** — the reference leak fixtures run through the live `/scrub` endpoint: every Tier-1 + Tier-2 identifier absent from the response; substance survives verbatim.
|
|
2. **Map-leak** — no real value (incl. Tier-1) in any response body; Tier-1 values absent from the server map entirely.
|
|
3. **Round-trip** — `/rehydrate` via the server-held map reproduces the original (Tier-1 → `[redacted]`, the only lossy part).
|
|
4. **Handle reuse** — same entity → same token across items and across calls reusing `map_handle` (cache-stable for Claude prompt caching).
|
|
5. **Tripwires** — 409 on a strict unmapped token; 410 on expired/unknown handle; 422 fail-closed on `tier1_action=reject`.
|
|
6. **Live NER** — unknown names tokenized + descriptive re-identifier redacted against the real local Qwen.
|
|
|
|
## Cutover
|
|
|
|
Point your `SCRUB_BACKEND=gateway` client at `https://<spark-control-host>/scrub` and
|
|
`/rehydrate` (same TLS-skip / Root-CA story as the other endpoints). The request/
|
|
response shapes match your in-repo module, so agents cut over with no app changes.
|
|
|
|
## Honest caveat (unchanged from your design)
|
|
|
|
The NER pass is the probabilistic layer — it will not catch every free-text or
|
|
descriptive re-identifier. The strong defenses remain: **minimize-first** (does Claude
|
|
need the record content at all?), the deterministic dictionary + rules, and the
|
|
re-identification spot-check. Treat the gateway as the enforcement *point*, not a
|
|
guarantee that any text is safe to send.
|