ten31-database/docs/spark-control-scrub-endpoints.md

# Spark Control — `/scrub` + `/rehydrate` endpoints (handover prompt)

*Hand this to the Spark Control developer to build the gateway redaction endpoints.
**OPTIONAL for Phase 1** — the Architect's grounding boundary already ships in our app
repo (the `backend/redaction/` module, deterministic scrub + golden-file leak tests).
Build this when we move to **multi-agent** enforcement (Phase 2 Analyst, Phase 3 Closer),
where we want ONE bypass-proof point with the pseudonym map living next to the local
models. Our in-repo module is the **reference implementation**; the contract below is what
our `SCRUB_BACKEND=gateway` client already calls, so aim for behavioral parity and we cut
agents over with zero app changes. Related: `docs/redaction-rehydration.md`.*

---

## Purpose

Add two gateway endpoints, `POST /scrub` and `POST /rehydrate`, that let any agent send
LP-specific context to the Claude API without exposing sovereign data. `/scrub`
de-identifies an agent's assembled context; the agent sends the de-identified text to
Claude; `/rehydrate` puts the real values back into Claude's response for human review.
The **pseudonym map** that links tokens to real values is the de-anonymization key — it
**must stay on the Spark, next to the local models, and never be sent or logged to any
third party.** This sits behind the same trusted Spark Control URL, TLS, and access
control that already front `/v1/chat/completions`, `/v1/embeddings`, `/v1/rerank`, and
`/api/search`. Spark Control does **not** call Claude here — it is the scrub and rehydrate
transform pair plus a server-held map.

## The three-tier classification (the rules `/scrub` enforces)

Classify every span into exactly one tier:

- **TIER-1 NEVER-SEND** — excluded entirely, not even tokenized: full LP list/export or
  bulk relationship graph, raw account/wire/routing numbers, SSN/passport/gov-ID, anything
  under a confidentiality obligation. Default: excise the span. `tier1_action="reject"`
  makes the whole call fail-closed (422) if any Tier-1 is detected.
- **TIER-2 TOKENIZE** — stable placeholders, swapped back locally after Claude: person
  names, org/fund names, emails, phones, addresses, exact $ amounts, identity-pinning
  dates. Placeholders are `[TYPE_N]`: `[PERSON_1] [ORG_1] [FUND_1] [EMAIL_1] [PHONE_1]
  [ADDR_1] [AMOUNT_1] [DATE_1] [LOC_1] [MISC_1]`. `N` is 1-based and **stable within a
  task**: the same real entity maps to the same token across every item in the call and
  across later `/scrub` calls reusing the same `map_handle`, so Claude can reason about
  relationships (`[PERSON_1] introduced [PERSON_2] to [FUND_1]`).
- **SEND-AS-IS** — passed through untouched: the substance Claude needs (objections,
  sentiment, generic deal mechanics, the drafted message body minus identifiers).

## The round-trip

`SCRUB` (your endpoint) → `REASON` (the agent calls Claude with placeholders only — your
gateway does NOT call Claude) → `RE-HYDRATE` (your endpoint) → human review (in our app).

## Request / response contracts

### `POST /scrub`
```json
{
  "task_id": "string, required, caller-chosen, stable across the round-trip",
  "actor": "string, agent name e.g. 'analyst' | 'closer' (for logging)",
  "items": [ {"id": "ctx_1", "text": "..."} ],
  "known_entities": {                         // CALLER-SUPPLIED dictionary (see "Dictionary" below)
    "persons": ["..."], "orgs": ["..."], "funds": ["..."], "emails": ["..."]
  },
  "tier1_action": "drop | reject",            // default "drop"
  "bucket": {"amounts": false, "dates": false}, // default FALSE for grounding: tokenize reversibly (no magnitudes to Claude). bucket=true only when a caller genuinely needs coarse magnitude
  "ner": "auto | rules_only | qwen",          // default "auto"
  "map_handle": "string, optional, reuse/extend an existing task map"
}
```
Response `200`:
```json
{
  "task_id": "...",
  "map_handle": "opaque server key to the map (NOT the map itself)",
  "items": [ {"id":"ctx_1","scrubbed_text":"...","tokens_used":["PERSON_1","AMOUNT_1"]} ],
  "stats": {"tier1_dropped": 2, "tier2_tokenized": 14, "distinct_entities": 9,
            "descriptive_flags": [{"item":"ctx_1","span":"the family that sold the mining company in Texas","action":"redacted"}]},
  "expires_at": "ISO-8601 map TTL (short-lived, e.g. 2h)"
}
```
`422 {"error":"tier1_detected","spans":[...]}` when `tier1_action="reject"` and Tier-1 found.
`400` on malformed input.

### `POST /rehydrate`
```json
{
  "task_id": "string, required",
  "map_handle": "string, required, from /scrub",
  "items": [ {"id":"out_1","text":"...with [PERSON_1] tokens..."} ],
  "actor": "string",
  "strict": true   // default true
}
```
Response `200`:
```json
{ "items": [ {"id":"out_1","rehydrated_text":"...real values..."} ],
  "stats": {"tokens_substituted": 6, "unknown_tokens": []} }
```
`409 {"error":"unknown_tokens","tokens":["PERSON_9"]}` when `strict` and the text contains a
token with no map entry — this is your tripwire for a Claude-hallucinated/smuggled token; do
NOT silently pass it through. `410 {"error":"map_expired"}` if the map TTL lapsed.

## Dictionary: CALLER-SUPPLIED (Option 1 — decided)

The known-entity dictionary is supplied by the **caller** in each `/scrub` request
(`known_entities`), NOT read by Spark Control from the CRM. We chose this over giving the
gateway CRM access because it keeps Spark Control **generic and portable** (no coupling to
Ten31's schema, so the package still works for another dual-Spark user), needs **no CRM
credentials on the gateway** (least privilege — the LP list lives in one place), stays
**fresh** (built live from the CRM each call), and matches the reference `scrub()` signature.
Our app builds it with its `build_known_entities` (names + name-parts + email local-parts)
and sends it scoped to the request. Treat it as **sensitive** (a slice of the LP list): hold
it only transiently with the map, never log or forward it. Tokens are per-task (stable within
a `task_id`/`map_handle`), which is also the lower re-identification-risk choice. If per-call
payload ever becomes a perf issue, an optional gateway-side cached export (Option 3) is the
escape hatch — but do not build that coupling now.

## Local-Qwen NER (how to find Tier-2 entities), cheapest/most-authoritative first

1. **The caller-supplied `known_entities`.** Tokenize every supplied person/org/fund/email
   surface deterministically (case- and unicode-fold insensitive, longest-match-first, with
   hyphenated-surname extension). This is the bulk of the work and needs no model. (This is
   the deterministic FLOOR — load-bearing but NOT complete; the NER pass below is required.)
2. **Rules** for structured PII: regexes for emails, phones, $ amounts, dates, and
   SSN/account-number shapes (account/SSN shapes route to Tier-1).
3. **Local-Qwen NER pass** for residual named entities the map and rules miss (a person/org
   in free text not yet in the CRM). Call the SAME local Qwen you already serve at
   `/v1/chat/completions` (enable_thinking=false, temperature 0), with a strict JSON-only
   extraction prompt returning `{entities:[{text,type,tier}]}`. Never a remote model — the
   input is exactly the sensitive text we are keeping local. `ner="rules_only"` skips this;
   `ner="qwen"` forces it; `auto` runs it only on spans unresolved by steps 1–2. Qwen should
   also flag **descriptive** re-identifiers (e.g. "the family that sold the mining company in
   Texas"), not just named entities.

## Map-stays-local (non-negotiable)

The pseudonym map `{token -> real_value}` is stored ONLY on the Spark, keyed by
`map_handle`, in memory or a short-lived local store, TTL-expired (default ~2h). Never
returned in full, never written to any log/metric/trace that leaves the box, never in any
Claude-bound payload. `/scrub` returns only the opaque `map_handle` and counts. If you add
`GET /scrub/map/{map_handle}` for same-box debugging, gate it behind the same auth and keep
it off by default.

## Logging

One row per call to our `interaction_log` (we give you a write path or ingest an emitted
event): `action='redaction.scrub' | 'redaction.rehydrate'`, `actor_type='agent'`,
`actor_id=<actor>`, `source='spark_control'`, `payload` = **COUNTS ONLY** (tiers dropped,
tokens by type, distinct_entities, descriptive_flags spans WITHOUT real values, model used).
The payload MUST NOT contain any real Tier-2 value, any Tier-1 content, or the map.

## Acceptance tests (must pass before agents route through it)

1. **Golden-file diff** — fixed inputs, recorded expected scrubbed output; assert NO Tier-1
   string and NO real Tier-2 identifier appears in any `/scrub` response or Claude-bound
   payload. (We will share our `backend/redaction/` fixtures + golden files.)
2. **Round-trip identity** — scrub → echo tokens → rehydrate reproduces the original real
   values exactly; token stability holds across items and across a second `/scrub` reusing
   the same `map_handle`.
3. **Re-identification spot-check** — feed ONLY the de-identified prompt to the local Qwen
   and ask it to name the real people/orgs/amounts; anything it recovers (esp. descriptive
   re-identifiers and amount+date+sector inference) is flagged and must be driven to zero or
   escalated to Tier-1/bucketing.
4. **Map-leak assertion** — scan every response body, log row, and Claude-bound payload in
   the suite; assert the map and all real values are absent.
5. **Strict-rehydrate tripwire** — a rehydrate input with an unmapped token returns 409.
6. **Fail-closed** — `tier1_action="reject"` returns 422 on any Tier-1 input, emits nothing.

## Notes

- This does NOT replace **minimization**: the agent must first ask "does Claude need this
  record content at all?" — often a local retrieval summary suffices. These endpoints are
  for when the answer is genuinely yes.
- Keep placeholders **cache-stable** (deterministic token assignment per task_id/map_handle)
  so they compose with prompt caching on the Claude side.
- We will hand you our in-repo `redaction` module + golden files as the reference; aim for
  behavioral parity first, then we cut agents over to the gateway as the single enforcement
  point.