Fix people double-count + duplicate-queue explosion (v0.1.0:51)

Root cause: grid contacts (fundraising_contacts) are the SAME people as the
contacts table (the app syncs them by name/email), but resolution matched grid
rows by (name + investor-canon) where the two sides derive the investor key from
different tables that rarely line up — so nearly every grid contact minted a
duplicate person (715 + ~692 ≈ 1406), and the duplicate finder then flagged each
twin against its real self (~676 candidates).

Fix (entity_resolution.py):
- Grid pass matches a grid contact to its existing contacts-table person by
  PROVABLE keys only (exact email, else exact name within the same investor) and
  records membership; on a miss it MINTS NOTHING (the old else-branch mint was the
  double-count source, and guessing by name across firms risks binding two
  different same-named people).
- Targeted, audited cleanup soft-deletes leftover grid-only "twins" (person rows
  with no 'contacts' link) and superseded pre-:48 'lp'/'organization' rows, guarded
  so any row carrying enrichment/human data is never dropped (guardrail #3); the
  tombstoned ids are logged to interaction_log (guardrail #5).
- _upsert_entity clears deleted_at on conflict so a re-emitted id is un-tombstoned
  (no permanent burial); fuzzy-merge losers stay buried via _redirect.

entity_merge.py / server.py: the duplicate queue + pending count now filter to
candidates whose both sides are still live, so self-healed twins drop out.

Verified: offline reproduction test (backend/ingest/test_entity_resolution.py,
10/10) reproduces the 1406-style doubling and proves it collapses; no regression
on the synthetic dev set; two adversarial review passes. Known pre-existing
identity-key weaknesses (same name+firm+no email collision; shared role inbox
over-link) are unchanged by this fix and will be resolved structurally by the
contact_id link in the grid/contacts unification.

Run "Build search index" after upgrading to recompute the canonical layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Keysat
2026-06-05 14:49:39 -05:00
parent 3d9caac178
commit d16264f401
7 changed files with 293 additions and 11 deletions
+7 -1
View File
@@ -3517,8 +3517,14 @@ class CRMHandler(BaseHTTPRequestHandler):
except Exception:
out['recent_activity'] = []
try:
# Count only candidates whose both sides are still live (mirror the
# review queue in entity_merge.list_candidates) — self-healed twins
# whose duplicate was soft-deleted no longer count as pending work.
out['pending_merge_candidates'] = conn.execute(
"SELECT COUNT(*) FROM entity_merge_candidates WHERE status='pending'").fetchone()[0]
"""SELECT COUNT(*) FROM entity_merge_candidates mc
JOIN canonical_entities a ON a.id=mc.entity_a AND a.deleted_at IS NULL
JOIN canonical_entities b ON b.id=mc.entity_b AND b.deleted_at IS NULL
WHERE mc.status='pending'""").fetchone()[0]
except Exception:
out['pending_merge_candidates'] = None
out['index_job'] = entity_jobs.get_status() if entity_jobs else None