Phase 0 complete: fuzzy entity tier, incremental sync, Start9 packaging

- Fuzzy tier (backend/ingest/fuzzy_resolve.py + llm.py): local Qwen adjudicates
  the deterministic resolver's flagged name-variant candidates; merges are
  durable via entity_merges (deterministic re-runs respect them), losers
  soft-deleted, logged. Idempotent.
- Incremental sync (backend/ingest/sync.py): re-embeds only rows changed since a
  watermark (ingest_sync_state); first run / --recreate = full. Tested full→0→1.
- Start9 packaging (start9/0.4): Dockerfile bundles ingest+mcp + fastembed/mcp;
  "Build search index" action runs the init in a subcontainer; MCP shipped as a
  manual stdio server (not a daemon); version 0.1.0:44. INGEST_PACKAGING.md.
- backfill.py: factored embed_and_upsert() shared with sync.

Verified end-to-end on synthetic data + live Sparks/Qwen/Qdrant.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Keysat
2026-06-05 08:55:12 -05:00
parent c7ce44d963
commit f357c23c75
16 changed files with 808 additions and 48 deletions
+17
View File
@@ -57,5 +57,22 @@ else
echo "[entrypoint] Gmail integration: DISABLED (no key at $GMAIL_SA_KEY)"
fi
# ── Phase-0 ingest / retrieval env ──────────────────────────────
# These are consumed by the ingest pipeline (backend/ingest/) and the MCP
# server (backend/mcp/) — NOT by the CRM web server, which ignores them.
# They are exported here so the "Build search index" StartOS action and any
# manual `python3 /app/backend/ingest/...` / `backend/mcp/server.py` run on the
# box inherit them.
#
# OPERATOR: the values below are LAN defaults for the Ten31 deployment. Set the
# real ones for your network — either by editing them here before building the
# image, or by overriding the env vars in the StartOS service environment.
# Point SPARK_CONTROL_URL at the Spark Control gateway (TLS, self-signed by
# default → SPARK_CONTROL_VERIFY_TLS=false) and QDRANT_URL at Qdrant on Spark 2.
export CRM_DB_PATH="${CRM_DB_PATH:-$DATA_DIR/crm.db}"
export SPARK_CONTROL_URL="${SPARK_CONTROL_URL:-https://192.168.1.72:62419}"
export SPARK_CONTROL_VERIFY_TLS="${SPARK_CONTROL_VERIFY_TLS:-false}"
export QDRANT_URL="${QDRANT_URL:-http://192.168.1.87:6333}"
# ── Launch the app ──────────────────────────────────────────────
exec python3 /app/backend/server.py