Phase 0 complete: fuzzy entity tier, incremental sync, Start9 packaging

- Fuzzy tier (backend/ingest/fuzzy_resolve.py + llm.py): local Qwen adjudicates
  the deterministic resolver's flagged name-variant candidates; merges are
  durable via entity_merges (deterministic re-runs respect them), losers
  soft-deleted, logged. Idempotent.
- Incremental sync (backend/ingest/sync.py): re-embeds only rows changed since a
  watermark (ingest_sync_state); first run / --recreate = full. Tested full→0→1.
- Start9 packaging (start9/0.4): Dockerfile bundles ingest+mcp + fastembed/mcp;
  "Build search index" action runs the init in a subcontainer; MCP shipped as a
  manual stdio server (not a daemon); version 0.1.0:44. INGEST_PACKAGING.md.
- backfill.py: factored embed_and_upsert() shared with sync.

Verified end-to-end on synthetic data + live Sparks/Qwen/Qdrant.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Keysat
2026-06-05 08:55:12 -05:00
parent c7ce44d963
commit f357c23c75
16 changed files with 808 additions and 48 deletions
+17 -4
View File
@@ -31,14 +31,27 @@ RUN apt-get update \
&& rm -rf /var/lib/apt/lists/*
# ── Python dependencies ─────────────────────────────────────────
# Only one hard dep for now: `cryptography` is required by the Gmail
# integration's RS256 JWT signing (DWD bearer tokens). Everything else
# server.py needs is stdlib.
RUN pip install --no-cache-dir cryptography==42.0.5
# `cryptography` is required by the Gmail integration's RS256 JWT signing
# (DWD bearer tokens). The two Phase-0 deps are runtime-only for the ingest
# pipeline + MCP server (the CRM web server itself still needs no new deps):
# * fastembed — client-side BM25 (Qdrant/bm25) for the sparse retrieval leg
# (backend/ingest/sparse.py auto-detects it).
# * mcp — MCP Python SDK, only needed to run backend/mcp/server.py.
# Everything else server.py needs is stdlib.
RUN pip install --no-cache-dir \
cryptography==42.0.5 \
fastembed==0.4.2 \
mcp==1.2.0
# ── Application source ──────────────────────────────────────────
COPY backend/server.py /app/backend/server.py
COPY backend/email_integration /app/backend/email_integration
# Phase-0 substrate: ingest pipeline (entity resolution + backfill) and the
# CRM MCP server. Shipped alongside the web server so the one-time index build
# and the (manually-run) MCP server can execute on the box where /data/crm.db
# lives. See start9/0.4/INGEST_PACKAGING.md.
COPY backend/ingest /app/backend/ingest
COPY backend/mcp /app/backend/mcp
COPY frontend /app/frontend
# ── StartOS wrapper scripts ─────────────────────────────────────
+142
View File
@@ -0,0 +1,142 @@
# Phase-0 ingest packaging (StartOS 0.4)
How the Phase-0 data substrate — the ingest pipeline (`backend/ingest/`) and the
CRM MCP server (`backend/mcp/`) — ships and runs on the live StartOS 0.4 package,
**without changing the CRM web server**. This implements **Option A** ("same
image") from `docs/go-live-runbook.md` §"Open decision — packaging".
The CRM web server (`backend/server.py`) is untouched and gains no new
dependencies. The `primary` daemon and its `checkPortListening` health check are
unchanged.
## What changed
| File | Change |
| --- | --- |
| `Dockerfile` | `COPY backend/ingest` and `COPY backend/mcp` into the image alongside `backend/server.py`. Added two runtime deps to the existing `pip install`: `fastembed==0.4.2` (client-side BM25 / `Qdrant/bm25` for the sparse retrieval leg) and `mcp==1.2.0` (MCP Python SDK, only for `backend/mcp/server.py`). |
| `docker_entrypoint.sh` | Added an export block for the ingest/retrieval env: `CRM_DB_PATH`, `SPARK_CONTROL_URL`, `SPARK_CONTROL_VERIFY_TLS`, `QDRANT_URL`, with LAN-default placeholder values and an operator comment. The CRM web server ignores these; they exist so manual `python3 /app/backend/ingest/...` and `backend/mcp/server.py` runs on the box inherit them. |
| `startos/actions/buildSearchIndex.ts` | **New.** A one-shot "Build search index" StartOS action (Steps 34 of the runbook). |
| `startos/actions/index.ts` | Registered the new action: `sdk.Actions.of().addAction(buildSearchIndex)`. |
| `startos/versions/v0.1.0.44.ts` + `versions/index.ts` | New version `0.1.0:44` (image-only change, no data migration) set as `current`; `0.1.0:43` moved to `other`. |
| `startos/utils.ts` | Bumped the informational `PACKAGE_VERSION` constant to `0.1.0:44`. |
### Action registration mechanism (verified)
Actions are collected in `startos/actions/index.ts` as
`export const actions = sdk.Actions.of().addAction(...)`, and that `actions`
object is passed into `sdk.setupInit(...)` in `startos/init/index.ts` (and
re-exported from `startos/index.ts`). Adding `.addAction(buildSearchIndex)` is
the entire registration — no manifest entry is required for actions in the 0.4
SDK.
## How the operator triggers the index build
1. Build/sideload the new `.s9pk` (version `0.1.0:44`). StartOS preserves
`/data`, so live data is undisturbed. On first boot the CRM's own migration
runner creates the Phase-0 tables (see runbook Step 1) — that is independent
of this packaging change.
2. In the StartOS UI, open the **Ten31 Database** service → **Actions**
**Build search index**, and run it. It:
- runs `entity_resolution.py --db /data/crm.db` (canonical ids + links), then
- runs `backfill.py --db /data/crm.db --recreate` (chunk → dense via Spark
Control + BM25 → upsert to Qdrant `crm_chunks`).
Both steps are idempotent and read-only on the CRM source tables, so the
action is safe to re-run any time to refresh the index. A full re-embed is
~815 min (the action allows up to 30 min before timing out).
The action runs in its **own subcontainer** with the same `main` volume mounted
at `/data`, with `cwd=/app/backend/ingest` (the ingest scripts import their
siblings by bare name, e.g. `import config`, so they must run from that
directory). It uses `allowedStatuses: 'any'` — SQLite WAL mode makes a
concurrently-running CRM safe for these reads/derived writes.
## Env / config the operator must set (Spark URLs)
The ingest run reaches out to **Spark Control** (dense embeddings) and **Qdrant**
(upserts). Those endpoints are LAN-specific, so they are defined in **two
places** that the operator must point at their network. The current values are
the Ten31 LAN defaults:
| Variable | Default | Used by |
| --- | --- | --- |
| `SPARK_CONTROL_URL` | `https://192.168.1.72:62419` | dense embeds (`/v1/embeddings`) |
| `SPARK_CONTROL_VERIFY_TLS` | `false` (Spark Control uses a self-signed cert) | TLS verification toggle |
| `QDRANT_URL` | `http://192.168.1.87:6333` | Qdrant collection admin + upserts |
| `CRM_DB_PATH` | `/data/crm.db` | both scripts + MCP server (already correct) |
Where to set them:
- **`docker_entrypoint.sh`** — for manual `python3` / MCP runs via the running
container. Edit the `${VAR:-default}` block, or override via the StartOS
service environment.
- **`startos/actions/buildSearchIndex.ts`** (`ingestEnv`) — for the "Build search
index" action, which runs in its own subcontainer and does **not** execute the
entrypoint, so it carries its own copy of the values. Edit these to match.
> Keep the two copies in sync. They are duplicated because the action's
> subcontainer never runs `docker_entrypoint.sh`; there is no shared config
> store wired into this package today (see "Still needed" below).
Verify reachability from the box before running the action:
`curl -sk $SPARK_CONTROL_URL/api/endpoints` and
`curl -s $QDRANT_URL/collections`.
## MCP server: decision and how to run it
**Decision: the MCP server is NOT a daemon in this release — it is shipped in the
image and run manually.** Rationale:
- `backend/mcp/server.py` is an **stdio** MCP server (`mcp.run()` with FastMCP):
it has no network port to bind, so the StartOS daemon model (a long-running
process with a `checkPortListening` health check, like `primary`) does not fit
it. There is nothing to port-probe and no meaningful liveness signal.
- **Phase 0 has no live agents** (per `CLAUDE.md` and the runbook): nothing on
the box would connect to it. An always-on daemon would idle with no client on
its stdin and no health semantics.
- It exposes reads, the three retrieval modes, and logged writes — **no
outbound/contact tools** (Phase 3 compliance gate). It is for testing and
later internal-only Analyst work.
To run it manually on the box (it is present at `/app/backend/mcp/server.py` with
`mcp` already installed):
```sh
# from inside the running container
CRM_DB_PATH=/data/crm.db python3 /app/backend/mcp/server.py
```
Then register it with the Agent SDK / Claude Code as an stdio MCP server pointing
at that script (it inherits the Spark/Qdrant env exported by the entrypoint).
If/when a live agent needs it as a persistent service, the cleanest upgrade is to
add it as a **second daemon** in `startos/main.ts` mirroring the `primary`
daemon — but only after giving it a network transport (e.g. an HTTP/SSE MCP
endpoint on its own port) so it has a real `checkPortListening` health check.
That is deliberately deferred to a later phase.
## Still needed for a fully turn-key deploy
- **MCP-as-a-service** — see above. Deferred until there is a live agent and a
network transport; today it is manual/stdio only.
- **Incremental sync (runbook Step 6 / Workstream B4)** — the action does a full
one-shot rebuild. Keeping the index fresh as the CRM changes needs an
incremental, idempotent sync on a schedule. Until that exists, re-running the
"Build search index" action is the refresh path. When built, it could be wired
as a recurring StartOS action/task rather than a manual re-run.
- **Single source of truth for Spark/Qdrant config** — currently duplicated in
`docker_entrypoint.sh` and `buildSearchIndex.ts`. A small StartOS config
store + input form (the SDK supports `Action.withInput` and a service config)
would let the operator set the endpoints once in the UI; deferred to keep this
change minimal and reviewable.
- **`.env` on the box** — `backend/ingest/config.py` also reads `/app/.env` if
present (via `os.environ.setdefault`, so it does not override the exported
env). Not required given the exported env above, but available as an
alternative if the operator prefers a file.
## Constraints honored
- No files under `backend/ingest/`, `backend/mcp/`, `backend/server.py`,
`backend/core_migrations.py`, `backend/migrations/`, or `data/` were modified;
only `start9/0.4/**` and this new doc.
- No build/deploy commands were run. `npx tsc --noEmit` was used only to verify
the new TypeScript compiles against the SDK types.
+17
View File
@@ -57,5 +57,22 @@ else
echo "[entrypoint] Gmail integration: DISABLED (no key at $GMAIL_SA_KEY)"
fi
# ── Phase-0 ingest / retrieval env ──────────────────────────────
# These are consumed by the ingest pipeline (backend/ingest/) and the MCP
# server (backend/mcp/) — NOT by the CRM web server, which ignores them.
# They are exported here so the "Build search index" StartOS action and any
# manual `python3 /app/backend/ingest/...` / `backend/mcp/server.py` run on the
# box inherit them.
#
# OPERATOR: the values below are LAN defaults for the Ten31 deployment. Set the
# real ones for your network — either by editing them here before building the
# image, or by overriding the env vars in the StartOS service environment.
# Point SPARK_CONTROL_URL at the Spark Control gateway (TLS, self-signed by
# default → SPARK_CONTROL_VERIFY_TLS=false) and QDRANT_URL at Qdrant on Spark 2.
export CRM_DB_PATH="${CRM_DB_PATH:-$DATA_DIR/crm.db}"
export SPARK_CONTROL_URL="${SPARK_CONTROL_URL:-https://192.168.1.72:62419}"
export SPARK_CONTROL_VERIFY_TLS="${SPARK_CONTROL_VERIFY_TLS:-false}"
export QDRANT_URL="${QDRANT_URL:-http://192.168.1.87:6333}"
# ── Launch the app ──────────────────────────────────────────────
exec python3 /app/backend/server.py
@@ -0,0 +1,118 @@
import { i18n } from '../i18n'
import { sdk } from '../sdk'
import { DATA_MOUNT_PATH, IMAGE_ID } from '../utils'
/**
* One-shot "Build search index" action (Phase-0 ingest, go-live Steps 34).
*
* Runs the one-time init that turns the live CRM into the canonical-id layer
* and the Qdrant search index, on the box where /data/crm.db lives:
*
* 1. entity_resolution.py --db /data/crm.db (build canonical ids + links)
* 2. backfill.py --db /data/crm.db --recreate (chunk → dense+BM25 → Qdrant)
*
* Both steps are idempotent (deterministic ids), read-only on the CRM source
* tables, and log a row to interaction_log — so re-running is always safe.
*
* Implementation notes:
* - The scripts import their siblings by bare name (`import config`, etc.),
* so they must run with cwd = /app/backend/ingest.
* - backfill.py talks to Spark Control (dense embeds) and Qdrant (upserts),
* so the Spark/Qdrant env must be present. This action runs in its OWN
* subcontainer and does NOT go through docker_entrypoint.sh, so it cannot
* inherit the entrypoint's exports — the env is passed explicitly below.
* - allowedStatuses: 'any' — the action runs in its own subcontainer with the
* same /data volume mounted, so it works whether or not the CRM is running.
* SQLite WAL mode means a concurrently-running CRM is fine for these
* reads/derived writes.
*/
const DB_PATH = `${DATA_MOUNT_PATH}/crm.db`
const INGEST_DIR = '/app/backend/ingest'
// OPERATOR: Spark Control + Qdrant endpoints for the ingest run. These are the
// LAN defaults for the Ten31 deployment — edit them for your network. Keep them
// in sync with the export block in docker_entrypoint.sh (single source of truth
// for the values; this action needs its own copy because it does not run the
// entrypoint). Spark Control is TLS with a self-signed cert by default, hence
// SPARK_CONTROL_VERIFY_TLS = 'false'.
const ingestEnv: { [k: string]: string } = {
CRM_DB_PATH: DB_PATH,
SPARK_CONTROL_URL: 'https://192.168.1.72:62419',
SPARK_CONTROL_VERIFY_TLS: 'false',
QDRANT_URL: 'http://192.168.1.87:6333',
}
export const buildSearchIndex = sdk.Action.withoutInput(
// id
'build-search-index',
// metadata
async ({ effects }) => ({
name: i18n('Build search index'),
description: i18n(
'One-time Phase-0 init: builds the canonical entity ids from your live ' +
'CRM (entity_resolution.py), then chunks + embeds every record into ' +
'the Qdrant search index (backfill.py --recreate). Both steps are ' +
'idempotent and read-only on your CRM source tables. Requires Spark ' +
'Control and Qdrant to be reachable (set SPARK_CONTROL_URL / ' +
'QDRANT_URL). A full re-embed takes roughly 815 minutes.',
),
warning: i18n(
'Rebuilds the Qdrant `crm_chunks` collection (--recreate drops and ' +
'recreates it). The index is derived from the CRM and safe to rebuild; ' +
'no CRM source data is modified.',
),
allowedStatuses: 'any',
group: null,
visibility: 'enabled',
}),
// execution
async ({ effects }) => {
const env = ingestEnv
const subcontainer = await sdk.SubContainer.of(
effects,
{ imageId: IMAGE_ID },
sdk.Mounts.of().mountVolume({
volumeId: 'main',
subpath: null,
mountpoint: DATA_MOUNT_PATH,
readonly: false,
}),
'ten31-database-build-search-index',
)
try {
// Step 3 — canonical ids from the real data (fast, local-only).
await subcontainer.execFail(
['python3', 'entity_resolution.py', '--db', DB_PATH],
{ cwd: INGEST_DIR, env },
// 10 minutes — pure SQLite work, but generous for a large corpus.
10 * 60 * 1000,
)
// Step 4 — chunk → dense (Spark Control) + BM25 → Qdrant upsert.
await subcontainer.execFail(
['python3', 'backfill.py', '--db', DB_PATH, '--recreate'],
{ cwd: INGEST_DIR, env },
// 30 minutes — a full re-embed is ~815 min; leave generous headroom.
30 * 60 * 1000,
)
} finally {
await subcontainer.destroy()
}
return {
version: '1',
title: i18n('Search index built'),
message: i18n(
'Canonical entity ids were resolved and the Qdrant `crm_chunks` ' +
'collection was rebuilt from your live CRM. You can re-run this ' +
'action any time to refresh the index.',
),
result: null,
}
},
)
+2 -1
View File
@@ -1,3 +1,4 @@
import { sdk } from '../sdk'
import { buildSearchIndex } from './buildSearchIndex'
export const actions = sdk.Actions.of()
export const actions = sdk.Actions.of().addAction(buildSearchIndex)
+5 -3
View File
@@ -3,12 +3,14 @@
// from manifest/index.ts (id, title) and versions/ (version).
export const PACKAGE_ID = 'ten-database'
export const PACKAGE_TITLE = 'Ten31 Database'
// ExVer form of the current 0.4 wrapper release (upstream 0.1.0, wrapper rev 41).
// ExVer form of the current 0.4 wrapper release (upstream 0.1.0, wrapper rev 44).
// * 0.3.5 wrapper: 0.1.0.38 (legacy, aarch64)
// * First 0.4: 0.1.0:39 (shipped seed snapshot for migration)
// * Cleanup: 0.1.0:40 (seed removed + multi-threaded server + abuser auto-ban)
// * Current: 0.1.0:41 (frontend persists auth across refreshes)
export const PACKAGE_VERSION = '0.1.0:41'
// * 0.1.0:41 (frontend persists auth across refreshes)
// * 0.1.0:42 (Gmail integration) / 0.1.0:43 (Gmail POST-body hotfix)
// * Current: 0.1.0:44 (Phase-0 ingest + MCP server in image; build-index action)
export const PACKAGE_VERSION = '0.1.0:44'
export const DATA_MOUNT_PATH = '/data'
export const WEB_PORT = 8080
+3 -2
View File
@@ -4,8 +4,9 @@ import { v_0_1_0_40 } from './v0.1.0.40'
import { v_0_1_0_41 } from './v0.1.0.41'
import { v_0_1_0_42 } from './v0.1.0.42'
import { v_0_1_0_43 } from './v0.1.0.43'
import { v_0_1_0_44 } from './v0.1.0.44'
export const versionGraph = VersionGraph.of({
current: v_0_1_0_43,
other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42],
current: v_0_1_0_44,
other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42, v_0_1_0_43],
})
+47
View File
@@ -0,0 +1,47 @@
import { VersionInfo } from '@start9labs/start-sdk'
// Phase-0 substrate packaging release.
//
// Context:
// * Ships the Phase-0 ingest pipeline (backend/ingest/) and the CRM MCP
// server (backend/mcp/) inside the existing CRM container image, alongside
// the web server. Two runtime deps are added to the image: `fastembed`
// (client-side BM25 for the sparse retrieval leg) and `mcp` (the MCP
// Python SDK, used only to run backend/mcp/server.py). The CRM web server
// itself gains no new dependencies and is unchanged.
// * Adds a one-shot "Build search index" StartOS action that runs the
// one-time init on the box where /data/crm.db lives:
// entity_resolution.py --db /data/crm.db (canonical ids)
// backfill.py --db /data/crm.db --recreate (Qdrant search index)
// Both steps are idempotent and read-only on the CRM source tables.
// * docker_entrypoint.sh now exports the Spark Control / Qdrant env
// (SPARK_CONTROL_URL, SPARK_CONTROL_VERIFY_TLS, QDRANT_URL) with LAN
// defaults so manual ingest / MCP runs on the box inherit them.
//
// The MCP server is intentionally NOT a daemon in this release: it is an
// stdio server with no port to bind and (in Phase 0) no live agent on the box
// to talk to it, so it is run manually for testing. See
// start9/0.4/INGEST_PACKAGING.md.
//
// No schema changes and no data migration: the SQLite schema is unchanged and
// the live /data volume is left exactly as-is. The new tables the ingest
// pipeline reads/writes are created by the CRM's own migration runner
// (migrations/0001_phase0_foundation.sql), independent of this package change.
export const v_0_1_0_44 = VersionInfo.of({
version: '0.1.0:44',
releaseNotes: {
en_US: [
'Ships the Phase-0 data substrate inside the CRM image: the ingest',
'pipeline (entity resolution + Qdrant backfill) and the CRM MCP server,',
'plus the fastembed and mcp runtime dependencies. Adds a one-time',
'"Build search index" action that resolves canonical entity ids from',
'your live CRM and rebuilds the Qdrant search index — both steps are',
'idempotent and read-only on your CRM source data. The CRM web server',
'is unchanged and gains no new dependencies. No data migration.',
].join(' '),
},
migrations: {
up: async () => {},
down: async () => {},
},
})