ten31-database/start9/0.4/INGEST_PACKAGING.md

Phase-0 ingest packaging (StartOS 0.4)

How the Phase-0 data substrate — the ingest pipeline (backend/ingest/) and the CRM MCP server (backend/mcp/) — ships and runs on the live StartOS 0.4 package, without changing the CRM web server. This implements Option A ("same image") from docs/go-live-runbook.md §"Open decision — packaging".

The CRM web server (backend/server.py) is untouched and gains no new dependencies. The primary daemon and its checkPortListening health check are unchanged.

What changed

File	Change
Dockerfile	COPY backend/ingest and COPY backend/mcp into the image alongside backend/server.py. Added two runtime deps to the existing pip install: fastembed==0.4.2 (client-side BM25 / Qdrant/bm25 for the sparse retrieval leg) and mcp==1.2.0 (MCP Python SDK, only for backend/mcp/server.py). These two pins carry a pre-release multi-arch verification requirement — see "Pre-release checks" below.
docker_entrypoint.sh	Added an export block for the ingest/retrieval env: CRM_DB_PATH, SPARK_CONTROL_URL, SPARK_CONTROL_VERIFY_TLS, QDRANT_URL, CRM_INGEST_SYNC_INTERVAL_MIN, with LAN-default placeholder values and an operator comment. The CRM web server ignores these; they exist so manual python3 /app/backend/ingest/... and backend/mcp/server.py runs on the box inherit them. Also launches the background ingest sync scheduler (sync_scheduler.py) before exec-ing the web server, guarded so it only starts when Spark Control + Qdrant are configured — see "Automatic scheduled refresh" below.
startos/actions/buildSearchIndex.ts	New. A one-shot "Build search index" StartOS action (Steps 3–4 of the runbook) — full rebuild with --recreate.
startos/actions/refreshSearchIndex.ts	New. A manual "Refresh search index" action — incremental, idempotent sync.py (no --recreate); the manual counterpart to the background scheduler.
startos/actions/index.ts	Registered both actions: sdk.Actions.of().addAction(buildSearchIndex).addAction(refreshSearchIndex).
startos/versions/v0.1.0.44.ts + versions/index.ts	New version 0.1.0:44 (image-only change, no data migration) set as current; 0.1.0:43 moved to other.
startos/utils.ts	Bumped the informational PACKAGE_VERSION constant to 0.1.0:44.

Action registration mechanism (verified)

Actions are collected in startos/actions/index.ts as export const actions = sdk.Actions.of().addAction(...), and that actions object is passed into sdk.setupInit(...) in startos/init/index.ts (and re-exported from startos/index.ts). Adding .addAction(buildSearchIndex) is the entire registration — no manifest entry is required for actions in the 0.4 SDK.

How the operator triggers the index build

1. Build/sideload the new .s9pk (version 0.1.0:44). StartOS preserves /data, so live data is undisturbed. On first boot the CRM's own migration runner creates the Phase-0 tables (see runbook Step 1) — that is independent of this packaging change.
2. In the StartOS UI, open the Ten31 Database service → Actions → Build search index, and run it. It:
   • runs entity_resolution.py --db /data/crm.db (canonical ids + links), then
   • runs backfill.py --db /data/crm.db --recreate (chunk → dense via Spark Control + BM25 → upsert to Qdrant crm_chunks). Both steps are idempotent and read-only on the CRM source tables, so the action is safe to re-run any time to refresh the index. A full re-embed is ~8–15 min (the action allows up to 30 min before timing out).

The action runs in its own subcontainer with the same main volume mounted at /data, with cwd=/app/backend/ingest (the ingest scripts import their siblings by bare name, e.g. import config, so they must run from that directory). It uses allowedStatuses: 'any' — SQLite WAL mode makes a concurrently-running CRM safe for these reads/derived writes.

Keeping the index fresh (hands-off refresh)

The "Build search index" action above is a full one-shot rebuild. To keep the index current as the CRM changes, there are now two incremental paths — both run sync.py (chunk → dense+BM25 → Qdrant upsert) for changed records only, with NO --recreate, so they never drop the collection and are safe to run any time.

Manual: "Refresh search index" action

startos/actions/refreshSearchIndex.ts adds a second StartOS action, Refresh search index (id refresh-search-index). It mirrors buildSearchIndex.ts exactly — same subcontainer, same /data mount, same explicit ingestEnv — but runs python3 sync.py --db /data/crm.db (no --recreate) with cwd = /app/backend/ingest. An incremental delta is usually seconds to a few minutes (the action allows up to 30 min of headroom). Use it for an on-demand refresh; use "Build search index" only for a full rebuild.

Automatic: background sync scheduler

For hands-off freshness, docker_entrypoint.sh launches backend/ingest/sync_scheduler.py as a background process just before it execs the web server. sync_scheduler.py loops the incremental sync every CRM_INGEST_SYNC_INTERVAL_MIN minutes (default 60, exported in the entrypoint's env block with an operator comment). It logs to /data/ingest-sync.log.

The launch is guarded: it only starts when both SPARK_CONTROL_URL and QDRANT_URL are set (both are exported just above it, so the default LAN values satisfy the guard; an operator who clears them to disable ingest also disables the scheduler). The entrypoint prints STARTED or SKIPPED (Spark/Qdrant not configured) so the choice is visible in the service logs.

Why a background process and not a StartOS daemon

The prior agent deliberately avoided adding the stdio MCP server as a daemon because StartOS daemons are built around a network port + checkPortListening health check, and a portless process has no liveness signal to probe (see "MCP server" below). sync_scheduler.py is the same shape — a long-running loop with no port — so adding it as a second daemon in main.ts would hit the same mismatch.

Launching it as a child of the entrypoint sidesteps that entirely:

• Pro: no portless-daemon contortion; it shares the primary container's /data and inherited env; the existing primary daemon and its checkPortListening health check are untouched.
• Con: StartOS does not supervise it independently. If the scheduler dies it is not auto-restarted on its own (the container as a whole is still health-checked via the web server), and it has no separate status tile in the UI. Crashes surface only in /data/ingest-sync.log. The manual "Refresh search index" action is the always-available fallback.

If a future phase wants first-class supervision/visibility, promote it to a real StartOS daemon — but, as with the MCP server, only after giving the work a network transport (e.g. a tiny HTTP health endpoint) so it has a meaningful checkPortListening probe.

Env / config the operator must set (Spark URLs)

The ingest run reaches out to Spark Control (dense embeddings) and Qdrant (upserts). Those endpoints are LAN-specific, so they are defined in two places that the operator must point at their network. The current values are the Ten31 LAN defaults:

Variable	Default	Used by
SPARK_CONTROL_URL	https://192.168.1.72:62419	dense embeds (/v1/embeddings)
SPARK_CONTROL_VERIFY_TLS	false (Spark Control uses a self-signed cert)	TLS verification toggle
QDRANT_URL	http://192.168.1.87:6333	Qdrant collection admin + upserts
CRM_DB_PATH	/data/crm.db	both scripts + MCP server (already correct)
CRM_INGEST_SYNC_INTERVAL_MIN	60	background sync scheduler loop interval (entrypoint only)

Where to set them:

• docker_entrypoint.sh — for manual python3 / MCP runs via the running container and for the background sync scheduler. Edit the ${VAR:-default} block, or override via the StartOS service environment.
• startos/actions/buildSearchIndex.ts and startos/actions/refreshSearchIndex.ts (ingestEnv) — for the "Build search index" and "Refresh search index" actions, which run in their own subcontainers and do not execute the entrypoint, so each carries its own copy of the values. Edit these to match. (CRM_INGEST_SYNC_INTERVAL_MIN only matters to the entrypoint's scheduler loop, not to the actions.)

Keep the copies in sync. They are duplicated because the actions' subcontainers never run docker_entrypoint.sh; there is no shared config store wired into this package today (see "Still needed" below).

Verify reachability from the box before running the action: curl -sk $SPARK_CONTROL_URL/api/endpoints and curl -s $QDRANT_URL/collections.

MCP server: decision and how to run it

Decision: the MCP server is NOT a daemon in this release — it is shipped in the image and run manually. Rationale:

• backend/mcp/server.py is an stdio MCP server (mcp.run() with FastMCP): it has no network port to bind, so the StartOS daemon model (a long-running process with a checkPortListening health check, like primary) does not fit it. There is nothing to port-probe and no meaningful liveness signal.
• Phase 0 has no live agents (per CLAUDE.md and the runbook): nothing on the box would connect to it. An always-on daemon would idle with no client on its stdin and no health semantics.
• It exposes reads, the three retrieval modes, and logged writes — no outbound/contact tools (Phase 3 compliance gate). It is for testing and later internal-only Analyst work.

To run it manually on the box (it is present at /app/backend/mcp/server.py with mcp already installed):

# from inside the running container
CRM_DB_PATH=/data/crm.db python3 /app/backend/mcp/server.py

Then register it with the Agent SDK / Claude Code as an stdio MCP server pointing at that script (it inherits the Spark/Qdrant env exported by the entrypoint).

If/when a live agent needs it as a persistent service, the cleanest upgrade is to add it as a second daemon in startos/main.ts mirroring the primary daemon — but only after giving it a network transport (e.g. an HTTP/SSE MCP endpoint on its own port) so it has a real checkPortListening health check. That is deliberately deferred to a later phase.

Still needed for a fully turn-key deploy

• MCP-as-a-service — see above. Deferred until there is a live agent and a network transport; today it is manual/stdio only.
• Incremental sync (runbook Step 6 / Workstream B4) — done. The background sync scheduler (sync_scheduler.py, started by the entrypoint) keeps the index fresh automatically, and the manual "Refresh search index" action provides an on-demand incremental sync. See "Keeping the index fresh" above. A future enhancement could still promote the scheduler to a first-class StartOS daemon (with a network transport for a real health check) for independent supervision/visibility.
• Single source of truth for Spark/Qdrant config — currently duplicated in docker_entrypoint.sh, buildSearchIndex.ts, and refreshSearchIndex.ts. A small StartOS config store + input form (the SDK supports Action.withInput and a service config) would let the operator set the endpoints once in the UI; deferred to keep this change minimal and reviewable.
• .env on the box — backend/ingest/config.py also reads /app/.env if present (via os.environ.setdefault, so it does not override the exported env). Not required given the exported env above, but available as an alternative if the operator prefers a file.

Pre-release checks

Verify before cutting a release:

• Multi-arch dependency build (BLOCKER). The fastembed==0.4.2 and mcp==1.2.0 pins in Dockerfile were chosen best-effort and have not been confirmed to build on both x86_64 and aarch64. StartOS targets arm64, and fastembed pulls onnxruntime (which may have no prebuilt arm64 wheel and fall back to a slow source build) plus downloads a model on first use. Build the image on aarch64 and run the ingest once end-to-end before release. Do not bump either pin without re-verifying on both arches. (Flagged inline above the pip line in Dockerfile.)
• Scheduler smoke test. With Spark Control + Qdrant reachable, start the container and confirm the entrypoint logs [entrypoint] ingest sync scheduler: STARTED, that /data/ingest-sync.log accumulates sync output, and that clearing one of the endpoints flips the log to SKIPPED.
• Actions present. Confirm both Build search index and Refresh search index appear under the service's Actions in the StartOS UI and run to success.

Constraints honored

• No files under backend/ingest/, backend/mcp/, backend/server.py, backend/core_migrations.py, backend/migrations/, or data/ were modified; only start9/0.4/** and this doc. The entrypoint and the refresh action reference backend/ingest/sync_scheduler.py and backend/ingest/sync.py by path only — those scripts are owned/created by a separate process.
• No build/deploy commands were run. npx tsc --noEmit was used only to verify the new TypeScript compiles against the SDK types.