# Phase-0 ingest packaging (StartOS 0.4)

How the Phase-0 data substrate — the ingest pipeline (`backend/ingest/`) and the
CRM MCP server (`backend/mcp/`) — ships and runs on the live StartOS 0.4 package,
**without changing the CRM web server**. This implements **Option A** ("same
image") from `docs/go-live-runbook.md` §"Open decision — packaging".

The CRM web server (`backend/server.py`) is untouched and gains no new
dependencies. The `primary` daemon and its `checkPortListening` health check are
unchanged.

## What changed

| File | Change |
| --- | --- |
| `Dockerfile` | `COPY backend/ingest` and `COPY backend/mcp` into the image alongside `backend/server.py`. Added two runtime deps to the existing `pip install`: `fastembed==0.4.2` (client-side BM25 / `Qdrant/bm25` for the sparse retrieval leg) and `mcp==1.2.0` (MCP Python SDK, only for `backend/mcp/server.py`). |
| `docker_entrypoint.sh` | Added an export block for the ingest/retrieval env: `CRM_DB_PATH`, `SPARK_CONTROL_URL`, `SPARK_CONTROL_VERIFY_TLS`, `QDRANT_URL`, with LAN-default placeholder values and an operator comment. The CRM web server ignores these; they exist so manual `python3 /app/backend/ingest/...` and `backend/mcp/server.py` runs on the box inherit them. |
| `startos/actions/buildSearchIndex.ts` | **New.** A one-shot "Build search index" StartOS action (Steps 3–4 of the runbook). |
| `startos/actions/index.ts` | Registered the new action: `sdk.Actions.of().addAction(buildSearchIndex)`. |
| `startos/versions/v0.1.0.44.ts` + `versions/index.ts` | New version `0.1.0:44` (image-only change, no data migration) set as `current`; `0.1.0:43` moved to `other`. |
| `startos/utils.ts` | Bumped the informational `PACKAGE_VERSION` constant to `0.1.0:44`. |

### Action registration mechanism (verified)

Actions are collected in `startos/actions/index.ts` as
`export const actions = sdk.Actions.of().addAction(...)`, and that `actions`
object is passed into `sdk.setupInit(...)` in `startos/init/index.ts` (and
re-exported from `startos/index.ts`). Adding `.addAction(buildSearchIndex)` is
the entire registration — no manifest entry is required for actions in the 0.4
SDK.

## How the operator triggers the index build

1. Build/sideload the new `.s9pk` (version `0.1.0:44`). StartOS preserves
   `/data`, so live data is undisturbed. On first boot the CRM's own migration
   runner creates the Phase-0 tables (see runbook Step 1) — that is independent
   of this packaging change.
2. In the StartOS UI, open the **Ten31 Database** service → **Actions** →
   **Build search index**, and run it. It:
   - runs `entity_resolution.py --db /data/crm.db` (canonical ids + links), then
   - runs `backfill.py --db /data/crm.db --recreate` (chunk → dense via Spark
     Control + BM25 → upsert to Qdrant `crm_chunks`).
   Both steps are idempotent and read-only on the CRM source tables, so the
   action is safe to re-run any time to refresh the index. A full re-embed is
   ~8–15 min (the action allows up to 30 min before timing out).

The action runs in its **own subcontainer** with the same `main` volume mounted
at `/data`, with `cwd=/app/backend/ingest` (the ingest scripts import their
siblings by bare name, e.g. `import config`, so they must run from that
directory). It uses `allowedStatuses: 'any'` — SQLite WAL mode makes a
concurrently-running CRM safe for these reads/derived writes.

## Env / config the operator must set (Spark URLs)

The ingest run reaches out to **Spark Control** (dense embeddings) and **Qdrant**
(upserts). Those endpoints are LAN-specific, so they are defined in **two
places** that the operator must point at their network. The current values are
the Ten31 LAN defaults:

| Variable | Default | Used by |
| --- | --- | --- |
| `SPARK_CONTROL_URL` | `https://192.168.1.72:62419` | dense embeds (`/v1/embeddings`) |
| `SPARK_CONTROL_VERIFY_TLS` | `false` (Spark Control uses a self-signed cert) | TLS verification toggle |
| `QDRANT_URL` | `http://192.168.1.87:6333` | Qdrant collection admin + upserts |
| `CRM_DB_PATH` | `/data/crm.db` | both scripts + MCP server (already correct) |

Where to set them:

- **`docker_entrypoint.sh`** — for manual `python3` / MCP runs via the running
  container. Edit the `${VAR:-default}` block, or override via the StartOS
  service environment.
- **`startos/actions/buildSearchIndex.ts`** (`ingestEnv`) — for the "Build search
  index" action, which runs in its own subcontainer and does **not** execute the
  entrypoint, so it carries its own copy of the values. Edit these to match.

> Keep the two copies in sync. They are duplicated because the action's
> subcontainer never runs `docker_entrypoint.sh`; there is no shared config
> store wired into this package today (see "Still needed" below).

Verify reachability from the box before running the action:
`curl -sk $SPARK_CONTROL_URL/api/endpoints` and
`curl -s $QDRANT_URL/collections`.

## MCP server: decision and how to run it

**Decision: the MCP server is NOT a daemon in this release — it is shipped in the
image and run manually.** Rationale:

- `backend/mcp/server.py` is an **stdio** MCP server (`mcp.run()` with FastMCP):
  it has no network port to bind, so the StartOS daemon model (a long-running
  process with a `checkPortListening` health check, like `primary`) does not fit
  it. There is nothing to port-probe and no meaningful liveness signal.
- **Phase 0 has no live agents** (per `CLAUDE.md` and the runbook): nothing on
  the box would connect to it. An always-on daemon would idle with no client on
  its stdin and no health semantics.
- It exposes reads, the three retrieval modes, and logged writes — **no
  outbound/contact tools** (Phase 3 compliance gate). It is for testing and
  later internal-only Analyst work.

To run it manually on the box (it is present at `/app/backend/mcp/server.py` with
`mcp` already installed):

```sh
# from inside the running container
CRM_DB_PATH=/data/crm.db python3 /app/backend/mcp/server.py
```

Then register it with the Agent SDK / Claude Code as an stdio MCP server pointing
at that script (it inherits the Spark/Qdrant env exported by the entrypoint).

If/when a live agent needs it as a persistent service, the cleanest upgrade is to
add it as a **second daemon** in `startos/main.ts` mirroring the `primary`
daemon — but only after giving it a network transport (e.g. an HTTP/SSE MCP
endpoint on its own port) so it has a real `checkPortListening` health check.
That is deliberately deferred to a later phase.

## Still needed for a fully turn-key deploy

- **MCP-as-a-service** — see above. Deferred until there is a live agent and a
  network transport; today it is manual/stdio only.
- **Incremental sync (runbook Step 6 / Workstream B4)** — the action does a full
  one-shot rebuild. Keeping the index fresh as the CRM changes needs an
  incremental, idempotent sync on a schedule. Until that exists, re-running the
  "Build search index" action is the refresh path. When built, it could be wired
  as a recurring StartOS action/task rather than a manual re-run.
- **Single source of truth for Spark/Qdrant config** — currently duplicated in
  `docker_entrypoint.sh` and `buildSearchIndex.ts`. A small StartOS config
  store + input form (the SDK supports `Action.withInput` and a service config)
  would let the operator set the endpoints once in the UI; deferred to keep this
  change minimal and reviewable.
- **`.env` on the box** — `backend/ingest/config.py` also reads `/app/.env` if
  present (via `os.environ.setdefault`, so it does not override the exported
  env). Not required given the exported env above, but available as an
  alternative if the operator prefers a file.

## Constraints honored

- No files under `backend/ingest/`, `backend/mcp/`, `backend/server.py`,
  `backend/core_migrations.py`, `backend/migrations/`, or `data/` were modified;
  only `start9/0.4/**` and this new doc.
- No build/deploy commands were run. `npx tsc --noEmit` was used only to verify
  the new TypeScript compiles against the SDK types.