diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..1f95836 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,101 @@ +# CLAUDE.md + +Browser-based StartOS 0.4 package controlling a dual NVIDIA DGX Spark AI cluster: one-click vLLM model swaps, plus health, proxying, and APIs for speech (STT/diarization/TTS), embeddings, and redaction. + +## Stack + +- Two halves, always coordinated: + - `image/` — standalone FastAPI app (Python ≥3.11; ships on `python:3.12-slim`; UI on port 9999; vanilla HTML/CSS/JS, no framework). + - `package/` — StartOS 0.4 wrapper (TypeScript, `@start9labs/start-sdk` pinned `1.3.3`, Node ≥22, bundled by `@vercel/ncc`). +- Build host needs `start-cli`, Node ≥22 + npm, and Docker (the s9pk embeds the Docker image). +- Cluster runtimes live **on the Sparks, not in this repo** (`spark-vllm-docker`, the parakeet/kokoro/embeddings containers). This repo is the controller; it reaches them over SSH + HTTP. +- Sparks are ARM64 (GB10 Grace-Blackwell, sm_121, CUDA 13). Services: vLLM `:8888` (Spark 1); `parakeet-asr` `:8000` (Parakeet TDT 0.6B v3 + Sortformer diarizer + TitaNet voiceprints), Kokoro TTS `:8880`, bge-m3 embeddings + Qdrant (Spark 2). See `docs/` for API contracts. + +## Commands + +### Build & deploy the s9pk +```bash +cd package +npm i # one-time +make x86 # typecheck + ncc bundle + docker build + pack → spark-control_x86_64.s9pk +make install # sideload to the Start9 server; needs "host: http://.local" in ~/.startos/config.yaml +``` +`make aarch64` for ARM Start9 servers. `make install` picks the newest `*.s9pk` in `package/`. + +### Local dev (FastAPI only, no StartOS) +```bash +cd image +python3 -m venv .venv && source .venv/bin/activate +pip install -e . +export SPARK1_HOST= SPARK1_USER= SPARK2_HOST= SPARK2_USER= SSH_KEY_PATH= +uvicorn app.server:app --host 0.0.0.0 --port 9999 --reload +``` +Other env vars: `BIND_PORT`, `MODELS_YAML`, `SSH_DIR`, `SSH_KNOWN_HOSTS`. + +### Tests +No pytest harness — each suite is a standalone script (that *is* how you run a single test): +```bash +cd image +python3 -m app.redaction.test_gateway # /scrub + /rehydrate acceptance; offline, no cluster needed +python3 app/redaction/test_scrub_leak.py # vendored golden-file leak test; offline +./scripts/test-audio-with-speakers.sh # (from repo root) end-to-end audio pipeline — hits the LIVE cluster +``` +Both Python suites must pass before shipping anything touching redaction. + +### Typecheck / format (TypeScript) +```bash +cd package +npm run check # tsc --noEmit — run after any startos/ edit; make x86 also runs it +npm run prettier # prettier --write startos (no semicolons, single quotes, trailing commas) +``` +Python has no configured linter/formatter — match the style of the file you're editing. + +## Layout + +- `image/app/server.py` — FastAPI entry; routers live in sibling modules (`audio_proxy.py`, `llm_proxy.py`, `embeddings_proxy.py`, `redaction_gateway.py`, `swap.py`, `health.py`, `deep_health.py`, `connectivity.py`, …). +- `image/app/static/` — the dashboard UI. +- `image/models.yaml` — vLLM model catalog bundled into the image. +- `image/parakeet_patches/` — overlay (`main.py`, `diarizer.py`) copied into the `parakeet-asr` container on Spark 2 by the "Reapply speech-model patches" action. The **only** durable way to change that container. +- `image/app/redaction/` — `scrub.py` + `test_scrub_leak.py` vendored byte-for-byte from the CRM repo (sha in `__init__.py`). The gateway around it is `redaction_gateway.py`. +- `image/spark_embed/` — Dockerfile + app for the embeddings container; built ON a Spark (ARM64, NGC PyTorch base). +- `package/startos/` — manifest, interfaces, actions (`configureSparks`, `showPublicKey`), `versions/v0_1_0.ts` (current version string + release notes). +- `docs/` — `AUDIO_API.md`, `EMBEDDINGS.md`, `REDACTION_GATEWAY.md` (consumer-facing API refs; update them with API changes). +- `README.md` (overview), `HANDOFF.md` (fresh-user install guide), `runbook.md` (ops notes), `known-issues.md`. + +## Conventions + +- Version format is `X.Y.Z:N` (`:N` = revision). Bump in `package/startos/versions/v0_1_0.ts`; **replace** the release notes — never leave old notes behind under an extra key (any unknown key fails `tsc`). +- Commit messages: `vX.Y.Z:N - short lowercase summary`. **Never add a Co-Authored-By / Claude attribution trailer.** +- Every shipped change = version bump + release notes + rebuilt s9pk (`make x86 && make install`). +- The package owner is non-technical: explain infra effects in plain English and get an explicit go/no-go before mutating the cluster. +- Pydantic request models go at **module scope**, never inside a `build_router()` body (FastAPI silently 422s otherwise). +- New external-facing endpoints get documented in `docs/` and noted in release notes for downstream app developers (Recap Relay, Ten31 Transcripts, CRM, Signal Engine consume these APIs). + +## Always / Never + +**Always** +- Confirm with the user before swap/stop/restart of anything on the live cluster. Read-only probes and dry-runs are fine without asking. +- Use the Spark's **IP** for HTTP probes — `.local` mDNS names can resolve IPv6-first and hang httpx (vLLM and friends bind IPv4 only). +- Pass `SSH_KEY_PATH` / `-i ` explicitly in scripted SSH; non-interactive shells have no ssh-agent identities. +- Make parakeet-container changes via `image/parakeet_patches/` + the Reapply action. `docker exec` / pip changes inside the container die on `docker rm`. +- Test audio endpoints with **real speech** (e.g. macOS `say`), not tones/silence — zero-token audio skips the decoder paths where crashes live. +- Send audio requests to Spark 2 **sequentially** in tests/scripts. Parallel audio requests can race (cuFFT → 503), and the single GPU serializes them anyway. +- Pin/constrain torch versions when pip-installing anything into NGC-based containers on the Sparks (ABI breaks otherwise); expect ARM64 wheel gaps and source builds (`--no-build-isolation` for torchaudio). +- Keep the redaction leak tests green against the vendored `scrub.py` after any re-vendor. + +**Never** +- Never install `cuda-python` in `parakeet-asr` to "fix" the startup warning about CUDA graphs being disabled. The warning is harmless; enabling the graph path crashes real decode with illegal memory access on this GPU/CUDA-13 stack. Leave it alone — the slow path served 11k+ requests with zero failures. +- Never edit `image/app/redaction/scrub.py` or `test_scrub_leak.py` here — change them in the CRM repo, re-vendor (`cp`), update the sha in `redaction/__init__.py`, re-run the leak test. +- Never commit owner-specific hostnames, IPs, usernames, or names into package strings, UI text, or docs — this package gets shared; use placeholders (`` style). +- Never route audio or transcripts to cloud services — speech stays on the LAN. (Scrubbed text via `/scrub` is the only sanctioned path toward frontier models.) +- Never trust `.local` hostnames inside HTTP client code (see IPv6 note above) and never assume the ssh-agent is loaded. +- Never ship a redaction change without both redaction suites passing. + +## Current state + +- **Working (v0.18.0:0, installed and serving):** swap dashboard; chat / transcribe / diarize(+chunk) / TTS proxies; embeddings + rerank + hybrid search (Qdrant); `/scrub` + `/rehydrate`; label-merge incl. dual-channel mode. Spark 2 audio stack is healthy (11k+ requests/12h, all 200). +- **In progress — Signal Engine "flakiness":** diagnosed, not a server bug — transient 1–4s unresponsiveness while the single GPU is continuously busy. Remedy is client-side; a drafted message (in-flight cap 2, hard ceiling 3 global across audio endpoints, retry-with-backoff on timeout/503) is with the owner to forward to that dev. +- **Decided, not implemented:** remote access stays WireGuard/Tailscale split-tunnel — no public interface, so no API auth built; an empirical concurrency sweep is offered but needs the owner's explicit OK in a quiet window. +- **Known limits:** `/health` blips while the GPU is busy (mitigated client-side); dual-channel can miss a quiet local word under loud remote bleed; the connectivity log misses sub-5s outages between 5s polls; diarizer caps at 4 speakers. +- **Repo wart:** HEAD's message says `v0.13.0:4` but the commit contains everything through v0.18.0:0 — per-version commits for v0.14–v0.18 are missing. Keep commit messages accurate going forward. +- **Next:** (1) owner forwards the concurrency note to the Signal Engine dev; (2) commit CLAUDE.md + ROADMAP.md; (3) run the concurrency sweep if the dev wants the measured knee; (4) add the `--memory` cap to parakeet-asr via the Reapply-patches action; (5) pick the next item from ROADMAP.md. diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..7cfe021 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,24 @@ +# ROADMAP + +Longer-term backlog, roughly ordered. An item moves to "Current state" in CLAUDE.md when picked up. + +## Near term +- parakeet-asr `--memory` cap, shipped via the Reapply-patches action (guards against swap-thrash on very long audio). +- Controlled concurrency sweep of the audio endpoints in a quiet window — replace the reasoned in-flight cap (2, ceiling 3) with the measured knee. + +## Audio quality +- Echo cancellation for dual-channel label-merge — removes the mic-bleed limit when the local user isn't wearing headphones. +- LLM "referee" pass for low-confidence label-merge speaker naming. + +## Platform hardening +- Qdrant auth (API key) + scheduled snapshots/backups. +- Observability: request metrics + GPU-busy tracking, so load questions are answered from data instead of log archaeology. +- API-key auth on Spark Control — only if public (non-VPN) exposure is ever needed; current stance is LAN + split-tunnel VPN only. + +## Throughput (only if audio load outgrows one GPU) +- Second audio worker / queueing layer; revisit which services share Spark 2. + +## Dashboard +- Per-model configurable vLLM flags editable from the UI (today: edit `models.yaml` and rebuild). +- Spark host update actions (OS/driver) from the UI. +- Open WebUI link-out integration; richer per-service detail views.