grant/spark-control
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity
Files
9debeb4bbee4e412cb4eedc7c04d9d510f3953bf
spark-control/AGENTS.md
T

9.5 KiB
Raw Blame History

AGENTS.md

This file provides guidance to coding agents (Claude Code and others) when working with code in this repository. (Claude Code reads it via the CLAUDE.md symlink.)

Browser-based StartOS 0.4 package controlling a dual NVIDIA DGX Spark AI cluster: one-click vLLM model swaps, plus health, proxying, and APIs for speech (STT/diarization/TTS), embeddings, and redaction.

Subsystem guidance lives in docs/guides/ and loads when matching files are touched (Claude Code lazy-loads via .claude/rules/ symlinks; other agents read the guides directly): startos-package.md (build/versioning, package/**), fastapi-image.md (dev server/env/layout, image/**), redaction.md (vendoring + test gates), audio-speech.md (parakeet patches, cluster-container footguns, audio testing). Read docs/guides/audio-speech.md before touching the Sparks' containers over SSH — ops sessions don't trip the path scoping.

Inbox check: At session start, if ~/Projects/standards/INBOX.md exists, scan it for items tagged (spark-control) and surface them before proposing next steps; triage with /triage.

Stack

  • Two halves, always coordinated:
    • image/ — standalone FastAPI app (Python ≥3.11; UI on port 9999; vanilla HTML/CSS/JS).
    • package/ — StartOS 0.4 wrapper (TypeScript) that ships the Docker image as an s9pk.
  • Build host needs start-cli, Node ≥22 + npm, and Docker.
  • Cluster runtimes live on the Sparks, not in this repo (spark-vllm-docker, the parakeet/kokoro/embeddings containers). This repo is the controller; it reaches them over SSH + HTTP.
  • Sparks are ARM64 (GB10 Grace-Blackwell, sm_121, CUDA 13). Services: vLLM :8888 (Spark 1); parakeet-asr :8000, Kokoro TTS :8880, bge-m3 embeddings + Qdrant (Spark 2). See docs/ for API contracts.

Commands (headlines — details in the scoped rules)

(cd package && make x86)                                  # build the s9pk; make install sideloads (restarts live service — ask first)
(cd image && uvicorn app.server:app --port 9999)          # local dev — needs env vars, see fastapi-image rule
(cd image && .venv/bin/python -m pytest)                          # offline unit suite (launch-cmd injection, label-merge)
(cd image && .venv/bin/python -m app.redaction.test_gateway)      # offline redaction suite 1
(cd image && .venv/bin/python app/redaction/test_scrub_leak.py)   # offline redaction suite 2
./scripts/test-audio-with-speakers.sh <audio-file>        # e2e audio — hits the LIVE cluster

Layout

  • image/app/ — FastAPI app (server.py entry, routers in sibling modules, static/ dashboard UI).
  • package/startos/ — StartOS manifest, interfaces, actions, version + release notes.
  • docs/ — AUDIO_API.md, EMBEDDINGS.md, REDACTION_GATEWAY.md (consumer-facing API refs; update with API changes).
  • README.md (overview), HANDOFF.md (fresh-user install guide), runbook.md (ops notes), known-issues.md, ROADMAP.md (longer-term backlog — items move into "Current state" below when picked up).

Conventions

  • Every shipped change = version bump + release notes + rebuilt s9pk (version format X.Y.Z:N; details in the startos-package rule).
  • Commit messages: vX.Y.Z:N - short lowercase summary. Never add a Co-Authored-By / Claude attribution trailer.
  • The package owner is non-technical: explain infra effects in plain English and get an explicit go/no-go before mutating the cluster.
  • New external-facing endpoints get documented in docs/ and noted in release notes for downstream app developers (Recap Relay, Ten31 Transcripts, CRM, Signal Engine consume these APIs).
  • Doc layout: AGENTS.md is the canonical file; CLAUDE.md is a symlink to it (don't overwrite it). Subsystem guides are real files in docs/guides/<topic>.md (with paths: frontmatter); .claude/rules/<topic>.md are relative symlinks into them. A new guide = add docs/guides/<topic>.md, symlink it from .claude/rules/, and add an index line above.

Always / Never (cluster-wide)

  • Always confirm with the user before swap/stop/restart of anything on the live cluster. Read-only probes and dry-runs are fine without asking.
  • Always use the Spark's IP for HTTP probes — .local mDNS names can resolve IPv6-first and hang httpx (vLLM and friends bind IPv4 only). Never trust .local hostnames inside HTTP client code.
  • Always pass SSH_KEY_PATH / -i <key> explicitly in scripted SSH; non-interactive shells have no ssh-agent identities.
  • Never route audio or transcripts to cloud services — speech stays on the LAN. (Scrubbed text via /scrub is the only sanctioned path toward frontier models.)
  • Never commit owner-specific hostnames, IPs, usernames, or names into package strings, UI text, or docs — this package gets shared; use placeholders. Canonical set: <spark-1-ip> / <spark-2-ip>, <spark-1-host> / <spark-2-host>, <spark-user>, and generic example names (Alice/Bob).
  • Never install cuda-python in parakeet-asr — crashes real decode on this GPU/CUDA-13 stack; full story in the audio-speech rule.

Current state

  • Working (v0.21.0:1, installed and serving): swap dashboard; chat / transcribe / diarize(+chunk) / TTS proxies; embeddings + rerank + hybrid search (Qdrant); /scrub + /rehydrate; label-merge incl. dual-channel; per-Spark SSH-key copy + WireGuard VPN <ip> hardware-card badge. Spark 2 audio stack healthy. Security hardening (v0.19.0:0 — shellsafe SSH-injection guard, Qdrant path-injection, same-origin CSRF guard) shipped and stable; evidence in EVALUATION.md.
  • matrix-bridge tile (v0.21.0:1 — installed & verified 2026-06-16): "matrix-bridge" service tile (kind bot) on the Always-on services panel. Status badge (docker-state only — no HTTP health port), Restart/Stop/Start (generic /api/services path), Update (streamed git fetch && git reset --hard origin/<branch> && docker compose up -d --build via app/matrix_bridge.py, 25-min cap, fail-loud), View logs (docker logs --tail 100). Driven as a dedicated SSH user directly (no sudo -iu — spark2 has no passwordless sudo). The user is a blank-default "Configure Sparks" field (matrix_bridge_user, set to modelo); blank → service unconfigured → tile hidden (portable, no hardcoded username). Host reuses spark2_host (= 192.168.1.87 = the bot's box spark-32d0); container/dir/branch env-overridable defaults. Verified live: badge Healthy, Update (rebuild+recreate), Restart, View logs all work. Note: a fast docker restart won't visibly flip the badge (status re-checked only after the command returns, by which point it's back up — matches the sub-5s-poll-miss limit). Ops dependency: the Update button runs git fetch as modelo on the Spark, which relies on modelo's ~/.ssh/config pinning the Gitea deploy key (Host immense-voyage.local … IdentityFile ~/.ssh/id_ed25519 + IdentitiesOnly yes) — without it another key is offered first and Gitea denies (publickey). Don't remove that block. New endpoints: POST /api/matrix-bridge/update (+/{id}, /{id}/stream), GET /api/matrix-bridge/logs. Owner prereq before Update works: convert ~/matrix-bridge to a Gitea clone, and authorize the package key for modelo unless spark2_user == modelo. Then bump-build-install (cd package && make x86 && make install — restarts live service, get go/no-go).
  • Tests: offline pytest harness in image/tests/ — cd image && .venv/bin/python -m pytest (70 passing). Covers build_launch_command (incl. the shell-injection round-trip), the transcript↔diarizer label-merge, the shellsafe validators, and matrix_bridge.build_update_command (+ phase detection). Mock-heavy swap/proxy tests deliberately skipped (low ROI). Redaction + live-audio suites remain standalone scripts.
  • Signal Engine "flakiness": diagnosed as not a server bug — transient 1–4s unresponsiveness while the single GPU is busy. Client-side remedy (in-flight cap 2 / ceiling 3 / retry-on-timeout+503) drafted and forwarded to that dev (owner confirmed 2026-06-15). Awaiting whether they want the measured concurrency knee.
  • Stance (decided, not built): no public interface / no API-token auth — LAN + WireGuard/Tailscale split-tunnel only; the CSRF guard covers the browser-driven vector.
  • Known limits: /health blips while the GPU is busy (mitigated client-side); dual-channel can miss a quiet local word under loud remote bleed; connectivity log misses sub-5s outages between 5s polls; diarizer caps at 4 speakers.
  • Infra gotcha (safety): passwordless sudo is NOT configured on spark2 — design unprivileged probes for any Spark feature (the badge uses ip, not sudo wg show). spark2 sits on the starttunnel WireGuard subnet (10.59.211.6/24, survives reboot). Owner declined SSH-key rotation after the 2026-06-12 history scrub (only the key name leaked) — don't re-flag.
  • Hosting: self-hosted Gitea — remote gitea, branch master, over SSH; push after committing. (Wart: commit 8d839e3 is mislabeled v0.13.0:4 but contains through v0.18.0:0.)
  • Next: (1) matrix-bridge Phase 3 — done & shipped (v0.21.0:1, pushed); nothing outstanding unless the dev wants the optional Docker HEALTHCHECK follow-up (running-but-disconnected detection, spec §Note). (2) audio concurrency sweep — only if the Signal Engine dev wants the measured knee; needs owner OK in a quiet window. (3) Otherwise pull from ROADMAP.md: local-path/fine-tuned model support (new) or P2 tech-debt. Parakeet long-audio guard is deferred (rationale in ROADMAP).
Reference in New Issue View Git Blame Copy Permalink