Grant
5827683a09
vLLM trips on launching Qwen3.6-35B-A3B-NVFP4 with: AssertionError: In Mamba cache align mode, block_size (2096) must be <= max_num_batched_tokens (2048). Qwen3.6 uses a Mamba-attention hybrid. The default --max-num-batched-tokens of 2048 is just under the model's required block_size of 2096. The upstream sibling recipe (qwen3.5-35b-a3b-fp8.yaml) sets it to 16384; use the same value. Earlier qwen36 swaps in this session worked because vLLM hadn't reached the Mamba-validation code path on that prior path (different attention backend pick or auto-retry). Whatever the reason, the explicit flag avoids the dance. Also documented in known-issues.md.
34 lines
2.6 KiB
Markdown
34 lines
2.6 KiB
Markdown
# Known issues
|
|
|
|
## ~~magpie-tts crash loop (Spark 2)~~ — RESOLVED 2026-05-12
|
|
|
|
**What Magpie is:** NVIDIA's multilingual text-to-speech (TTS) model, served via the NIM (NVIDIA Inference Microservices) framework — a Riva Speech Server container that converts text into spoken audio. It's the counterpart to Parakeet (which is speech-to-text / STT). When working, it exposes `/v1/audio/speech` on port 9000 and is used by clients like Open WebUI for the "read aloud" feature.
|
|
|
|
**Root cause:** The Docker volume `magpie-model-cache` (mounted at `/opt/nim/.cache` in the container) was owned by `root:root` on the host. The container drops to uid 1000 (`riva-server`) and tries to download model manifests into that cache — got `I/O error Permission denied (os error 13)` — exited 1 — restarted infinitely (24,000+ restarts before the fix).
|
|
|
|
**Fix:**
|
|
|
|
```bash
|
|
ssh <spark-user>@<spark-2-host> 'docker run --rm -v magpie-model-cache:/cache alpine chown -R 1000:1000 /cache && docker restart magpie-tts'
|
|
```
|
|
|
|
The trick is the `docker run --rm alpine chown` — it runs as root inside the throwaway container, which is enough to chown the bind-mounted volume on the host, without needing `sudo` on the host itself. After the chown + restart, magpie downloaded its ~3 GB model from NGC into the cache and came up healthy on `:9000`.
|
|
|
|
**If it ever recurs** (e.g. after wiping the volume), re-run that one command.
|
|
|
|
## Qwen3.6-35B-A3B `--moe_backend=flashinfer_cutlass` may fail on launch
|
|
|
|
This flag is Blackwell-specific. If vLLM in the container reports `unrecognized arguments: --moe_backend` or similar, edit `models.yaml` for `qwen36` and drop that flag. The swap UI does NOT auto-fallback in v0.1 — failure surfaces in the log stream.
|
|
|
|
## Qwen3.6 Mamba block-size assertion (fixed in v0.6.0:1)
|
|
|
|
Qwen3.6 uses a Mamba-attention hybrid that requires `--max-num-batched-tokens >= 2096`. vLLM's default is 2048, which trips `AssertionError: In Mamba cache align mode, block_size (2096) must be <= max_num_batched_tokens (2048)`. Fix: bake `--max-num-batched-tokens=16384` into the bundled qwen36 entry — matches the upstream qwen3.5-35b-a3b-fp8 recipe.
|
|
|
|
## Two SSH paths to Spark 1 from the laptop
|
|
|
|
`ssh <spark-user>@<spark-1-ip>` does NOT work from the laptop because the NVIDIA Sync ssh_config only has a Host entry for `<spark-1-host>.local`. Always use the `.local` hostname or `<spark-2-ip>`-style entries that ARE matched.
|
|
|
|
## Older models in `models.yaml`
|
|
|
|
The `qwen3-235b-fp8` and `qwen25-72b` catalog entries are conservative guesses for vLLM flags — they're on disk but were never the focus of this project. First launch of either may fail or be suboptimal; capture working flags here.
|