Files

T

Grant 5827683a09 v0.6.0:1 - fix Qwen3.6 Mamba block-size assertion at launch

vLLM trips on launching Qwen3.6-35B-A3B-NVFP4 with:
  AssertionError: In Mamba cache align mode, block_size (2096) must be
  <= max_num_batched_tokens (2048).

Qwen3.6 uses a Mamba-attention hybrid. The default --max-num-batched-tokens of 2048 is just under the model's required block_size of 2096. The upstream sibling recipe (qwen3.5-35b-a3b-fp8.yaml) sets it to 16384; use the same value.

Earlier qwen36 swaps in this session worked because vLLM hadn't reached the Mamba-validation code path on that prior path (different attention backend pick or auto-retry). Whatever the reason, the explicit flag avoids the dance.

Also documented in known-issues.md.

2026-05-12 13:22:24 -05:00

2.6 KiB

Raw Blame History

Known issues

magpie-tts crash loop (Spark 2) — RESOLVED 2026-05-12

What Magpie is: NVIDIA's multilingual text-to-speech (TTS) model, served via the NIM (NVIDIA Inference Microservices) framework — a Riva Speech Server container that converts text into spoken audio. It's the counterpart to Parakeet (which is speech-to-text / STT). When working, it exposes /v1/audio/speech on port 9000 and is used by clients like Open WebUI for the "read aloud" feature.

Root cause: The Docker volume magpie-model-cache (mounted at /opt/nim/.cache in the container) was owned by root:root on the host. The container drops to uid 1000 (riva-server) and tries to download model manifests into that cache — got I/O error Permission denied (os error 13) — exited 1 — restarted infinitely (24,000+ restarts before the fix).

Fix:

ssh <spark-user>@<spark-2-host> 'docker run --rm -v magpie-model-cache:/cache alpine chown -R 1000:1000 /cache && docker restart magpie-tts'

The trick is the docker run --rm alpine chown — it runs as root inside the throwaway container, which is enough to chown the bind-mounted volume on the host, without needing sudo on the host itself. After the chown + restart, magpie downloaded its ~3 GB model from NGC into the cache and came up healthy on :9000.

If it ever recurs (e.g. after wiping the volume), re-run that one command.

Qwen3.6-35B-A3B `--moe_backend=flashinfer_cutlass` may fail on launch

This flag is Blackwell-specific. If vLLM in the container reports unrecognized arguments: --moe_backend or similar, edit models.yaml for qwen36 and drop that flag. The swap UI does NOT auto-fallback in v0.1 — failure surfaces in the log stream.

Qwen3.6 Mamba block-size assertion (fixed in v0.6.0:1)

Qwen3.6 uses a Mamba-attention hybrid that requires --max-num-batched-tokens >= 2096. vLLM's default is 2048, which trips AssertionError: In Mamba cache align mode, block_size (2096) must be <= max_num_batched_tokens (2048). Fix: bake --max-num-batched-tokens=16384 into the bundled qwen36 entry — matches the upstream qwen3.5-35b-a3b-fp8 recipe.

Two SSH paths to Spark 1 from the laptop

ssh <spark-user>@<spark-1-ip> does NOT work from the laptop because the NVIDIA Sync ssh_config only has a Host entry for <spark-1-host>.local. Always use the .local hostname or <spark-2-ip>-style entries that ARE matched.

Older models in `models.yaml`

The qwen3-235b-fp8 and qwen25-72b catalog entries are conservative guesses for vLLM flags — they're on disk but were never the focus of this project. First launch of either may fail or be suboptimal; capture working flags here.

2.6 KiB Raw Blame History