Why Qwen3.6 with thinking off became our DGX Spark default

For the first stretch of owning this box, the language model alone ate about 74 GiB of the DGX Spark's 121 GiB pool, leaving 1 to 7 GiB free. A 20-document ingest needed the admin UI, chatbox, sandbox, ML worker, Flower and reranker all shut down to free 15-20 GiB, and still peaked at 114 GB with swap taking the overflow. That was the seventh attempt: six earlier ones failed on two straight OOM kills (Exited 137), GLM-OCR unable to get its ~6 GB alongside gpt-oss, a knowledgebase-API disruption and a two-phase handoff gap.

Swapping the language model to Qwen3.6-35B-A3B in NVFP4 gave back about 25 GiB and ended all of that. A single reasoning flag, set on the wrong code path, then cost me a day and turned out to be worth 12x on ingest throughput.

The memory maths

Qwen3.6-35B-A3B-NVFP4 runs on vLLM at --gpu-memory-utilization 0.45, a fraction of total unified memory (the trap I wrote about in "128GB of VRAM" is the most misread spec on the DGX Spark). The grant is about 54 GiB; the measured resident figure is ~49 GiB, against gpt-oss's ~74 GiB. Weights are a small part: NVFP4 W4A4 is about 17.5 GiB, plus roughly 1.5 GiB for the DFlash draft model, the rest going to the KV pool, CUDA graphs and activations.

The whole stack at steady state, measured on the box on 3 June 2026:

The KV pool is small for a 256K-context model because only 10 of Qwen3.6's 48 layers are attention layers; the rest are Gated-DeltaNet linear layers with fixed state. One full 262,144-token sequence at FP16 costs 12.57 GiB of KV, which is also why --gpu-memory-utilization 0.29 refuses to start: the grant of 35 GiB minus about 30 GiB of overhead leaves only 4.7 GiB of KV, and vLLM's _check_enough_kv_cache_memory says no, so I run at 0.45.

The airgap model manifest also came down, from about 84 GB in the gpt-oss era to roughly 34 GB across six models; purging stale weights took the Hugging Face cache from 97 GB to 36 GB. That headroom is what lets OCR, embedding and merge run at real concurrency instead of starving each other.

The 189-second summary

The graph-merge worker on the bundle was pinned at --concurrency=1, so I raised it to 4 (and briefly 16), which caused a lock-timeout storm. On a 100-document corpus of highly overlapping HR and remuneration PDFs, where every document mentions "Superannuation Guarantee" and "Base Salary", merge@4 logged 217 timeout lines, backed the graph_merge_queue up to 32, and got 9 documents out of 100 merged in 40 minutes. Dropping back to merge@1 gave zero timeouts.

I treated it as a concurrency problem for about a day before finding the mechanism: Certant's merge takes a per-entity Redis lock wrapping read, summary and write together, with the LLM call inside the held lock. Extraction fans out across chunks and shares the server's batched throughput; a lock-serialised summary issues one sequence at a time, at single-stream speed, around 71-80 tok/s, while queueing behind extraction for the same 32-sequence budget. Each summary also re-reads the entity's whole accumulated description and, past a size threshold, runs an iterative map-reduce, so a hot entity costs several calls per occurrence.

The same code against a fast cloud model at merge@10, higher concurrency, merged 100 of 100 documents with zero timeouts; the slower local model at merge@4 stormed again, confirming it.

One flag fixed it: Qwen3.6 is a reasoning model, and it was thinking during merge summaries. Adding a per-call enable_thinking override with summaries forced to False took a summary call from 189 s to about 4 s, and the timeout storm disappeared. Raising the merge fan-out (MAX_ASYNC) to 8 on top of that took the summary wall time on a queue drain from 59 s to about 15 s.

I never counted reasoning tokens per merge summary, so I don't have the exact split. During the airgap bring-up, I asked gpt-oss to reply with exactly two words: 81 prompt tokens, 80 completion tokens, of which 70 were reasoning and 10 visible. That's roughly the shape of what was happening inside each held lock during the 120-second timeouts.

The "217" is a count of timeout log lines, not distinct entities, since Celery retries re-emit; and the original per-run logs have since rotated, so these figures come from the recorded test report rather than a fresh re-derivation.

What thinking cost across a whole ingest

Once the merge path was fixed, the question was whether extraction should run thinking-on at all. I ran the same 20-document HR corpus through three configurations at rag@16, merge@1, ocr@8, end to end from OCR through extraction to merge.

Thinking on ran at roughly 20 minutes per document against 1.4 minutes with it off, the 12x figure; I capped the run at 2 documents once the pattern was clear. It also produced fewer entities per document, about 43 against 58: the model spent its budget on reasoning traces of around 3,000 tokens, often truncated and retried, instead of on extraction.

The arithmetic backs this up. A document here is about 100 chunks, so at gleaning 1 that's roughly 200 extraction calls. Thinking on, at ~2,000 output tokens per call, that's about 400K decode tokens per document, which against a 533 tok/s ceiling works out to 4-5 documents per hour. Thinking off, at ~500 tokens per call, it's nearer 100K, so 15-19 documents per hour once prefill is counted; prefill isn't free (each call carries 1-3K prompt tokens) and pulls the real rate toward the bottom of those ranges.

Against the gpt-oss baseline's 112 normalised-unique entities, Qwen with thinking off recovered 77, or 68.8%, at roughly 4x the ingestion speed and a far denser graph across the fuller corpus. That percentage is an exact-string lower bound, so semantic equivalents such as "CEO" against "Chief Executive Officer" don't count. gpt-oss also only managed 6 documents to Qwen's 20, for the memory reasons above, so the comparison is partial.

Worker concurrency, from 2 to 16

The rag_worker extraction pool was hardcoded to 2 for a reason: in the gpt-oss era, raising it OOM-thrashed the box and collapsed throughput from 92 tok/s to 18. With 28-30 GiB free, that cap stops being necessary, so I raised it to 8 and swept it properly.

Extraction on the GB10 is memory-bound. The model server scales to 128 concurrent requests without errors, but each in-flight document holds chunks and embeddings in the same unified pool, so the pipeline tops out at rag@16 to rag@24 and rag@32 gains nothing. OCR went from 2 to 6 during the session and settled at 8, about 9 documents a minute, keeping GLM-OCR ahead of extraction without stealing its GPU. Merge stays at 1 for a single overlapping knowledgebase, 4 for diverse or multi-tenant work.

The rule of thumb on the DGX now is to keep rag_worker × MAX_ASYNC at roughly 32, matching vLLM's sequence ceiling; past that, extra concurrency adds queueing, database contention and tail latency instead of throughput.

Where extra concurrency stops helping

The RAG-shaped sweep on gx10, with DFlash on and utilisation at 0.45, ran 82 tok/s aggregate at concurrency 1, then 164 at 4, 232 at 8, 388 at 16, 533 at 32, and 482 at 48. The compute peak is at 32; past it, per-stream speed collapses while the aggregate plateaus and drops. Single-stream sits at 78-80 tok/s, the hardware ceiling on this box, confirmed by building the Spark Arena reference image and running their exact NVFP4 recipe on our box: 77 tok/s, 3% ahead of the published recipe.

--max-num-seqs was at 16, and the KV pool at 0.45 holds 211-222K tokens, room for roughly 170 short RAG requests. Raising the sequence cap from 16 to 32 lifted peak aggregate from 388 to 533 tok/s, a 37% gain at no memory cost, and it went into the bundle, though that only pays off when the workload drives more than 16 concurrent requests.

A separate load test, thinking off with a 380-token prompt and 512 max tokens, kept climbing to 2,805 tok/s aggregate at 128 concurrent requests with zero errors, at a p50 latency of 63.2 s against 4.8 s single-stream. That 2,805 figure is the serving ceiling for short generations, used in Your DGX Spark isn't slow. You're testing it wrong.; the 533 knee is what an extraction workload with long outputs and merge behind it sees.

The recipe we shipped

The customer bundle serves the model like this, on vllm/vllm-openai:cu130-nightly:

vllm serve RedHatAI/Qwen3.6-35B-A3B-NVFP4 \
  --served-model-name qwen3.6-35b openai/gpt-oss-120b \
  --max-model-len 262144 \
  --max-num-batched-tokens 32768 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.45 \
  --quantization compressed-tensors \
  --attention-backend flash_attn \
  --kv-cache-dtype auto \
  --enable-prefix-caching --enable-chunked-prefill \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-35B-A3B-DFlash","num_speculative_tokens":6}'

--attention-backend flash_attn is the only backend that works on sm_121: fa3 crashes, FlashInfer returns garbage, TensorRT-LLM errors out, and it rejects --kv-cache-dtype fp8, so KV stays FP16. DFlash is worth +67% single-stream (65.8 against 39.5 tok/s, mean acceptance length 3.39), lossless since the target model verifies every drafted token. The dual --served-model-name avoids a 404: knowledgebase config overrides the model-name environment variable and existing knowledgebases still ask for openai/gpt-oss-120b, so serving both names sidesteps it without touching per-knowledgebase config.

On a later run at --gpu-memory-utilization 0.55 (about 64 GB), the reranker CUDA-OOMed at startup: the LLM, embedding model, app containers and an in-progress database import together left under 2 GB free. It self-healed once memory freed up, which is why 0.45 stays the default.

What we have not measured

Several numbers in the capacity model are estimates rather than measurements. The in-flight ceiling of 24-30 documents assumes 0.5-1 GiB of working set per concurrent document, and per-document RSS growth hasn't been measured. Same-knowledgebase merge contention on Postgres and AGE hasn't been characterised, nor has KV eviction at 32 simultaneous extraction contexts, nor the exact prefill share of a real extraction call, which is why the documents-per-hour figures are ranges.

If you're setting up a GB10 for document work, run Qwen3.6-35B-A3B-NVFP4 at 0.45 utilisation with --max-num-seqs 32, thinking off for extraction and forced off for merge summaries, extraction workers at 16, OCR at 8, merge at 1 while your corpus overlaps. That leaves 28-30 GiB of headroom and a stack that runs without offloading anything.