Which LLM should you run on a DGX Spark? gpt-oss-120B vs Qwen3.6-35B vs Gemma-4 26B

I ran three LLM stacks on one DGX Spark for a month. Same box, same 128 GB on the sticker, same documents going in. Here's the ranked verdict: gpt-oss-120B, the model I expected to win, came third.

The hardware is an NVIDIA GB10 Grace Blackwell (mine is the ASUS GX10, same silicon, different case, so I'll call it "DGX Spark" here). It has roughly 121 GiB of unified LPDDR5X shared between CPU and GPU, fed by about 260 to 273 GB/s of bandwidth, with no discrete VRAM. The model, its KV cache, the embedding model, the reranker and the OCR model all come out of the same pool, and when that pool runs out the box doesn't get slow: it OOM-kills you.

All three landed within 14 points of each other on the same 49-question set, and the biggest accuracy gain in the study came from fixing my own pipeline, not from swapping models. What separated them was speed, memory headroom and moving parts. Ranked: Qwen3.6-35B with thinking off for most of what most people do, Gemma-4 26B if you want one model doing every job, and gpt-oss-120B only when you can hand it the whole machine.

Accuracy didn't pick the winner

All three extracted a graph from the same corpus of enterprise agreements, scored by the same deterministic scorer.

The gpt-oss and Qwen rows isolate extraction, with a common answer model reading both graphs; Gemma's row extracts and answers, the harder test, so 91.8% is a floor. The gap to gpt-oss is pure recall: its sparser graph drops facts that are in the documents.

Gemma's first run scored 42.9%, from a three-week-stale bundle image that predated three fixes we'd already shipped. Rebuilding the app layer from current source took the same model on the same corpus to 91.8%. One fix, where OCR read tables perfectly but the values never reached the chunk stream, was worth 40% to 97% on table-heavy questions by itself.

1. Qwen3.6-35B-A3B, the default for most people

35B total, about 3B active per token, NVFP4 on vLLM. Single-stream it does 119.9 tok/s with thinking off; at 128 concurrent requests it reaches 2,805 tok/s, the highest batched ceiling of the three, a 23.4x gain. The useful knee comes earlier: aggregate throughput hits roughly 533 tok/s near concurrency 32, and per-stream latency collapses past it. Set max-num-seqs 32, with a worker fan-out to match.

Turn thinking off

Thinking on, graph extraction ran at about 20 minutes per document. Thinking off, 1.4 minutes, a 12x difference, and the slow mode was also worse, yielding roughly 43 raw entities per document against 58. It spends its budget on around 3,000-token reasoning traces, truncated and retried instead of extracting. I capped the thinking-on run at two documents.

It fits with room to work

Qwen's 49 GiB of weights and KV, plus the embedding model, reranker and GLM-OCR, come to roughly 91 GiB used and 28 to 30 GiB free. Extraction here is memory-bound, not compute-bound: 16 extraction workers is the sweet spot at around 106 GiB, 24 brushes the ceiling, 32 plateaus. More on the tuning in Seven levers that took a DGX Spark from "won't boot" to 2,800 tokens a second.

2. Gemma-4 26B-A4B, the consolidation play

26B total, only 3.8B active, FP8-Dynamic on a vLLM nightly. One server here does four jobs: chat, extraction, reranking and vision OCR are all the same /v1/chat/completions call with a different prompt or an image attached.

Speculative decoding pays for itself here: MTP at k=4 took single-stream from 39.4 to 56.2 tok/s, a free 43% gain, lossless, at around 47% draft acceptance. Two caveats: MTP is FP8-only today (vLLM's NVFP4 path lacks the tie_weights the draft needs), and vision costs nothing at decode, 56.3 tok/s with images on versus 56.2 without. Batched it tops out near 1,011 tok/s at 128 concurrent, but the comfortable knee is 64, where p95 time-to-first-token is still sub-second.

The OCR trade-off

Gemma's built-in vision OCR reads a page in roughly 18 seconds, against 0.42 seconds for dedicated GLM-OCR, forty times slower. For months I quoted that as the reason to keep a dedicated OCR model for bulk ingest.

Measuring the phase instead of the page reversed the conclusion. On the same 100-document, 392-page corpus the consolidated path finished OCR at 339 docs/hr against the dedicated path's around 139: dropping the layout model removed a per-fork memory wall pinning both GLM-OCR runs to two OCR workers, and Gemma's vision path batches on the GPU instead. It loses 40x per page and wins 2.4x per phase, finishing 100 documents in 70.4 minutes, 85.2 docs/hr, measured. The gpt-oss and Qwen equivalents (roughly 5.5 and 24 to 32 docs/hr) are projections, not completed runs.

3. gpt-oss-120B, the whole-machine model

120B total, about 5B active, MXFP4 on SGLang. It's the most bandwidth-starved of the three single-stream at 61.4 tok/s, so it recovers most from batching: 2,621 tok/s at 128 concurrent, a 42.7x gain on identical silicon. That argument in full is Your DGX Spark isn't slow. You're testing it wrong.

One model per job means four resident models, and with gpt-oss underneath them the box sits at 111 to 114 GiB used, 1 to 7 GiB free. Getting it to ingest at all took seven attempts; the fix was shutting off the admin UI, chatbox, sandbox, ml_worker, flower and the reranker (15 to 20 GiB of services), after which it managed six documents in 33 minutes. That's 0.18 docs/min, four times slower than Qwen, and two concurrent ingests is enough to risk an OOM.

gpt-oss-120B leaves 1 to 7 GiB free with the full stack up. That's not something you can tune around; it just OOMs. On this box, usability comes down to memory headroom more than tokens per second.

What actually ranks them: active params and free memory

Single-stream decode here is arithmetic: tok/s ≈ bandwidth ÷ active bytes per token. Active parameters matter, total ones don't: all three models above are Mixture-of-Experts designs activating 3 to 5B out of 26 to 120B. The one dense model I tried, a 31B Gemma activating all 30.7B every token (around 15.4 GB of reads per token at 4-bit), measured around 7 tok/s and was disqualified on the spot. The MoE variant runs 4 to 5.5x faster on the same box.

Where the memory goes, by stack, measured at idle and peak on the same box.

What I'd actually set

The Qwen recipe I actually run:

--max-num-seqs 32   # match your worker fan-out
rag_worker=16       # ~106 GiB sweet spot; 24 brushes the ceiling
ocr_worker=8
graph_merge_worker=1            # 4 only for diverse/multi-tenant data
MAX_ASYNC=8                     # merge fan-out: 59 s -> ~15 s
GLMOCR_PIPELINE_MAX_WORKERS=8   # any batching vision-OCR server
swap >= 48 GiB                  # a 61 GB model double-buffers to ~115 GB on load

Two of those look wrong until you've been bitten. Merge concurrency of 1 seems absurd until you ingest a corpus where every document says "Base Salary": four merge workers produced 217 lock timeouts, one produced zero. Pushing max-num-seqs past the sweet spot, 64 to 160, thrashed OCR roughly 3x too, because the GPU was already saturated on prefill (more in Most people tuning local LLMs optimise the wrong half of the problem).

How this was measured

One box, no cluster: a single GB10 with Prometheus, node-exporter and the DCGM GPU exporter feeding a Grafana dashboard, plus a benchmark script running five prompts at temperature 0, with warmup and repeats. The accuracy test set was held constant across stacks, same corpus, same 49 questions, same deterministic scorer, so the ranking turns on speed, memory and ops. Documents-per-hour is the shakiest row: only the Gemma run was driven to completion on current code, so treat that figure as directional and the rest as measured.

Certant is built so deployment follows the compliance team: cloud, your own infrastructure, or fully air-gapped, and a DGX Spark is the shape of box a regulated customer can put in their own rack. Qwen3.6-35B with thinking off is my pick for 90% of cases: it serves a team, ingests their documents, answers with citations, and never touches a network.