The Gemma-4 experiment: one model for chat, OCR and reranking

A Certant deployment on a DGX Spark normally loads four models: a chat and extraction LLM, Qwen3-Embedding-4B for vectors, Qwen3-Reranker-0.6B for reranking, and GLM-OCR with PP-DocLayout-V3 for parsing. All four share about 121 GiB of unified memory: what one model takes, the rest cannot use. Gemma 4 arrived in April 2026, multimodal and long-context, and the question was whether one model could do all four jobs.

After a fortnight of measuring on the box, three of the four folded into one Gemma-4 server: chat, extraction and reranking. OCR kept its own dedicated model.

The idea, and why it fits this hardware

The GB10 gives roughly 121 GiB of unified LPDDR5X at 260 to 273 GB/s. Single-stream decode is memory-bandwidth bound, so tokens per second tracks bytes read per token, not parameter count (physics covered in Your DGX Spark isn't slow. You're testing it wrong.).

The candidate was Gemma-4 26B-A4B, a Mixture-of-Experts model with 26B total parameters and 3.8B active per token, served on vLLM in FP8-Dynamic with the vision tower loaded. Reranking became a FastAPI adapter presenting the SGLang rerank contract and scoring documents through the chat endpoint; OCR became a second adapter answering the OpenAI vision calls the parser makes. Neither adapter holds a model, so both cost near-zero memory and kept the existing Docker aliases (vllm_llm, vllm_reranker, glm_ocr); nothing under rag_service/ or agent_api/ had to change to test it.

Embeddings did not fold in. A generate-mode vLLM server will not emit pooled embedding vectors (it 404s; SGLang tells you to add --is-embedding), and prompting Gemma to write an embedding as JSON produced plausible-looking but wrong numbers: similar texts scored cosine 0.14, unrelated texts 0.69, inverted. Qwen3-Embedding-4B stayed, keeping the 2560-dimension vectors intact and avoiding re-embedding.

With the chat server and the Qwen embedder both resident, steady state was 54.5 GiB plus 11.4 GiB, about 66 GiB of the 121, leaving roughly 55 GiB free: the most headroom of any stack on this box.

The dense 31B detour

I started with the wrong model: assuming a denser model would be smarter, Phase A stood up Gemma-4 31B dense and ran the full document pipeline through it. It worked, but too slowly to use.

The 31B reads 30.7B of its parameters per token. At roughly 0.5 bytes per parameter that is about 15.4 GB of traffic per token, and 260 GB/s divided by 15.4 GB gives about 7 tokens a second. Measured: 7.1 tok/s. Quantisation does not help, since the arithmetic already assumes a 4-bit-ish footprint; speculative decoding does not either.

OCR took roughly 21 minutes per document, and a grounded RAG query took 234 seconds. The 26B MoE on the same knowledgebase, embeddings and query came back in 61 seconds with highlighting on, a 3.8x improvement. With semantic highlighting off, the FP8 plus speculative-decoding config answered in 10.7 seconds against the NVFP4 config's 25.3 seconds.

Single-stream decode measured with rigorous_bench.py, five distinct prompts, warmup, five repetitions, temperature 0, on vLLM with the GB10's --moe-backend marlin path for NVFP4. From the Gemma-4 GB10 benchmark report.

The E4B row is the smallest model in the family, yet the slowest of the MoE configs: bf16 reads more bytes per token than FP8.

Speculative decoding on legal prose

Multi-token prediction was the free win, though I wasted embarrassing time misreading it as a missing capability on sm_121 silicon. Our vLLM image, cu130-nightly 0.19.2, answered every MTP attempt with NotImplementedError: Unsupported speculative method: mtp. It was just a stale image: Gemma-4 MTP merged in vLLM v0.21.0, and pulling vllm/vllm-openai:nightly (0.22.1rc1.dev491, CUDA 13.0) fixed it, no build needed. A related dead end cost another hour: the draft model needs a newer transformers than the image ships, and pip install "transformers @ git+..." fails inside the container because git isn't installed.

Once it ran, MTP at k=4 took the 26B FP8 from 39.4 to 56.2 tok/s single-stream, a 43% gain, lossless because every drafted token is verified against greedy decoding.

On legal prose the model accepts about 47% of drafted tokens overall, with a per-position breakdown of 75%, 52%, 35%, 26%, averaging about 1.87 accepted tokens per step out of four drafted.

DFlash at k=15 reached 8% acceptance on general text and landed at 37.9 tok/s, below the 39.4 no-speculation baseline.

Speculative decoding is FP8-only for now: NVFP4 through vLLM's modelopt path has no tie_weights implementation, and the shared-embedding draft path needs it, so any NVFP4 plus draft combination dies at load. Vision coexists with MTP at no cost, 56.3 tok/s with the vision tower enabled against 56.2 text-only, CUDA graphs on, no --enforce-eager needed.

OCR still ran a second model

PaddleOCR's PP-DocLayout-V3 was running client-side, finding page regions before handing each to Gemma for text. Tracing rag_service/glmocr_local/parser.py turned up something worse than one copy per worker: parse_pdf opened the parser context per PDF, so the ~1.7 GiB layout model loaded and freed once per document, in every ocr_worker prefork, on the CPU, enable_layout hardcoded true.

With about 19 GiB free after the Gemma and embedding stack, 1.7 to 1.8 GiB per fork capped OCR at around 2 concurrent workers; raising the worker count made things worse, since duplicated layout models spilled into swap and thrashed.

The fix: ask Gemma for the boxes too. One vLLM call per page returns a JSON array of {index, label, content, bbox_2d}, matching what PP-DocLayout plus per-region OCR produced, so chunking, the table path and PDF overlays kept working. A new gemma_vision parser and one enum value were the whole integration. Validation on the box: zero PaddleOCR loads, bounding boxes normalised 0 to 1000, real markdown headings where the legacy path emitted none, and a three-page salary schedule producing about 42 content items, including a table with its $340,000 and 35% cells intact. Per-fork memory in ocr_worker went to roughly zero, so OCR concurrency became a tuning knob instead of a wall.

The OCR client had also been pinning itself to one in-flight request, a holdover from a single-stream server locked around generate(), wrong for continuous batching. Lifting GLMOCR_PIPELINE_MAX_WORKERS and OCR_WORKER_CONCURRENCY to 8 took per-document OCR from 166 s to 110 s, peak batch from 4 to 57 of 64, and OCR-only throughput from about 87 to 222 documents an hour, GPU going from idle-between-pages to 96% busy.

Fifty documents, 38 minutes, GPU pinned

With OCR unblocked, I ran 50 numbered legal agreements through the pipeline (OCR, embed, graph extraction, merge) on the tuned stack: Gemma-4 26B FP8 plus MTP with vision on, --max-num-seqs 64, OCR workers at 8 by 8, RAG workers at 6 with MAX_ASYNC 8, Qwen embeddings alongside.

Full-run figures from the 50-document benchmark, via Prometheus, node-exporter and the DCGM GPU exporter.

Two regimes emerge in the throughput panel:

The GPU was the bottleneck throughout, busy on prefill rather than decode, against a synthetic decode ceiling of about 1,011 tok/s aggregate at 128 concurrent that ingest never gets near: each page carries thousands of vision tokens, each call a long chunk context. Decode-side tuning that wins the chat path (MTP, a bigger max-num-seqs) barely helps a document queue. More on that in Most people tuning local LLMs optimise the wrong half of the problem.

Speculative-decode acceptance was workload-dependent: around 98% during OCR, where the model emits structured markdown, and around 80% during extraction, where entity and relation output is freer-form, both well above the 47% measured on chat-style prose.

The extraction tail filled only 9 of 64 batch slots though the workers are configured for 48 concurrent calls: a single knowledgebase's graph merges serialise behind a per-knowledgebase lock, so documents queue at merge rather than extract in parallel. Fixing the OCR client bought about 2.5x on the OCR phase alone, but only about 10% end to end (72 to 79 docs/hr on comparable runs).

What consolidated OCR costs

Gemma-vision OCR on this stack ran at roughly 18 seconds per page (166 s per document on the 25-document run, 176 s on the 50). A dedicated GLM-OCR server on the same box does 0.42 seconds per page, a difference of about 40x, since the consolidated path spends vision-prefill compute per page, unlike the small, purpose-built dedicated model.

For a serving workload the trade is fine: queries are decode-bound, the memory saving is large, and an occasional upload gets parsed at 18 s a page on a GPU otherwise idle. For bulk ingest it is a bad trade: 40x on the pipeline's first stage sets the ceiling for everything behind it.

Keep Gemma-4 26B FP8 plus MTP for chat, extraction and reranking. Pair it with a dedicated OCR server when there's a corpus to load rather than a trickle of uploads.

What I would run

# vLLM, Gemma-4 26B-A4B, one server for chat + extraction + rerank + vision
#   image: vllm/vllm-openai (0.22.1 nightly or later; pin a digest for production)
#   weights: RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic
#   draft:   google/gemma-4-26B-A4B-it-assistant   (use -it, not base)
--max-num-seqs 64          # 16 was the cap hiding the real ceiling
# speculative decoding: method mtp, num_speculative_tokens 4
# vision: leave enabled, it costs nothing at decode time

# embeddings stay on their own server
#   Qwen/Qwen3-Embedding-4B, 2560-dim, SGLang --is-embedding

# ingest-side worker knobs (.env)
GLMOCR_PIPELINE_MAX_WORKERS=8
OCR_WORKER_CONCURRENCY=8
RAG_WORKER_CONCURRENCY=6
MAX_ASYNC=8

Three things I have not measured, and would check before a customer deployment. Multi-needle long-context recall on the MoE, where dense models are typically stronger (our probe was single-needle, about 32K). Tool-call streaming under MTP has an open vLLM issue about dropped first arguments, and our stack uses tool calls. Spatial fidelity of Gemma's bounding boxes against PP-DocLayout, which matters for pixel-exact PDF overlays. The rerank adapter also returned uniform scores (all 1.0, then all 0.8): the listwise scoring prompt is not yet producing discriminating per-document scores, so reranking behaves as top-n truncation rather than a reorder. Retrieval and generation stayed grounded and citation-backed regardless. Reranking works; whether it reorders well is still open.

How this was measured. All figures come from one ASUS GX10 (NVIDIA GB10, ~121 GiB unified memory), instrumented with Prometheus, node-exporter and DCGM GPU exporter feeding Grafana. Token-speed numbers come from a benchmark script at temperature 0 with warmup and repetitions; ingest numbers from a 50-document run of the pipeline. Documents per hour is within-run only, since corpora differ (small multi-page agreements, about 3 chunks each), and answer quality was scored separately, in Which LLM should you run on a DGX Spark? I care about this commercially because Certant has to run where the compliance team says, and for some customers that is one box inside their building, no internet. Every model loaded there is memory spent, so "how few models can we get away with" sets what the box can be sold to do.