Asking for JSON halved our extraction recall. JSON Lines fixed it.

We switched knowledge-graph entity extraction from a delimited-tuple format to JSON, the default choice for structured output from a model. Extraction got worse: on the Qwen model our appliance runs, the graph came back with about a third of the old relations, and it took weeks of measurement to work out why.

The cause was the shape of the container: ask for one JSON object and you're asking for a closing brace, and once the model produces something well-formed it wants to close it, ending generation early. Drop the enclosing array, ask for one compact JSON object per line, and the same model on the same prompt keeps going. On gpt-oss-120b, our production extraction model, directed relation recall went from 19 on the delimited baseline to 50-56 on JSON Lines, and undirected recall on the attribution-sensitive subset went from 35 to 68-77 (commit 4ac01f1d, 28 June 2026).

Why we wanted out of delimited tuples

Our extraction prompt asks the model to emit lines shaped like this, then a terminator when it has finished:

entity{tuple_delimiter}NAME{tuple_delimiter}TYPE{tuple_delimiter}DESCRIPTION
relation{tuple_delimiter}SOURCE{tuple_delimiter}TARGET{tuple_delimiter}KEYWORDS{tuple_delimiter}DESCRIPTION
{completion_delimiter}

Inherited from LightRAG, it works well enough on frontier models, but small and MoE open-source models fill or mis-case the delimiter, drop the leading entity or relation literal, or emit the wrong number of fields. We already carry a "Delimiter Usage Protocol" block in the prompt, a fix_tuple_delimiter_corruption regex in utils.py, and _llm_call_with_completion_retry, up to three retries when the completion delimiter never arrives: three defensive mechanisms for one output format, a signal it's brittle.

Our self-hosted and appliance product runs extraction on open-source models: gpt-oss-120b on DeepInfra, Qwen3.6-35B-A3B on a DGX Spark under vLLM. A customer with an air-gapped box can't fall back to a frontier API, so a sparse local graph degrades the thing that makes us better than plain vector RAG. LightRAG's own code comment says JSON mode "significantly improves extraction quality and compatibility with smaller models", and both our serving stacks support guided JSON decoding natively: a free win on paper.

Our success check missed it

Our ingestion success check is effectively chunks > 0. A document can parse, chunk, embed and be marked processed with a hollow graph behind it and nothing logged, the same blind spot already documented for OCR under-extraction.

From outside, it looked like a knowledge base that answered lookup questions fine and went vague on anything relational. Internally, on Qwen3.6-35B-A3B, it was a relation count of around 10-13 records where the delimited path streamed around 37.

Measuring extraction: 28 gold edges and a variance problem

We built a scorer first: no defensible way existed to compare two runs, and every claim in this space, ours and upstream's, was a single-sample assertion. The pieces, committed alongside the fix:

Two metrics matter below. dirR is directed recall: the gold edge recovered with the correct source-to-target direction. S-undR is undirected recall on the attribution-sensitive subset. When S-undR is low, the model has anchored entitlements on a section heading or the most-mentioned organisation rather than the party that holds them, so the pair is never connected.

Extraction turned out to be high-variance run to run, even at temperature 0, once you bust the LLM cache: same model, same prompt, same document, three runs.

Cache-busted, three samples per cell, reasoning off, max_tokens=8000, scored against the 28-edge gold set.

The extreme case is the bottom-right cell: attribution recall on gpt-oss-120b in JSON mode ranged from 18.8% to 81.2% across three runs on identical input. Any single-sample comparison here is noise, ours included ("JSON is a regression") and upstream's ("JSON helps small models"). Everything below is a three-sample aggregate.

The rig surfaced two bugs of our own: an early run hardcoded enable_thinking=False for Qwen, a regime production doesn't use, and another used max_tokens=8000 against a production value of 65535, so with reasoning on, Qwen's chain of thought ate the budget and the JSON truncated before a single relation appeared. We logged rel=0 with 2,229 characters of raw output and briefly believed JSON mode was broken outright.

The rig also turned up a lever we weren't looking for: reasoning off roughly doubles Qwen's delimited directed recall, from 26.2 ± 9.4 to 52.4 ± 1.7, and takes S-undR from 43.8 to 60.4. Production keeps reasoning on, so it sits in the worse, noisier regime, a choice made on one document and not revisited.

Ruling out everything except the shape of the output

The next job: work out which part of "JSON" did the damage. Every suspect got its own cache-busted, multi-sample run.

The missing-scaffolding row cost us the most time. Our JSON prompt was a stripped-down port of upstream's, missing its output-limits section, its typed entity-guidance block and its section-context rule. Restoring them looked like the answer for about a day, until we ran the verbatim upstream prompt as a benchmark variant and it came out worst of the five, below our own broken production JSON:

Three cache-busted samples per variant on Qwen3.6-35B-A3B, gleaning at 1 except where noted.

Upstream validated their JSON prompt on Qwen 2.5 72B, a dense model, not the 3B-active MoE we were running it against, and ours failed the test.

The exhaustion variant held up until we ran a precision audit on it: it ties the delimited baseline on F1 at 74.4, but buys its recall with 77 edges against delimited's 40, gives up 7.6 points of precision, and most of the extra edges are the same fact triangulated several ways.

A dissociation broke the deadlock. Two variants each remove one property of verbose-array JSON: compact positional arrays drop the field-name overhead and keep the enclosing array, JSON Lines drops the enclosing array and keeps a per-line key. Compact recovered nothing. JSON Lines recovered on every model tried:

Three cache-busted samples per cell, direct to DeepInfra, scored against the same 28-edge gold set. Cells read dirR / S-undR / relation count.

The count column is the cleanest signal in the study: verbose-array JSON emits the fewest relations on every model, below the 28 edges in the gold set, and JSON Lines raises the count every time, by 12 to 23 records. We can't prove the mechanism inside the model, only the dissociation: removing the array recovers extraction, removing the verbosity alone does not.

Severity is gated by the model: the verbose-JSON penalty is worst on the small-active-parameter MoE, milder on the 72B, and on the 27B dense model it inverts, verbose JSON beating delimited there. The 27B dense running JSON Lines was the best cell in the study, on a smaller model than the 35B MoE our DGX box runs.

What JSON Lines did

The fix is small: the entity_extraction_use_json path now emits one compact object per line, with no enclosing array:

Entity line:        {"e": ["<name>", "<type>", "<description>"]}
Relationship line:  {"r": ["<source>", "<target>", "<keywords>", "<description>"]}

operate.py drops response_format on this path, because json_object forces exactly one object and defeats the point. Parsing goes through _coerce_extraction_json into _parse_jsonl_extraction, which reshapes the lines into the same dict the verbose-JSON parser produced, so ontology handling, edge weighting and cache rebuild stay byte-identical downstream. Delimited stays the default; use_json=True now means JSON Lines.

On gpt-oss-120b, JSON Lines scored dirR 50-56 and S-undR 68-77 against a delimited baseline of 19 and 35 on the same gold set. Across the four models tested it matches or beats delimited on three; on the 35B-A3B MoE it brings attribution to parity while directed recall stays about 8 points under delimited.

We then ran the whole pipeline on a real document: a 405-chunk enterprise agreement, ingested twice into two knowledge bases with everything else held constant. Extraction breadth went from 2,427 entities and 4,014 relations on delimited to 2,865 and 3,750 on JSON Lines, an 18% gain in entities against a 7% drop in relations. Fourteen hand-authored factual questions were then scored against each knowledge base with RAGAS under two independent judge models:

RAGAS scores on the 405-chunk agreement, same questions and same judges across both knowledge bases, so the deltas are the meaningful part.

Deterministic answer correctness, with no LLM judge involved, was 11 out of 14 for delimited and 14 out of 14 for JSON Lines, though delimited's three misses were phrasing mismatches ("one extra week" against "additional week"), not errors. That table reads as parity with a small precision edge, but fourteen questions with deltas of 0.03 to 0.07 sits inside the noise, so I wouldn't call it a retrieval win. The result we trust is extraction breadth, where the regression was.

What we did not measure

The model-by-format grid is one document, and some individual samples scored zero. The directional claims, dense beats MoE at this size, JSON Lines raises the record count on every model, held across every sample; the exact point deltas need a second corpus before I'd put them in a contract.

Precision was audited on the Qwen variants only, so the gpt-oss-120b JSON Lines numbers are recall-side. We didn't measure the cost or latency delta against delimited, though extraction is billed per token on the ingest path. The reasoning-off finding for Qwen is unvalidated on a second document and not shipped.

What I would change in your own prompts

For a long list of extracted records, ask for one record per line rather than one enclosing object or array: you keep clean parsing, lose the closure pull, and it's easier to merge across gleaning passes. Turn response_format: json_object off too, since the constraint that guarantees valid JSON also guarantees a single object.

Never trust one extraction run. Ours swung 18.8 to 81.2 on the same input at temperature 0. Three samples with a mean and a standard deviation is the minimum before you conclude anything, cheap next to the week you'd spend chasing a difference that was never there.

Give reasoning models a token budget that covers the thinking plus the payload. A truncated structured output looks exactly like a model that refused to extract.

On a fixed memory budget, try a dense extraction model before a bigger sparse one: Qwen3.6-27B dense beat the 35B-A3B MoE on every metric here while being the smaller model, the opposite of what the parameter count suggests.

To reproduce any of it: the scorer, the gold set and the variant runner are in the repository under rag_service/. _run_variant.sh runs three cache-busted samples of any format and model pair and prints the aggregate table.