Missing tables, not the extraction model, explain the accuracy gap

I built two knowledge bases from the same 20 enterprise-agreement PDFs, using the same answer model and changing only the extraction model. Both scored 28 out of 67 ground-truth questions, 41.8%.

Across all 67 questions, not one had one KB right and the other wrong. Two models that differ don't fail in lockstep, so something upstream of both was setting the score.

Setting up the two-model test

The corpus was the alphabetical head-20 of 100 numbered synthetic enterprise agreements (Quantum Dynamics, Helios, Cyber-Synth, Apex, Bio-Gen, Stellar Pharma and friends). Both KBs were built on the same DGX Spark box, in mix mode (entity, relationship and vector retrieval). The 67 questions came from a 1,996-pair ground-truth set, stratified and filtered to that 20-document subset: 28 single-hop factual, 16 multi-hop cross-document, 14 multi-hop intra-document, 4 disambiguation and 5 hallucination-negatives, 28 easy, 32 medium and 7 hard. Scoring was deterministic, with a matcher that understands currency, ABNs and number formatting; negatives counted as correct when the model declined to answer.

Mix-mode scores for the two knowledge bases, 8 June 2026, from the KB quality baseline document listed in Sources.

The negatives are perfect on both sides, so neither graph invented an answer when the fact was absent, and two categories sit at a flat zero: sixteen out of sixteen cross-document questions and four out of four disambiguation questions, missed by both. A gap that flat means the information was missing. An extraction-quality difference would show up as a few misses each, not a clean zero.

Chasing one salary number through the pipeline

I picked one failing question and followed its answer backwards. Stellar Pharma, Level 5, Year 4 base salary, ground truth $92,527.

Step one, the source PDF: pdftotext over document 14 gives the row cleanly, Level 5 $87,931 $89,463 $90,995 $92,527.

Step two, OCR: GLM-OCR parsed each document into four type:"table" blocks with full HTML table_body, salary schedule, executive remuneration, bonus cap, project budget, every cell value present.

Step three, the stores you can actually retrieve from: this is where it fell over.

Postgres row counts for three literal values across the retrievable stores, both knowledge bases. Values that also appear in a prose sentence survive; values that live only in a table do not.

That $71,247 column is the control: a number also written in a sentence, not just a table, and it reached the chunks three times. The two table-only values scored zero everywhere; not one <table tag reached the chunk stream.

The knowledge graph showed the same gap. Entities like "Schedule 1" had reasonable prose descriptions attached ("Schedule 1 is the Salary Schedule (Indicative)... the reference document for remuneration"), a description of the table rather than its contents. Of 1,196 entities in one KB, only 11 contained a dollar figure, and those came from prose.

The documents were not truncated: ingested token counts came out at or above the raw word estimate for each PDF, so the prose was all there. The loss was specific to table blocks: GLM-OCR's table_body HTML never got inserted into the text-chunk stream, and the multimodal table path wasn't landing cells into the entity, relation or vector stores either.

Both sides gave the same refusal: the text was identical, "the Salary Schedule (Schedule 1) is referenced but the specific figures are not in the provided excerpts." The system was telling me exactly what was wrong, and I'd been reading it as a model-quality problem.

The fix, and a correction to the comparison

The fix was small. In rag_service/workers/ocr_tasks.py, _level_to_md now emits table_body for blocks of type=="table" instead of dropping them (commit 86bbb6ad). Tables go into the chunk stream, get embedded, and become retrievable like any other text.

I rebuilt a fresh KB with the fix, 17 of the 20 documents, and re-ran a 49-question subset with the three Vertex Aerospace documents excluded.

Same corpus, same questions, same models, one chunk-assembly change. Snapshot in docs/testing/kb_quality_eval/qa_results_tablefix_qwen_2026-06-08.json.

Twenty-eight questions came back, all of them table-resident values: $92,527, $107,860, $89.00, $225,000, $39.7M, 76 headcount, 28% STI, and so on.

Checking which model had actually served each run, I read /proc/1/cmdline on the box and found the vLLM container serving Qwen3.6-35B-A3B-NVFP4 under the served-model-name alias openai/gpt-oss-120b. Both sides of my "Qwen versus gpt-oss" comparison may have been Qwen wearing the other model's name badge, an untrustworthy cross-model test. The evidence is inconclusive either way: the LLM cache was off, both images were present on the box, and the entity style was near-identical.

That's annoying, and it's the least surprising explanation for zero head-to-head differences across 67 questions. It doesn't change the table finding: the tables were absent from every retrievable store in both KBs, measurable directly in Postgres, not inferred from the scores.

Two other things broke while I was in there. The TableModalProcessor crashed on GLM-OCR tables with need more than 2 values to unpack and Event loop is closed, killing three Vertex documents (the chunk path was unaffected, so those three were excluded above). A half-applied gpt-oss environment overlay had also throttled the box to RAG_WORKER_CONCURRENCY=2 and MAX_ASYNC=4: extraction ran at 3 concurrent requests and 211 tok/s, about 11 minutes per document against Qwen's native 3.3 minutes.

Retesting the ontology theory

A second theory I liked: the graph was thin because the extraction prompt worked from 8 generic preset entity types instead of a domain ontology detected from the corpus. On a later run I'd disabled auto-ontology on one KB to keep a slow phase out of a concurrency measurement, and that KB scored 79.6% against 93.9% for the ontology-enabled one, a confound I introduced myself, so I dropped it.

With auto-detect on (union priority, max_auto_detected_types=30, ontology_sample_size=80000), gpt-oss produced a 16-type domain ontology: Employee, Employer, Compensation, WorkSchedule, Classification, Leave, DisputeResolution, EmploymentAgreement, Project, Location, Termination, Organization, Policy, Person, ConsultationProcess, LegislationReference, against 8 generic presets.

Graph density over the same 20-document corpus, 9 June 2026.

The graph got denser: about 16% more entities, roughly 62 per document against 39 on the fresh-KB measurement, and the premise held at the extraction level, so I scored the answers.

On the fair slice, 19 non-tabular questions, ontology-on and ontology-off gpt-oss both scored 84% (16/19): two wins offset by two losses in the same category, reshuffling which multi-hop question it got right, not how many. Qwen held 95% on that slice regardless of the ontology change, so the gap between the two extractors here is an extraction-quality difference, not an ontology artefact.

The overall number for that ontology-on KB was 36.7%, but I'm not presenting it as a finding: the run is confounded. The clean test, re-extracting the existing table-fixed chunks in place, was blocked by an asyncpg "Event loop is closed" bug in the force_reprocess_graph write path that corrupted entity attribution (one document ended up with 620 entities, others got 1 or 2), so the accuracy run used a fresh KB instead: clean graph, no table chunks, since the fix isn't in the fresh ingest path. It scored 7% on the 30 tabular questions, measuring the table gap again, not the ontology.

I re-scored all three KBs with a lenient matcher to rule out my own scoring script as the culprit. Stripping internal spaces and expanding scale notation moved Qwen from 93.9% to 93.9%, the ontology-off KB from 79.6% to 79.6%, and the ontology-on KB from 36.7% to 38.8%, a single Fair Work Commission reference where the ground truth read AE773264 and the answer read AE 773 264. It recovered none of the roughly 28 tabular misses, since those were literal "Information not available" responses, not mis-scored values.

The first attempt failed and cost me a night: the box was running the wrong Celery worker override, rag_worker at concurrency 12 against MAX_ASYNC=20, about 27 concurrent LLM requests. The ontology-detection phase sends 80,000-character prompts; at that fan-out each one starved to roughly 30-minute latency and timed out, failing 8 documents. Dropping to concurrency 2 with MAX_ASYNC=4 (about 8 concurrent requests) took running requests from 27 down to 5 and free memory from 1.2 GiB back to 6.6 GiB, with no timeouts after. Extraction prompts run about 2K, so the over-subscription was specific to the ontology phase.

What the failing questions had in common

Line up every miss from every run and the pattern is the same: salary by level and year, executive remuneration schedules, site-allowance daily rates, bonus caps, project budgets and headcounts. All of them are cells in a matrix, not sentences.

Graph extraction over prose models who reports to whom and which clause governs a dispute reasonably well. It models a 4-by-6 grid of numbers poorly: what makes the grid meaningful is a cell's position relative to its row and column headers, and that structure disappears the moment you flatten it into a paragraph and ask a language model to describe it. You get back an entity called "Schedule 1" with an accurate description of what it's for, and not one of the numbers in it.

After a fortnight comparing extraction models on the DGX Spark, the lever was chunk assembly, worth 57 percentage points on the same corpus, more than any model swap I measured moved anything.

If tabular facts need their structure preserved to be answerable, tables need to be ingested as data: rows, columns and types, kept queryable in their own right, and linked back to the source document, rather than melted into prose on the way in. That's what Certant Strata does, and it's the next series. First article: Building a data model from PDFs instead of designing one.

How this was measured

Everything ran on one box, an ASUS GX10 (NVIDIA GB10 Grace-Blackwell, the same hardware as a DGX Spark). Questions were sampled from the ground-truth set with a fixed seed and filtered to the documents under test. Scoring was deterministic in both strict and lenient forms; the eval scripts, selection script and scored snapshots are committed under docs/testing/kb_quality_eval/.

A clean model-versus-model extraction comparison wasn't measured. The first pair carries the served-name alias problem, the second the ontology confound plus a blocked in-place re-extraction, and the 93.9% against 79.6% figure is called directional, not definitive, in its own source document. A defensible gpt-oss-versus-Qwen number needs the force_reprocess_graph event-loop bug fixed first, then the same chunks re-extracted twice with only the model changed.