Five OCR engines and one enterprise agreement: what actually read the rate tables

Page 146 of an Australian disability-services enterprise agreement holds Schedule J, a pay table: eight classification groups down the left, four to nine year-steps inside each, four effective dates across the top, a weekly and an hourly rate under each. The classification label is a merged cell spanning the years beneath it, so the table's meaning sits in one HTML attribute: rowspan.

Get that attribute wrong by one and every row underneath shifts: the number stays correctly transcribed, accurate to the cent, just attached to the wrong person. That's why I tested five OCR engines against the same pages instead of trusting anybody's benchmark score.

The test corpus

The test set is six real enterprise agreements from Victorian care providers, 1,222 pages in total: Aruma at 155 pages, Home@Scope at 166, LWB at 165, Melba at 156 and Optia at 166, all born-digital with wage schedules in the back pages, plus the Victorian Public Mental Health Services agreement at 414 pages, fully scanned, with a noisy embedded OCR layer and about 56 pages of rate tables in Schedule 2. Melba has the worst stacked multi-row headers.

Two cells are ground truth, checked by eye against the rendered PDFs: Aruma p147, DDSO 2 / Year 4 / effective 1 July 2023, weekly = $1,370.25, and Victorian p288, RPN Grade 2 / Year 8 / 1 July 2023 = $1,834.20. In the overnight run over the whole corpus, 257 tables came out of raw_extracted (Aruma 36, Home@Scope 35, LWB 36, Melba 31, Optia 37, Victorian 82), and both probe cells were exact.

How a table becomes rows

What reaches us is a canonical content_list: one entry per layout element, each with a type, a 0-based page_idx, a bbox, a layout_label, and for tables a table_body holding raw HTML. Chunking consumes the synthesised markdown; the data-model layer consumes the HTML.

That HTML goes through rag_service/strata/extract.py, which runs a lenient BeautifulSoup grid parser first. pandas.read_html is the fallback rather than the default because it type-coerces: "1,250" becomes the integer 1250 and "N/A" becomes NaN, breaking the rule that raw stays raw. A markdown pipe parser sits behind that. If all three fail on one table, that table alone is marked extraction_failed and its .failed.html is preserved on disk; a single bad table never fails the document. Header inference takes up to three leading all-non-numeric rows and joins them per column with " / ", deduping empties to col_{i}.

Every backend needed its own normalisation:

These quirks are logged in .claude/skills/strata-html-table-parsing/SKILL.md.

MinerU: quick on scans, wrong about headings

MinerU is the legacy default, still the pydantic default in rag_config.py, and works well on scanned documents: the 414-page scanned Victorian agreement went through MinerU on RunPod in 7.5 minutes of OCR execution, producing 82 tables.

The failure is structural. Running the vlm-auto-engine backend on the same 155-page Aruma agreement, MinerU's region classifier tags numbered legal clauses (11.1 through 11.4) as H1 headings. The chunker believes it: section 11 shatters into five chunks, bare headings with nothing under them, orphaned clause bodies with none. Roughly 49% of that document's 405 chunks came out under 120 tokens. GLM-OCR on the identical PDF classifies those clauses as body text and emits a real #/## hierarchy, so section 11 stays one coherent 419-token chunk and the document totals 395 chunks. Retrieval over the first version is materially worse.

Docling and PaddleOCR, and why neither became the default

Docling is a premium module, and the cost is in shape. Captions are strings, not MinerU's lists or GLM-OCR's none; the content list is grouped by type, not document order, so finding a table's surrounding prose needs a page-scoped walk, not a positional one; the API and office-fallback variant can return table_body as a dict of grid cells instead of HTML. Each is a branch in our code. Parsing takes time too: MinerU and Docling run 30 to 120 seconds per document, per our multimodal configuration notes.

PaddleOCR-VL looked strong on paper: a 0.9B model, 109 languages, merged-cell table recognition, formula extraction, 11-plus chart categories, SOTA claims on OmniDocBench v1.5. Our integration analysis called it a partial replacement for MinerU or Docling: an adapter needed first, image extraction untested, metadata thinner than MinerU's (no rendered table_img_path or equation_img_path).

PaddlePaddle 3.2.2 segfaults on ARM64, which killed it for day-to-day work. PaddleOCR initialises, downloads its models, runs every line of Python, then predict() dies with SIGSEGV, exit code 139, on a blank image with simple text. Disabling textline orientation, minimal config, different image formats, none of it helped: the null-pointer dereference is in PaddlePaddle's C++ inference engine. PaddleOCR is non-functional on M-series Macs and ARM64 servers, and needs a CUDA GPU. For a team where the laptops are Macs, that is a hard stop.

GLM-OCR: right on headings, wrong on rowspans

GLM-OCR won our June 2026 parser comparison on exactly the ground MinerU lost: its PP-DocLayout-V3 layout stage classifies clauses correctly, emitting real heading hierarchy. The integration doc's comparison table records its OmniDocBench score at 94.62 against a lower MinerU figure, with 25 layout categories and a 0.9B recognition model running on 2 to 4 GB of VRAM.

Getting it fast took a rewrite. The original RunPod handler ran a single transformers model behind a threading.Lock, roughly 8 to 15 seconds per page, serial: the 155-page Aruma PDF took about 37 minutes, usually not finishing inside its 570-second poll window. Migrating to SGLang on 12 June 2026 took the same PDF to under 5 minutes, at an OCR spend of $0.465 for 155 pages.

On Schedule J, though, GLM-OCR emits over-reaching rowspan values on dense multi-group rate tables; the error cascades down the page.

It is also non-deterministic: across three re-OCR runs of the same page the errors moved around, except DDSO 1, which was wrong every time. The shift misplaced $1,270.92, and destroyed rows outright: DDSO 2 Year 1 at $33.45, and both DDSO 3 Year 1 and Year 2. Both consumers inherit the damage, retrieval through raganything/utils.py:_html_table_to_markdown and the data-model layer through strata/extract.py, and neither had any repair for it.

Baidu Unlimited-OCR gets the merged cells right

On 15 July 2026 I put Baidu's Unlimited-OCR through the same pages, a different animal: UnlimitedOCRForCausalLM, a DeepEncoder vision tower on a DeepseekV2-MoE language model, 3B parameters and about 6.6 GB in bf16, doing layout, OCR, table structure and bounding boxes in one pass.

On Schedule J it returned 28 of 28 rows correct, with the DDSO 1, 1Q, 2 and 3 rowspans exact where GLM-OCR had got all five wrong. The one blemish was DDSO 2A, emitted as rowspan="7" where 9 is correct: an under-span, all nine rows present, so a naive carry just shifts the last two rows rather than losing any. Across a six-page evaluation set (a rate table, an allowances page, a three-table page, a text-layout page, a complex table and an executive-remuneration page) every value matched ground truth, including $58,236 and $49,406 on the interpreting allowance. The complex table used repeated labels rather than merged cells, which is shift-proof by construction.

It also emits layout natively, one <|det|>TYPE [x1,y1,x2,y2]<|/det|> per element in roughly 0-1000 coordinates, compatible with our bbox_normalizer. A three-table page produced three tables; a text page produced 17 text blocks and 2 titles, inventing none. That would let us drop the separate PP-DocLayout-V3 stage: the GLM-OCR path is two-stage, crop with PP-DocLayout then run GLM-OCR per crop.

The end-to-end consumer verification on 22 July 2026 re-parsed the full Aruma document through the live endpoint and pulled out 43 tables where GLM-OCR had managed 36, Schedule J p146 perfect across DDSO 1, 1Q, 2, 2A, 3, 3A, 4 and 5, every row GLM-OCR had destroyed recovered. That run also caught three shipping gaps no unit test would have found: the ocr_worker service, which runs the parser, was missing the licence-bypass env var, so the module import died; the per-KB endpoint override was silently dropped because the two new fields were never added to the config-PUT allowlist; and every document reprocess was returning 500 from a pre-existing queue_history bug unrelated to OCR.

I could not run any of this locally. vllm-metal 0.3.0-dev supports only Qwen3-VL and PaddleOCR-VL, not a custom architecture; transformers on MPS loads the model, runs the vision tower, then generates zero tokens. CUDA-only, so it lives on RunPod serverless: cold start 6.7 minutes on a 20 GB image plus a 6.6 GB model, then about 20 seconds a parse.

Silent under-extraction in production

Separately, and more embarrassingly, I found this one in production in May 2026. Two scanned agreements in a customer knowledgebase sat at status=processed: no error, zero error or warning rows in the task log. AE530833.pdf, 30 pages, an 8.3 MB scan, had extracted 18,294 characters, about 609 per page, on legal pages that should carry 3,000 to 4,000. Many pages held 80 to 130 characters, a header crumb and nothing else. Its sibling averaged 876 characters a page, several pages at 19 characters.

The root cause is one line of definition: success in rag/document_processor.py:654 means chunk_count > 0 and no exception raised. content_length is recorded and never compared against anything. Page-count coverage was 100%, because every scanned page emitted a page-number block, so page-count reconciliation could not see the problem either. The likely mechanism was on the OCR worker side: the handler no-ops OCRClient.connect, so a worker whose GLM-OCR text server on port 5002 had died produced layout boxes with no text and reported a clean success. The PageIndex for that knowledgebase was built from the OCR output rather than the source PDF, so it inherited the same blindness.

The RunPod-side hardening shipped the same day: the start script now exits 1 if the text server never comes ready instead of carrying on layout-only, a pre-parse health gate returns ocr_text_model_not_ready, the OCR server returns 503 or 400 where it used to return HTTP 200 with an empty body, and OCR_MAX_TOKENS went from 4,096 to 32,768. Nine of nine handler tests pass.

The client-side safeguard only goes so far. _verify_extraction_complete() checks structural things: MinerU's per-page artefact exists and is non-empty, its page count equals the PyPDF2 source page count, and every source page index produced at least one content block. It raises on the permanent branch, so the document lands as failed rather than processed. There is deliberately no density check, because a characters-per-page threshold is document-type dependent and false-positive prone. Nine unit-test cases pass, it is not deployed, and the exact pattern that started the investigation still would not be caught, because every page emitted a header crumb. Catching that needs a categorical signal: pages whose only block types are page number, header and footer, which we scoped and did not build.

What we run today

Nothing here ended in a single clean winner: this is what's registered in the parser factory today.

Unlimited-OCR is the one I would pick for anything table-heavy. The global default flip is a decision I have not made yet: it runs per knowledgebase first, with a broader corpus and a retrieval regression owed before it becomes everyone's default. Alongside it is a planned deterministic, model-agnostic ordinal-reset repair for rowspans, extended to cover under-spans.

There is a sixth path now open on the on-premise side: on a DGX Spark, a single Gemma-4 vLLM call per page returns the layout labels and text in the same JSON shape, with the GPU pinned at 96% and zero PP-DocLayout model loads. That removes a 1.7 GiB per-document CPU layout stage that was capping OCR at about two concurrent documents, and it's what the air-gapped builds will use.

Do not benchmark an OCR engine on a page of prose. Find the ugliest merged-cell table in your corpus, one whose ground truth you can check by eye, and run it three times through each candidate before you trust the score.