The paper quantifies formatting-cue loss on 76 policy documents, but lacks public code and independent reproduction.
Study finds document converters erase cues that policy agents need
2026-10-22Frontier watchWATCH
A study of 76 real policy documents finds that common ingestion pipelines drop meaning encoded in highlights, colour and other formatting. A deterministic OOXML scorer reports micro-F1 falling from 0.690 with preserved cues to 0.150 with Docling and 0.075 with plain text across six answering models. The paper provides a concrete failure mode but no public code or independent reproduction.
What this means for you
For RAG and document-agent teams, text extraction can silently change policy meaning before retrieval or reasoning begins. Validation needs layout-aware test cases and provenance for author emphasis, not just OCR accuracy.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Sample | 76 real policy documents; 6,460-document prevalence analysis |
|---|---|
| Metric | micro-F1 0.690 with cues, 0.150 with Docling, 0.075 with plain text |
| Models | six answering models from Anthropic and OpenAI |
| Method | deterministic OOXML property scorer; no LLM judge |
| Limit | single paper, no public code and no independent reproduction |
| Time Basis | arXiv v1 published 2026-10-21T16:51:25Z |
Technical impact 3
Engineering value 4
Adoption signal 1
Maturity 1
Verification cost 2
Risk 3