Which open model reads historical documents best?
Best zero-shot layout F1 0.350, team of five
Built with
Not public
The repository belongs to a teammate and is private, so I can't link it.
We were set the question of which open foundation model is best for visual understanding of historical documents. Early on we decided "best" doesn't mean much until you name the task, since transcribing a medieval manuscript line is a different job from finding the layout of a page. So each comparison in our protocol changes one thing at a time. Everything ran locally on one MacBook.
We were a team of five. I built most of the evaluation pipeline and presented the results. CHURRO-3B, a model built for historical documents, was the strongest transcriber, but beyond transcription no model is usable as it is, and the best layout F1 was only 0.350.
I also ran some controlled comparisons. Running Qwen3-VL-8B at 4-bit instead of bf16 cut its memory from 18.7 GB to 7.0 GB with no measurable change in error rate. And when it was given a strong first draft from a fine-tuned TrOCR model, it made the draft worse, changing 14.9% of words that were already right.
What this does not show
- It was team work, so the protocol and the conclusions were shared, and my own part was mainly the pipeline and the results.
- Every run was zero-shot, and the bootstrap intervals treat lines as independent when lines from the same manuscript aren't, so they're probably too narrow.