Historical HTR Pipeline : Teaching a model to read handwriting it's never seen before

Early modern Spanish manuscripts, letters, notarial records, ships' logs from the 1500s and 1600s, sit in archives mostly unread, because transcribing them by hand takes forever and most OCR was built for printed text, not a scribe's handwriting from four hundred years ago.

This is an attempt to build a pipeline that can actually read them: find the text on the page, work out the order it's meant to be read in, and transcribe it, in a single pass through a vision-language model rather than a chain of separate tools handed off one to the next.

Historical handwriting breaks most of the assumptions modern OCR is built on. Letterforms vary scribe to scribe. Abbreviations and ligatures were common and were never standardized. Spelling wasn't standardized either, centuries before any Real Academia Española existed to fix it.

Pages are often faded or water-damaged, and frequently laid out in two columns with marginal notes that don't belong anywhere in the main reading order. Get the layout wrong and even a perfect character recognizer produces nonsense, because it's reading a margin note in the middle of a sentence.

Most HTR systems treat this as two jobs stapled together: a layout model finds text regions and reading order, then a separate recognizer, often something like TrOCR or a CRNN-style model, turns each line into text. Errors in the first stage compound silently into the second, and a vision-language model like Qwen3-VL usually only shows up at the end, as a cleanup pass over whatever the recognizer got wrong.

The approach I'm testing puts Qwen3-VL in the loop from the start: layout detection, reading-order recovery, and transcription as one pass, instead of surface-level correction after the fact. A model that already understands where a marginal note sits relative to the main text should have less to correct downstream.

This one is still in progress. What exists today is the pipeline scaffolding in PyTorch and Transformers, plus a working single-page inference path through Qwen3-VL. What doesn't exist yet is a held-out test set with ground-truth transcriptions to measure character error rate against. So I have no accuracy figure to report, and I'd rather say that plainly than publish one no evaluation stands behind.

For a sense of the target: a 2022 study on medieval Latin manuscripts (arXiv:2201.07661) cut character error rate from roughly 6% out of the box to about 3% by fine-tuning on two pages of ground truth per document. That's the range I'm aiming at. Whether this pipeline lands anywhere near it is still unmeasured.

Reading order is the open question that matters most right now, ahead of character accuracy. A model can get every letter in a line right and still produce a useless transcription if it stitches those lines together in the wrong sequence.

Getting that right, consistently, across pages with marginalia, insertions, and the occasional half-page written sideways, is what the next stage of this is about.