LLM benchmarking for HTR on french historical death records
Аннотация
This study evaluates the performance of recent multimodal large language models (LLM) for the transcription of historical handwritten and printed documents, with a focus on sources commonly used in historical demography. We benchmarked gpt-4o , several Qwen models, and other widely used LLMs against traditional OCR/HTR systems, including Transkribus and Ocelus, on a dataset of archival records containing complex layouts, marginal annotations, and variable handwriting styles. gpt-4o achieved the highest transcription accuracy across both paragraph text and marginal entries, while Qwen emerged as the most effective fully free and unrestricted alternative. Our analysis also highlights the limits of common evaluation metrics: the BLEU score did not consistently correlate with the Character Error Rate (CER), due to BLEU’s sensitivity to n-gram overlap rather than character-level fidelity. This distinction is critical for historical sources where spelling preservation and textual exactness are essential. Overall, the results demonstrate the growing potential of LLM-based approaches for large-scale historical data extraction and emphasize the need for continued benchmarking and domain-specific evaluation frameworks as new models rapidly emerge.