Decreasing Word Error Rates in Paragraph Handwritten Text Recognition with Synthetic Data

Abstract

Handwritten Text Recognition (HTR) faces a persistent challenge with the scarcity of data at the paragraph level, arising from the difficulty of acquiring diverse, cost-efficient, and cleanly labeled datasets for training. As such, works in HTR leverage segmentation, regularization techniques, and language modeling to excel in a low-data environment. While synthetic generation methods gain traction on the word and line-level recognition, this success has not translated to the paragraph level. Hence, our work seeks to mimic the nuances of paragraph-level text images with a custom synthetic data engine using Wikipedia texts. Experiments show that by using our synthetic dataset in tandem with a simple encoder-decoder Transformer, we can achieve the best Word Error Rate (WER) amongst the state-of-the-art methods for handwriting recognition on the IAM dataset. Additionally, we show the model pretrained on English texts can also recognize French and German texts with minimal finetuning.