SmartExp: An Adaptive Data Expansion Strategy for Improving Handwritten Text Recognition

Abstract

Constructing a highly accurate handwritten OCR system requires large amounts of high-quality training data, yet data collection is labor-intensive and costly. With the advance of generative models, high-quality synthetic images have been applied to enhance handwritten text recognition (HTR) models, a technique termed data expansion. However, we find that the generated data may sometimes provide only insignificant improvements to HTR. We investigate the causes behind the small gains from the perspectives of both data expansion and data augmentation. We reveal a balancing mechanism in HTR that stronger data expansion should be paired with weaker data augmentation. Informed by these insights, we propose SmartExp, a purely data-driven strategy without introducing any extra data or computational costs. Extensive experiments on three widely used benchmark datasets demonstrate that our SmartExp strategy can significantly improve various HTR models. Our code is publicly available at: https://github.com/Fyzjym/SmartExp.