MingJia Mao

An Untold Story of Preprocessing Task Evaluation: An Alignment-based Joint Evaluation Approach

A preprocessing task such as tokenization and sentence boundary detection (SBD) has commonly been considered as NLP challenges that have already been solved. This perception is due to their generally good performance and the presence of pre-tokenized data. However, it’s important to note that the lo…