Hyunwoong Ko

Length-aware Byte Pair Encoding for Mitigating Over-segmentation in Korean Machine Translation

Byte Pair Encoding is an effective approach in machine translation across several languages. However, our analysis indicates that BPE is prone to over-segmentation in the morphologically rich language, Korean, which can erode word semantics and lead to semantic confusion during training. This semant…