Nuria Bel

Hints on the data for language modeling of synthetic languages with transformers

Language Models (LM) are becoming more and more useful for providing representations upon which to train Natural Language Processing applications. However, there is now clear evidence that attention-based transformers require a critical amount of language data to produce good enough LMs. The questio…