João Coelho

Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval

This study investigates the existence of positional biases in Transformer-based language models for text representation learning, particularly in the context of web document retrieval. We build on previous research that demonstrated loss of information in the middle of input sequences for causal lan…