Levente Zólyomi

Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models

Transformer models have become the dominant backbone for sequence modeling, leveraging self-attention to produce contextualized token representations. These are typically aggregated into fixed-size vectors via pooling operations for downstream tasks. While much of the literature has focused on atten…