Erik Ylipää Hellqvist

Semantic Re-tuning with Contrastive Tension

Extracting semantically useful natural language sentence representations from pre-trained deep neural networks such as Transformers remains a challenge. We first demonstrate that pre-training objectives impose a significant task bias onto the final layers of models with a layer-wise survey of the Se…