Qing Su*

Unsqueeze [CLS] Bottleneck to Learn Rich Representations

"Distillation-based self-supervised learning typically leads to more compressed representations due to its radical clustering process and the implementation of a sharper target distribution. To overcome this limitation and preserve more information from input, we introduce UDI, conceptualized as Uns…