← Search

Daniel Rotem

2 accepted papers

2023

Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings

ACL 2023long

Adaptive inference is a simple method for reducing inference costs. The method works by maintaining multiple classifiers of different capacities, and allocating resources to each test instance according to its difficulty. In this work, we compare the two main approaches for adaptive inference, Early…

2022

How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers

EMNLP 2022finding

The attention mechanism is considered the backbone of the widely-used Transformer architecture. It contextualizes the input by computing input-specific attention matrices. We find that this mechanism, while powerful and elegant, is not as important as typically thought for pretrained language models…