Language Models Can be Efficiently Steered via Minimal Embedding Layer Transformations

Diogo Tavares; David Semedo; Alexander Rudnicky; Joao Magalhaes

EMNLP 20250 citations

Language Models Can be Efficiently Steered via Minimal Embedding Layer Transformations

Diogo Tavares, David Semedo, Alexander Rudnicky, Joao Magalhaes

Abstract

Large Language Models (LLMs) are increasingly costly to fine-tune due to their size, with embedding layers alone accounting for up to 20% of model parameters. While Parameter-Efficient Fine-Tuning (PEFT) methods exist, they largely overlook the embedding layer. In this paper, we introduce TinyTE, a novel PEFT approach that steers model behavior via minimal translational transformations in the embedding space. TinyTE modifies input embeddings without altering hidden layers, achieving competitive performance while requiring approximately 0.0001% of the parameters needed for full fine-tuning. Experiments across architectures provide a new lens for understanding the relationship between input representations and model behavior—revealing them to be more flexible at their foundation than previously thought.

BibTeX

@inproceedings{emnlp2025_languagemodelsca,
  title = {Language Models Can be Efficiently Steered via Minimal Embedding Layer Transformations},
  author = {Diogo Tavares and David Semedo and Alexander Rudnicky and Joao Magalhaes},
  booktitle = {EMNLP 2025},
  year = {2025}
}