Federico Barbero — accepted AI-conference papers

2026

Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin

Enrique Queipo-de-Llano, Alvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael M. Bronstein, Yann LeCun +1

ICLR 2026poster

Attention sinks and compression valleys have attracted significant attention as two puzzling phenomena in large language models, but have been studied in isolation. In this work, we present a surprising connection between attention sinks and compression valleys, tracing both to the formation of mass…

Cited by 0 Source ↗Scholar ↗

2026

Extracting alignment data in open models

Federico Barbero, Xiangming Gu, Christopher A. Choquette Choo, Chawin Sitawarin, Matthew Jagielski, Itay Yona +3

ICML 2026poster

In this work, we show that it is possible to extract significant amounts of alignment training data from a post-trained model -- useful to steer the model to improve certain capabilities such as long-context reasoning, safety, instruction following, and maths. While the majority of related work on m…

Cited by 0 Source ↗Scholar ↗

2026

How do LLMs Compute Verbal Confidence?

Dharshan Kumaran, Arthur Conmy, Federico Barbero, Simon Osindero, Viorica Patraucean, Petar Veličković

ICML 2026poster

Verbal confidence—prompting LLMs to state their confidence as a number or category—is widely used to extract uncertainty estimates from black-box models. However, how LLMs internally generate such scores remains unknown. We address two questions: first, when confidence is computed -- just-in-time wh…

Cited by 0 Source ↗Scholar ↗

2026

Rotary Position Encodings for Graphs

Isaac Reid, Arijit Sehanobish, Cederik Höfs, Bruno Mlodozeniec, Leonhard Vulpius, Federico Barbero +4

ICML 2026spotlight

We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data. We find that rotating tokens depending on the spectrum of the graph…

Cited by 0 Source ↗Scholar ↗

2025

Bundle Neural Network for message diffusion on graphs

Jacob Bamberger, Federico Barbero, Xiaowen Dong, Michael M. Bronstein

ICLR 2025spotlight

The dominant paradigm for learning on graphs is message passing. Despite being a strong inductive bias, the local message passing mechanism faces challenges such as over-smoothing, over-squashing, and limited expressivity. To address these issues, we introduce Bundle Neural Networks (BuNNs), a novel…

Cited by 1 Source ↗PDF ↗Scholar ↗

2025

Generating Creative Chess Puzzles

Xidong Feng, Vivek Veeriah, Marcus Chiam, Michael D Dennis, Federico Barbero, Johan Obando-Ceron +5

NeurIPS 2025poster

While Generative AI rapidly advances in various domains, generating truly creative, aesthetic, and counter-intuitive outputs remains a challenge. This paper presents an approach to tackle these difficulties in the domain of chess puzzles. We start by benchmarking Generative AI architectures, and the…

Cited by 0 Source ↗Scholar ↗

2025

On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning

Alvaro Arroyo, Alessio Gravina, Benjamin Gutteridge, Federico Barbero, Claudio Gallicchio, Xiaowen Dong +2

NeurIPS 2025poster

Graph Neural Networks (GNNs) are models that leverage the graph structure to transmit information between nodes, typically through the message-passing operation. While widely successful, this approach is well-known to suffer from representational collapse as the number of layers increases and insens…

Cited by 0 Source ↗Scholar ↗

2025

Round and Round We Go! What makes Rotary Positional Encodings useful?

Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, Petar Veličković

ICLR 2025poster

Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the…

Cited by 14 Source ↗PDF ↗Scholar ↗

2025

Softmax is not Enough (for Sharp Size Generalisation)

Petar Veličković, Christos Perivolaropoulos, Federico Barbero, Razvan Pascanu

ICML 2025poster

A key property of reasoning systems is the ability to make sharp decisions on their input data. For contemporary AI systems, a key carrier of sharp behaviour is the softmax function, with its capability to perform differentiable query-key lookups. It is a common belief that the predictive power of n…

Cited by 0 Source ↗PDF ↗Scholar ↗

2024

Locality-Aware Graph Rewiring in GNNs

Federico Barbero, Ameya Velingker, Amin Saberi, Michael M. Bronstein, Francesco Di Giovanni

ICLR 2024poster

Graph Neural Networks (GNNs) are popular models for machine learning on graphs that typically follow the message-passing paradigm, whereby the feature of a node is updated recursively upon aggregating information over its neighbors. While exchanging messages over the input graph endows GNNs with a s…

Cited by 29 Source ↗PDF ↗Scholar ↗

2024

Transformers need glasses! Information over-squashing in language tasks

Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João Guilherme Madeira Araújo, Alex Vitvitskyi +2

NeurIPS 2024poster

We study how information propagates in decoder-only Transformers, which are the architectural foundation of most existing frontier large language models (LLMs). We rely on a theoretical signal propagation analysis---specifically, we analyse the representations of the last token in the final layer of…

Cited by 21 Source ↗PDF ↗Scholar ↗

2023

Latent Graph Inference using Product Manifolds

Haitz Sáez de Ocáriz Borde, Anees Kazi, Federico Barbero, Pietro Lio

ICLR 2023poster

Graph Neural Networks usually rely on the assumption that the graph topology is available to the network as well as optimal for the downstream task. Latent graph inference allows models to dynamically learn the intrinsic graph structure of problems where the connectivity patterns of data may not be…

Cited by 25 Source ↗PDF ↗Scholar ↗

2023

On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topology

Francesco Di Giovanni, Lorenzo Giusti, Federico Barbero, Giulia Luise, Pietro Lio, Michael M. Bronstein

ICML 2023poster

Message Passing Neural Networks (MPNNs) are instances of Graph Neural Networks that leverage the graph to send messages over the edges. This inductive bias leads to a phenomenon known as over-squashing, where a node feature is insensitive to information contained at distant nodes. Despite recent met…

Cited by 158 Source ↗PDF ↗code ↗Scholar ↗