NeurIPS 2025spotlight0 citations
Differentiable Hierarchical Visual Tokenization
Marius Aasan, Martine Hjelkrem-Tan, Nico Catalano, Changkyu Choi, Adín Ramírez Rivera
Abstract
Vision Transformers rely on fixed patch tokens that ignore the spatial and semantic structure of images. In this work, we introduce an end-to-end differentiable tokenizer that adapts to image content with pixel-level granularity while remaining backward-compatible with existing architectures for retrofitting pretrained models. Our method uses hierarchical model selection with information criteria to provide competitive performance in both image-level classification and dense-prediction tasks, and even supports out-of-the-box raster-to-vector conversion.
ViTPartitioningTokenizationSegmentationVectorizationAdaptiveHierarchicalModel Selection
BibTeX
@inproceedings{
aasan2025differentiable,
title={Differentiable Hierarchical Visual Tokenization},
author={Marius Aasan and Martine Hjelkrem-Tan and Nico Catalano and Changkyu Choi and Ad{\'\i}n Ram{\'\i}rez Rivera},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=y8VWYf5cVI}
}