Distilling and Scaling Hierarchical Vision Transformer to 30B Parameters

Abstract

Recent efforts have successfully scaled plain Vision Transformers (ViTs) to unprecedented sizes, ranging from 6B to 22B parameters. In contrast, their hierarchical counterparts have remained largely constrained to less than 1B parameters. To bridge this gap, we propose EHV, an efficient hierarchical ViT architecture. Our dense model scales from 200M to 5B parameters, and we further extend it into a Sparse Mixture-of-Experts (SMoE) variant, achieving an industry-leading scale of 30B parameters. The unsupervised pretraining process consists of two stages: first, we pretrain on ImageNet-21K using a Masked Autoencoder (MAE); second, the resulting model is distilled from multiple state-of-the-art foundation models on a nearly 27M-image dataset. With only 6.7B active parameters, EHV-5B-MoE demonstrates exceptional transfer learning performance across image classification, fine-grained classification, as well as video and dense prediction tasks, specially achieving a linear evaluation accuracy of 89.0\% on ImageNet-1K. This result surpasses those of comparable models such as EVA-CLIP-18B and DINOv3-7B, indicating that its SMoE architecture learns high-quality, generalizable, and linearly separable feature representations.