Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment
We identify the Spectral Energy Gain in extreme model compression, where low-rank binary approximations outperform tiny-rank floating-point baselines for heavy-tailed spectra. However, prior attempts fail to realize this potential, trailing state-of-the-art 1-bit methods. We attribute this degradati…