Akira Sakai

Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression

Sub-bit model compression seeks storage below one bit per weight, where the sign bit becomes a fixed-cost bottleneck as magnitudes are aggressively compressed. Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank compression and are spectrally indistinguishable from i.i.d. Rade…