Sankalp Dayal

S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations

Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization, creating excessively large ranges that cause severe accuracy drops during quantization. We empirically observe that outlier severity intensifies with pre-training scale (e.g., progressing from C