General and Efficient Steering of Unconditional Diffusion Models

Abstract

Conditioning unconditional diffusion models typically requires either retraining with conditional inputs or per-step gradient computations (e.g., classifier-based guidance), both of which incur substantial computational overhead. We present a general recipe for efficiently steering unconditional diffusion *without gradient guidance during inference*, enabling fast controllable generation. Our approach is built on two observations about diffusion model structure: **Noise Alignment**: even in early, highly corrupted stages, coarse semantic steering is possible using a lightweight, offline-computed guidance signal, avoiding any per-step or per-sample gradients. **Transferable concept vectors**: a concept direction in activation space once learned through Recursive Feature Machines (RFMs) transfers across both *timesteps* and *samples*; the same fixed steering vector learned near clean time remains effective when injected at intermediate noise levels for every generation trajectory, providing refined conditional control with efficiency. Experiments on CIFAR-10, ImageNet, and CelebA demonstrate improved accuracy/quality relative to gradient-based guidance, while achieving significant inference speedups.