Align Forward, Adapt Backward: Closing the Discretization Gap in Logic Gate Networks
Differentiable discrete selection uses soft mixtures during training but hard selection at deployment, resulting in a training-inference gap. We decompose this gap into selection gap (method-dependent, reducible) and computation gap (input-dependent, irreducible). Our key finding: the selection gap …