7 accepted papers
We study the fundamental expressivity limits of transformer models by formalizing the notion of accessible sequences---those that a transformer can produce for some prompt---and characterizing how accessibility depends on prompt length and model parameters. Our analysis provides a theoretical explan…
In order to reduce the quadratic cost of matrix-vector multiplications in dense and attention layers, Monarch matrices have been recently introduced, achieving a sub-quadratic complexity. It consists in factorizing a matrix using fixed permutations and learned block diagonal matrices, at the price o…
Weighted least squares are often used to approximate log-likelihoods when solving inverse problems involving non-Gaussian noise as they are more appealing from an optimization perspective. Although a theoretical expression of the weights can be derived for specific noises, this may become intractabl…
The *straight-through estimator* (STE) is commonly used to optimize quantized neural networks, yet its contexts of effective performance are still unclear despite empirical successes. To make a step forward in this comprehension, we apply STE to a well-understood problem: *sparse support recovery*.…
We address the problem of filtering out localized time-frequency components in signals. The problem is formulated as a minimization of a suitable quadratic form, that involves a data fidelity term on the short-time Fourier transform outside the support of the undesired component, and an energy penal…
In Nuclear Magnetic Resonance (NMR) spectroscopy, an efficient analysis and a relevant extraction of different molecule properties from a given chemical mixture are important tasks, especially when processing bidimensional NMR data. To that end, using a blind source separation approach based on a va…
When using optimization methods with matrix variables in signal processing and machine learning, it is customary to assume some low-rank prior on the targeted solution. Nonnegative matrix factorization of spectrograms is a case in point in audio signal processing. However, this low-rank prior is not…