Iterative Hessian Sketch with Momentum

Abstract

We propose a novel randomized linear least squares solver which is an improvement of Iterative Hessian Sketch and randomized preconditioning. In the proposed Momentum-IHS technique (M-IHS), Heavy Ball Method is used to accelerate the convergence of iterations. It is shown that for any full rank data matrix, rate of convergence depends on the ratio between the feature size and the sketch size. Unlike the Conjugate Gradient technique, the rate of convergence is unaffected by either the condition number or the eigenvalue spectrum of the data matrix. As demonstrated over many examples, the proposed M-IHS provides compatible performance with the state of the art randomized preconditioning methods such as LSRN or Blendenpik and yet, it provides a completely different perspective in the area of iterative solvers which can pave the way for future developments.