NeurIPS 2023poster6 citations
Nearly Optimal Bounds for Cyclic Forgetting
William Joseph Swartworth, Deanna Needell, Rachel Ward, Mark Kong, Halyun Jeong
Abstract
We provide theoretical bounds on the forgetting quantity in the continual learning setting for linear tasks, where each round of learning corresponds to projecting onto a linear subspace. For a cyclic task ordering on $T$ tasks repeated $m$ times each, we prove the best known upper bound of $O(T^2/m)$ on the forgetting. Notably, our bound holds uniformly over all choices of tasks and is independent of the ambient dimension. Our main technical contribution is a characterization of the union of all numerical ranges of products of $T$ (real or complex) projections as a sinusoidal spiral, which may be of independent interest.
catastrophic forgettinglinear systems
BibTeX
@inproceedings{
swartworth2023nearly,
title={Nearly Optimal Bounds for Cyclic Forgetting},
author={William Joseph Swartworth and Deanna Needell and Rachel Ward and Mark Kong and Halyun Jeong},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
year={2023},
url={https://openreview.net/forum?id=X25L5AjHig}
}