Abstract
The Nesterov gradient descent algorithm serves as a performance benchmark forconvex optimization problems. Like many other gradient-based methods, the Nesterov
algorithm requires choosing a constant step size before optimization begins,
and the performance of the algorithm heavily depends on the step size. Here, we
propose three novel adaptive algorithms which adaptively determine the step size
based on the searching history. The new adaptive methods were tested alongside
the original Nesterov algorithm on a list of commonly-used optimization test
functions in a range of dimensions. The experimental results showed that they consistently
outperformed the Nesterov algorithm by a wide margin. We also discuss
ways that the adaptive methods could be improved.