A Minibatch Proximal Stochastic Recursive Gradient Algorithm Using a Trust-Region-Like Scheme and Barzilai-Borwein Stepsizes
We consider the problem of minimizing the sum of an average of a large number of smooth convex component functions and a possibly nonsmooth convex function that admits a simple proximal mapping. This class of problems arises frequently in machine learning, known as regularized empirical risk minimiz...
| Main Authors: | , , , |
|---|---|
| Format: | Journal Article |
| Language: | English |
| Published: |
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
2021
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/20.500.11937/91427 |
| _version_ | 1848765517603536896 |
|---|---|
| author | Yu, T. Liu, X.W. Dai, Y.H. Sun, Jie |
| author_facet | Yu, T. Liu, X.W. Dai, Y.H. Sun, Jie |
| author_sort | Yu, T. |
| building | Curtin Institutional Repository |
| collection | Online Access |
| description | We consider the problem of minimizing the sum of an average of a large number of smooth convex component functions and a possibly nonsmooth convex function that admits a simple proximal mapping. This class of problems arises frequently in machine learning, known as regularized empirical risk minimization (ERM). In this article, we propose mSRGTR-BB, a minibatch proximal stochastic recursive gradient algorithm, which employs a trust-region-like scheme to select stepsizes that are automatically computed by the Barzilai-Borwein method. We prove that mSRGTR-BB converges linearly in expectation for strongly and nonstrongly convex objective functions. With proper parameters, mSRGTR-BB enjoys a faster convergence rate than the state-of-the-art minibatch proximal variant of the semistochastic gradient method (mS2GD). Numerical experiments on standard data sets show that the performance of mSRGTR-BB is comparable to and sometimes even better than mS2GD with best-tuned stepsizes and is superior to some modern proximal stochastic gradient methods. |
| first_indexed | 2025-11-14T11:36:31Z |
| format | Journal Article |
| id | curtin-20.500.11937-91427 |
| institution | Curtin University Malaysia |
| institution_category | Local University |
| language | English |
| last_indexed | 2025-11-14T11:36:31Z |
| publishDate | 2021 |
| publisher | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | curtin-20.500.11937-914272023-05-03T07:20:45Z A Minibatch Proximal Stochastic Recursive Gradient Algorithm Using a Trust-Region-Like Scheme and Barzilai-Borwein Stepsizes Yu, T. Liu, X.W. Dai, Y.H. Sun, Jie Science & Technology Technology Computer Science, Artificial Intelligence Computer Science, Hardware & Architecture Computer Science, Theory & Methods Engineering, Electrical & Electronic Computer Science Engineering Convergence Convex functions Risk management Gradient methods Learning systems Sun Barzilai-Borwein (BB) method empirical risk minimization (ERM) proximal method stochastic gradient trust-region MACHINE We consider the problem of minimizing the sum of an average of a large number of smooth convex component functions and a possibly nonsmooth convex function that admits a simple proximal mapping. This class of problems arises frequently in machine learning, known as regularized empirical risk minimization (ERM). In this article, we propose mSRGTR-BB, a minibatch proximal stochastic recursive gradient algorithm, which employs a trust-region-like scheme to select stepsizes that are automatically computed by the Barzilai-Borwein method. We prove that mSRGTR-BB converges linearly in expectation for strongly and nonstrongly convex objective functions. With proper parameters, mSRGTR-BB enjoys a faster convergence rate than the state-of-the-art minibatch proximal variant of the semistochastic gradient method (mS2GD). Numerical experiments on standard data sets show that the performance of mSRGTR-BB is comparable to and sometimes even better than mS2GD with best-tuned stepsizes and is superior to some modern proximal stochastic gradient methods. 2021 Journal Article http://hdl.handle.net/20.500.11937/91427 10.1109/TNNLS.2020.3025383 English IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC restricted |
| spellingShingle | Science & Technology Technology Computer Science, Artificial Intelligence Computer Science, Hardware & Architecture Computer Science, Theory & Methods Engineering, Electrical & Electronic Computer Science Engineering Convergence Convex functions Risk management Gradient methods Learning systems Sun Barzilai-Borwein (BB) method empirical risk minimization (ERM) proximal method stochastic gradient trust-region MACHINE Yu, T. Liu, X.W. Dai, Y.H. Sun, Jie A Minibatch Proximal Stochastic Recursive Gradient Algorithm Using a Trust-Region-Like Scheme and Barzilai-Borwein Stepsizes |
| title | A Minibatch Proximal Stochastic Recursive Gradient Algorithm Using a Trust-Region-Like Scheme and Barzilai-Borwein Stepsizes |
| title_full | A Minibatch Proximal Stochastic Recursive Gradient Algorithm Using a Trust-Region-Like Scheme and Barzilai-Borwein Stepsizes |
| title_fullStr | A Minibatch Proximal Stochastic Recursive Gradient Algorithm Using a Trust-Region-Like Scheme and Barzilai-Borwein Stepsizes |
| title_full_unstemmed | A Minibatch Proximal Stochastic Recursive Gradient Algorithm Using a Trust-Region-Like Scheme and Barzilai-Borwein Stepsizes |
| title_short | A Minibatch Proximal Stochastic Recursive Gradient Algorithm Using a Trust-Region-Like Scheme and Barzilai-Borwein Stepsizes |
| title_sort | minibatch proximal stochastic recursive gradient algorithm using a trust-region-like scheme and barzilai-borwein stepsizes |
| topic | Science & Technology Technology Computer Science, Artificial Intelligence Computer Science, Hardware & Architecture Computer Science, Theory & Methods Engineering, Electrical & Electronic Computer Science Engineering Convergence Convex functions Risk management Gradient methods Learning systems Sun Barzilai-Borwein (BB) method empirical risk minimization (ERM) proximal method stochastic gradient trust-region MACHINE |
| url | http://hdl.handle.net/20.500.11937/91427 |