Year: 2021
Author: Xin Liu, Zaiwen Wen, Ya-Xiang Yuan
CSIAM Transactions on Applied Mathematics, Vol. 2 (2021), Iss. 4 : pp. 585–651
Abstract
Subspace techniques such as Krylov subspace methods have been well known and extensively used in numerical linear algebra. They are also ubiquitous and becoming indispensable tools in nonlinear optimization due to their ability to handle large scale problems. There are generally two types of principals: i) the decision variable is updated in a lower dimensional subspace; ii) the objective function or constraints are approximated in a certain smaller subspace of their domain. The key ingredients are the constructions of suitable subspaces and subproblems according to the specific structures of the variables and functions such that either the exact or inexact solutions of subproblems are readily available and the corresponding computational cost is significantly reduced. A few relevant techniques include but not limited to direct combinations, block coordinate descent, active sets, limited-memory, Anderson acceleration, subspace correction, sampling and sketching. This paper gives a comprehensive survey on the subspace methods and their recipes in unconstrained and constrained optimization, nonlinear least squares problem, sparse and low rank optimization, linear and nonlinear eigenvalue computation, semidefinite programming, stochastic optimization and etc. In order to provide helpful guidelines, we emphasize on high level concepts for the development and implementation of practical algorithms from the subspace framework.
You do not have full access to this article.
Already a Subscriber? Sign in as an individual or via your institution
Journal Article Details
Publisher Name: Global Science Press
Language: English
DOI: https://doi.org/10.4208/csiam-am.SO-2021-0016
CSIAM Transactions on Applied Mathematics, Vol. 2 (2021), Iss. 4 : pp. 585–651
Published online: 2021-01
AMS Subject Headings: Global Science Press
Copyright: COPYRIGHT: © Global Science Press
Pages: 67
Keywords: Nonlinear optimization subspace techniques block coordinate descent active sets limited memory Anderson acceleration subspace correction subsampling sketching.
Author Details
-
Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging
Fast Iterative Algorithms for Blind Phase Retrieval: A Survey
Chang, Huibin | Yang, Li | Marchesini, Stefano2022
https://doi.org/10.1007/978-3-030-03009-4_116-1 [Citations: 1] -
Measurement and Characterization of Position and Posture of Conjunctive Multifreeform Surfaces Using Multisensors
Lyu, Haoyu | Kong, Lingbao | Wang, ShixiangIEEE Transactions on Instrumentation and Measurement, Vol. 72 (2023), Iss. P.1
https://doi.org/10.1109/TIM.2023.3267354 [Citations: 0] -
A hybrid inexact regularized Newton and negative curvature method
Zhu, Hong | Xiao, YunhaiComputational Optimization and Applications, Vol. 88 (2024), Iss. 3 P.849
https://doi.org/10.1007/s10589-024-00576-6 [Citations: 0] -
Adaptive reduced basis trust region methods for parameter identification problems
Kartmann, Michael | Keil, Tim | Ohlberger, Mario | Volkwein, Stefan | Kaltenbacher, BarbaraComputational Science and Engineering, Vol. 1 (2024), Iss. 1
https://doi.org/10.1007/s44207-024-00002-z [Citations: 1] -
Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging
Fast Iterative Algorithms for Blind Phase Retrieval: A Survey
Chang, Huibin | Yang, Li | Marchesini, Stefano2023
https://doi.org/10.1007/978-3-030-98661-2_116 [Citations: 0] -
Subspace Structured Neural Network for Rapid Trajectory Optimization
Tituaña, Luis | Xu, YunjunIFAC-PapersOnLine, Vol. 56 (2023), Iss. 3 P.37
https://doi.org/10.1016/j.ifacol.2023.11.007 [Citations: 0] -
Parameter-efficient fine-tuning of large-scale pre-trained language models
Ding, Ning | Qin, Yujia | Yang, Guang | Wei, Fuchao | Yang, Zonghan | Su, Yusheng | Hu, Shengding | Chen, Yulin | Chan, Chi-Min | Chen, Weize | Yi, Jing | Zhao, Weilin | Wang, Xiaozhi | Liu, Zhiyuan | Zheng, Hai-Tao | Chen, Jianfei | Liu, Yang | Tang, Jie | Li, Juanzi | Sun, MaosongNature Machine Intelligence, Vol. 5 (2023), Iss. 3 P.220
https://doi.org/10.1038/s42256-023-00626-4 [Citations: 170]