• Abstract

Multi-channel/modality image joint reconstruction has gained much research interest in recent years. In this paper, we propose to use a nonconvex and nonLipschitz joint regularizer in a general variational model for joint reconstruction under additive measurement noise. This framework has good ability in edge-preserving by sharing common edge features of individual images. We study the lower bound theory for the non-Lipschitz joint reconstruction model in two important cases with Gaussian and impulsive measurement noise, respectively. In addition, we extend previous works to propose an inexact iterative support shrinking algorithm with proximal linearization for multi-channel image reconstruction (InISSAPL-MC) and prove that the iterative sequence converges globally to a critical point of the original objective function. In a special case of single channel image restoration, the convergence result improves those in the literature. For numerical implementation, we adopt primal dual method to the inner subproblem. Numerical experiments in color image restoration and two-modality undersampled magnetic resonance imaging (MRI) reconstruction show that the proposed non-Lipschitz joint reconstruction method achieves considerable improvements in terms of edge preservation for piecewise constant images compared to existing methods.

• Abstract

We present a methodology to construct efficient high-order in time accurate numerical schemes for a class of gradient flows with appropriate Lipschitz continuous nonlinearity. There are several ingredients to the strategy: the exponential time differencing (ETD), the multi-step (MS) methods, the idea of stabilization, and the technique of interpolation. They are synthesized to develop a generic $k^{th}$ order in time efficient linear numerical scheme with the help of an artificial regularization term of the form $Aτ^k \frac{∂}{∂t} \mathcal{L}^{p(k)} u$ where $\mathcal{L}$ is the positive definite linear part of the flow, $τ$ is the uniform time step-size. The exponent $p(k)$ is determined explicitly by the strength of the Lipschitz nonlinear term in relation to $\mathcal{L}$ together with the desired temporal order of accuracy $k$. To validate our theoretical analysis, the thin film epitaxial growth without slope selection model is examined with a fourth-order ETD-MS discretization in time and Fourier pseudo-spectral in space discretization. Our numerical results on convergence and energy stability are in accordance with our theoretical results.

• Abstract

Study about theory and algorithms for nonlinear programming usually assumes that the feasible region of the problem is nonempty. However, there are many important practical nonlinear programming problems whose feasible regions are not known to be nonempty or not, and optimizers of the objective function with the least constraint violation prefer to be found. A natural way for dealing with these problems is to extend the nonlinear programming problem as the one optimizing the objective function over the set of points with the least constraint violation. Firstly, the minimization problem with least constraint violation is proved to be an Lipschitz equality constrained optimization problem when the original problem is a convex nonlinear programming problem with possible inconsistent constraints, and it can be reformulated as an MPCC problem; And the minimization problem with least constraint violation is relaxed to an MPCC problem when the original problem is an nonlinear programming problem with possible inconsistent non-convex constraints. Secondly, for nonlinear programming problems with possible inconsistent constraints, it is proved that a local minimizer of the MPCC problem is an M-stationary point and an elegant necessary optimality condition, named as L-stationary condition, is established from the classical optimality theory of Lipschitz continuous optimization. Thirdly, properties of the penalty method for the minimization problem with the least constraint violation are developed and the proximal gradient method for the penalized problem is studied. Finally, the smoothing Fischer-Burmeister function method is constructed for solving the MPCC problem related to minimizing the objective function with the least constraint violation. It is demonstrated that, when the positive smoothing parameter approaches to zero, any point in the outer limit of the KKT-point mapping is an L-stationary point of the equivalent MPCC problem.

• Abstract

Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, empirical studies reported a universal phenomenon of Frequency Principle (F-Principle), that is, a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very useful in providing both qualitative and quantitative understandings of DNNs. In this paper, we rigorously investigate the F-Principle for the training dynamics of a general DNN at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our results are general in the sense that they work for multilayer networks with general activation functions, population densities of data, and a large class of loss functions. Our work lays a theoretical foundation of the F-Principle for a better understanding of the training process of DNNs.

• Abstract

We propose a class of multipliers correction methods to minimize a differentiable function over the Stiefel manifold. The proposed methods combine a function value reduction step with a proximal correction step. The former one searches along an arbitrary descent direction in the Euclidean space instead of a vector in the tangent space of the Stiefel manifold. Meanwhile, the latter one minimizes a first-order proximal approximation of the objective function in the range space of the current iterate to make Lagrangian multipliers associated with orthogonality constraints symmetric at any accumulation point. The global convergence has been established for the proposed methods. Preliminary numerical experiments demonstrate that the new methods significantly outperform other state-of-the-art first-order approaches in solving various kinds of testing problems.

• Abstract

Random projections are able to perform dimension reduction efficiently for datasets with nonlinear low-dimensional structures. One well-known example is that random matrices embed sparse vectors into a low-dimensional subspace nearly isometrically, known as the restricted isometric property in compressed sensing. In this paper, we explore some applications of random projections in deep neural networks. We provide the expressive power of fully connected neural networks when the input data are sparse vectors or form a low-dimensional smooth manifold. We prove that the number of neurons required for approximating a Lipschitz function with a prescribed precision depends on the sparsity or the dimension of the manifold and weakly on the dimension of the input vector. The key in our proof is that random projections embed stably the set of sparse vectors or a low-dimensional smooth manifold into a low-dimensional subspace. Based on this fact, we also propose some new neural network models, where at each layer the input is first projected onto a low-dimensional subspace by a random projection and then the standard linear connection and non-linear activation are applied. In this way, the number of parameters in neural networks is significantly reduced, and therefore the training of neural networks can be accelerated without too much performance loss.