no code implementations • 12 Sep 2023 • Jingpu Cheng, Qianxiao Li, Ting Lin, Zuowei Shen
We investigate the expressive power of deep residual neural networks idealized as continuous dynamical systems through control theory.
no code implementations • 25 Nov 2022 • Ting Lin, Zuowei Shen, Qianxiao Li
We study the approximation of shift-invariant or equivariant functions by deep fully convolutional networks from the dynamical systems perspective.
no code implementations • 18 Aug 2022 • Qianxiao Li, Ting Lin, Zuowei Shen
We study the approximation of functions which are invariant with respect to certain permutations of the input indices using flow maps of dynamical systems.
no code implementations • 19 May 2022 • Zuowei Shen, Haizhao Yang, Shijun Zhang
It is proved by construction that height-$s$ ReLU NestNets with $\mathcal{O}(n)$ parameters can approximate $1$-Lipschitz continuous functions on $[0, 1]^d$ with an error $\mathcal{O}(n^{-(s+1)/d})$, while the optimal approximation error of standard ReLU networks with $\mathcal{O}(n)$ parameters is $\mathcal{O}(n^{-2/d})$.
1 code implementation • 10 Mar 2022 • Yong Zheng Ong, Zuowei Shen, Haizhao Yang
Discretization invariant learning aims at learning in the infinite-dimensional function spaces with the capacity to process heterogeneous discrete representations of functions as inputs and/or outputs of a learning model.
no code implementations • 15 Nov 2021 • Zuowei Shen, Haizhao Yang, Shijun Zhang
Furthermore, we show that the idea of learning a small number of parameters to achieve a good approximation can be numerically observed.
no code implementations • 6 Jul 2021 • Zuowei Shen, Haizhao Yang, Shijun Zhang
This paper develops simple feed-forward neural networks that achieve the universal approximation property for all continuous functions with a fixed finite number of neurons.
no code implementations • 28 Feb 2021 • Zuowei Shen, Haizhao Yang, Shijun Zhang
This paper concentrates on the approximation power of deep feed-forward neural networks in terms of width and depth.
no code implementations • 25 Oct 2020 • Zuowei Shen, Haizhao Yang, Shijun Zhang
A three-hidden-layer neural network with super approximation power is introduced.
no code implementations • 22 Jun 2020 • Zuowei Shen, Haizhao Yang, Shijun Zhang
More generally for an arbitrary continuous function $f$ on $[0, 1]^d$ with a modulus of continuity $\omega_f(\cdot)$, the constructive approximation rate is $\omega_f(\sqrt{d}\, N^{-\sqrt{L}})+2\omega_f(\sqrt{d}){N^{-\sqrt{L}}}$.
no code implementations • 18 Apr 2020 • Yongqiang Cai, Qianxiao Li, Zuowei Shen
We present the viewpoint that optimization problems encountered in machine learning can often be interpreted as minimizing a convex functional over a function space, but with a non-convex constraint set introduced by model parameterization.
no code implementations • 9 Jan 2020 • Jianfeng Lu, Zuowei Shen, Haizhao Yang, Shijun Zhang
This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously.
no code implementations • 22 Dec 2019 • Qianxiao Li, Ting Lin, Zuowei Shen
We build on the dynamical systems approach to deep learning, where deep residual networks are idealized as continuous-time dynamical systems, from the approximation perspective.
no code implementations • 13 Jun 2019 • Zuowei Shen, Haizhao Yang, Shijun Zhang
This paper quantitatively characterizes the approximation power of deep feed-forward neural networks (FNNs) in terms of the number of neurons.
no code implementations • ICLR 2019 • Yongqiang Cai, Qianxiao Li, Zuowei Shen
Despite its empirical success, the theoretical underpinnings of the stability, convergence and acceleration properties of batch normalization (BN) remain elusive.
no code implementations • 26 Feb 2019 • Zuowei Shen, Haizhao Yang, Shijun Zhang
In particular, for any function $f$ on $[0, 1]$, regardless of its smoothness and even the continuity, if $f$ can be approximated using a dictionary when $L=1$ with the best $N$-term approximation rate $\varepsilon_{L, f}={\cal O}(N^{-\eta})$, we show that dictionaries with $L=2$ can improve the best $N$-term approximation rate to $\varepsilon_{L, f}={\cal O}(N^{-2\eta})$.
no code implementations • ICLR 2019 • Yongqiang Cai, Qianxiao Li, Zuowei Shen
Despite its empirical success and recent theoretical progress, there generally lacks a quantitative analysis of the effect of batch normalization (BN) on the convergence and stability of gradient descent.
no code implementations • 17 Feb 2016 • Bin Dong, Zuowei Shen, Peichu Xie
In this paper, we introduce a generic wavelet frame based image restoration model, called the "general model", which includes most of the existing wavelet frame based models as special cases.
no code implementations • CVPR 2014 • Chenglong Bao, Hui Ji, Yuhui Quan, Zuowei Shen
Sparse coding and dictionary learning have seen their applications in many vision tasks, which usually is formulated as a non-convex optimization problem.
no code implementations • 6 Feb 2013 • Fei Yang, Hong Jiang, Zuowei Shen, Wei Deng, Dimitris Metaxas
We address the problem of reconstructing and analyzing surveillance videos using compressive sensing.
4 code implementations • 18 Oct 2008 • Jian-Feng Cai, Emmanuel J. Candes, Zuowei Shen
Off-the-shelf algorithms such as interior point methods are not directly amenable to large problems of this kind with over a million unknown entries.
Optimization and Control