betaFeedback

WorksMathematics

Training Neural Networks to be Locally Adaptive

Abstract

Deep neural networks fitted by global empirical risk minimization commit to a single model for the entire covariate space. In a regression problem, if the regression function f0f_0 is inhomogeneous — varying rapidly in one region, while being nearly flat in another — no single model can be appropriate everywhere, and the fit degrades where the function varies most rapidly. Here we study an alternative in which, instead of fitting the network globally, we fit it locally. For each query point xx, a local neural network is trained on the kk nearest-neighbours of xx alone, making the training sample and the fitted function query-dependent. We analyze the performance of the resulting estimator through a localization approach, establishing that the pointwise risk of the estimator with an appropriate choice of kk attains the rate n−2α/(2α+d∗)n^{-2α/(2α+d^{*})}, up to logarithmic factors, when f0f_0 is locally αα-Hölder with α∈(0,1]α\in(0,1] and the covariates are supported on a d∗d^{*}-dimensional manifold embedded in Rd\mathbb{R}^{d}. A matching Le Cam lower bound shows that this rate is pointwise minimax optimal. However, the local rate-optimality requires kk to be calibrated to the smoothness of f0f_0, which is unknown in practice. We therefore provide a locally adaptive choice of kk that makes the neural network adapt to the local Hölder exponent of f0f_0 and to the intrinsic dimension simultaneously, both of which are potentially varying over the covariate domain. An extensive simulation study demonstrates the dominance of the locally adaptive neural network over a globally trained network and several state-of-the-art methods.

Paper details

vlnc:W_xud9bh5n
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
arXiv nonexclusive license
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…