## Empirical Process Theory

*28 Jun 2023 12:48*

(I first used the next few paragraphs as part of a review of Pollard's book of lecture notes. I have no shame about self-plagiarism.)

The simplest sort of empirical process arises when trying to estimate a
probability distribution from sample data. The difference between the
empirical
distribution function \( F_n(x) \) and the true distribution function \(
F(x) \) converges to zero everywhere (by the law of large numbers), and —
this is non-trivial — the *maximum* difference between the
empirical and true distribution functions converges to zero, too (by
the Glivenko-Cantelli
theorem, a *uniform* law of large numbers). The "empirical process" \(
E_n(x) \) is the re-scaled difference, \( n^{1/2} \left[ F_n(x) - F(x) \right]
\), and *it* converges to a Gaussian stochastic process that only
depends on the true distribution (by
the functional
central limit theorem). Empirical process theory is concerned with
generalizing this sort of material to other stochastic processes determined by
random samples, and indexed by infinite classes (like the real line, or the
class of all Borel sets on the line, or some space parameterizing
a regression model). The typical objects of
concern are proving uniform limit theorems, and with establishing
distributional limits. (For instance, one might one want to prove that the
errors of *all* possible regression models in some class will come close
to their expected errors, so that maximum-likelihood or least-squares
estimation is consistent. [For more on that line of thought,
see Sara van de Geer's
book.]) This endeavor is closely linked to
Vapnik-Chervonenkis-style learning theory,
and in fact one can see VC theory as an application of empirical process
theory.

As usual, I am most interested in results for dependent data.

See also: Concentration of Measure

- Recommended:
- David Pollard
- "Asymptotics via Empirical Processes",
Statistical Science
**4**(1989): 341--354 - Convergence of Stochastic Processes
- Empirical Processes: Theory and Applications

- "Asymptotics via Empirical Processes",
Statistical Science
- Sara van de Geer, Empirical Processes in M-Estimation

- Recommended, close-ups:
- Patrizia Berti, Irene Crimaldi, Luca Pratelli, and Pietro Rigo,
"Rate of convergence of predictive distributions for dependent data",
Bernoulli
**15**(2009): 1351--1367 [Only for exchangeable sequences, sadly] - Bruce E. Hansen
- "The Likelihood Ratio Test Under Nonstandard
Conditions: Testing the Markov Switching Model of GNP", Journal of
Applied Econometrics
**7**(1992): S61--S82 [I very much like the approach of treating the likelihood ratio as an empirical process; why haven't I seen it before? (Also, the state-of-the-art in simulating Gaussian processes must be much better now than what Hansen had in '92, which would make this even more practical.) PDF reprint.] - "Inference when a nuisance parameter is not identified under the null hypothesis", Econometrica
**64**(1996): 413--430

- "The Likelihood Ratio Test Under Nonstandard
Conditions: Testing the Markov Switching Model of GNP", Journal of
Applied Econometrics
- Thomas Lumley, "A plug-in uniform law of large numbers", 28 September 2022, Biased and Inefficient
- Pascal Massart, Concentration Inequalities and Model Selection [Using empirical process theory to get finite-sample, i.e., non-asymptotic, risk bounds for various forms of model selection. Available for free as a large PDF preprint. My mini-review]
- Maxim Raginsky, "Empirical processes, typical sequences and coordinated actions in standard Borel spaces", IEEE Transactions on Information Theory
**59**(2013): 1288--1301, arxiv:1009.0282 - Ramon van Handel, "The universal Glivenko-Cantelli property", arxiv:1009.4434
- Mathukumalli Vidyasagar, A Theory of Learning and Generalization: With Applications to Neural Networks and Control Systems [Mini-review]
- Bin Yu, "Rates of Convergence for Empirical Processes of Stationary
Mixing Sequences," Annals of Probability
**22**(1994): 94--116

- To read:
- Radoslaw Adamczak, "A tail inequality for suprema of unbounded empirical processes with applications to Markov chains", arxiv:0709.3110
- Radoslaw Adamczak, Witold Bednorz, "Exponential Concentration Inequalities for Additive Functionals of Markov Chains", arxiv:1201.3569
- Donald W. K. Andrews and David Pollard, "An Introduction
to Functional Central Limit theorems for Dependent Stochastic Processes",
International Statistical Review
**62**91994): 119--132 [PDF reprint] - Miguel A. Arcones and Evarist Gine, "Limit Theorems for U-Processes",
Annals of Probability
**21**(1993): 1494--1542 - Yannick Baraud, "A Bernstein-type inequality for suprema of random processes with an application to statistics", arxiv:0904.3295
- Eric Beutner, Henryk Zähle, "Continuous mapping approach to the asymptotics of U- and V-statistics", arxiv:1203.1112
- S.G. Bobkov and F. Götze, "Concentration of empirical distribution functions with applications to non-i.i.d. models",
Bernoulli
**16**(2010): 1385--1414 - Axel Bücher, Johan Segers, Stanislav Volgushev, "When uniform weak convergence fails: empirical processes for dependence functions via epi- and hypographs", arxiv:1305.6408
- Jannis Buchsteiner, "Weak Convergence of the Sequential Empirical Process of some Long-Range Dependent Sequences with Respect to a Weighted Norm",
Statistics and Probability Letters
**96**(2015): 170--179, arxiv:1312.5894 - Victor Chernozhukov, Denis Chetverikov, Kengo Kato, "Gaussian
approximation of suprema of empirical processes",
Annals of Statistics
**42**(2014): 1564--1597, arxiv:1212.6885 - Yifeng Chu, Maxim Raginsky, "Majorizing Measures, Codes, and Information", arxiv:2305.02960
- Rainer Dahlhaus and Wolfgang Polonik, "Empirical spectral processes for locally stationary time series", Bernoulli
**15**(2009): 1--39, arxiv:902.1448 - Paul Deheuvels and Sarah Ouadah, "Uniform-in-Bandwidth Functional Limit Laws", Journal of Theoretical Probability
**26**(2013): 697--721 - Herold Dehling (ed.), Empirical Process Techniques for Dependent Data
- Herold Dehling and Olivier Durieu, "Empirical Processes of Multidimensional Systems with Multiple Mixing Properties", arxiv:1004.1088
- Herold Dehling, Olivier Durieu, Marco Tusche
- "Empirical Processes of Markov Chains and Dynamical Systems Indexed by Classes of Functions", arxiv:1201.2256
- "Approximating class approach for empirical processes of dependent sequences indexed by functions", Bernoulli
**20**(2014): 1372--1403

- Herold Dehling, Olivier Durieu and Dalibor Volny, "New Techniques for Empirical Process of Dependent Data", arxiv:0806.2941
- Eustacio del Barrio, Paul Deheuvels and Sara van de Geer, Lectures on Empirical Processes: Theory and Statistical Applications
- Emanuele Dolera and Eugenio Regazzini, "Uniform rates of the Glivenkoâ€“Cantelli convergence and their use in approximating Bayesian inferences",
Bernoulli
**25**(2019): 2982--3015 - P. Doukhan, P. Massart and E. Rio, "Invariance principles for absolutely regular empirical processes", Annales de l'institut Henri Poincaré B
**31**(1995): 393--427 - Lutz Duembgen, Perla Zerial, "On Low-Dimensional Projections of High-Dimensional Distributions", arxiv:1107.0417
- Olivier Durieu, Marco Tusche, "An Empirical Process Central Limit Theorem for Multidimensional Dependent Data",
Journal of Theoretical
Probability
**27**(2014): 249--277, arxiv:1110.0963 - Omar El-Dakkak, "Limit Behaviour of Sequential Empirical Measure Processes", arxiv:0810.5565
- James M. Feagin, Weighted Empirical Processes in Dynamic Nonlinear Models
- Daniel J. Fresen, Richard A. Vitale, "Concentration of random polytopes around the expected convex hull", arxiv:1402.2718
- Robert Hable, "Asymptotic Normality of Support Vector Machines for Classification and Regression", arxiv:1010.0535
- Bruce E. Hansen, "Stochastic Equicontinuity for Unbounded Dependent
Heterogeneous Arrays", Econometric Theory
**12**(1996): 347--359 [PDF reprint via Prof. Hansen] - Jikai Hou, "Empirical Process of Multivariate Gaussian under General Dependence", arxiv:1910.09319
- Michael R. Kosorok, Introduction to Empirical Processes and Semiparametric Inference [PDF preprint]
- James Kuelbs, Thomas Kurtz, Joel Zinn, "A CLT for Empirical Processes Involving Time Dependent Data", arxiv:1008.2697
- Johannes C. Lederer, Sara A. van de Geer, "New Concentration Inequalities for Suprema of Empirical Processes",
Bernoulli
**20**(2014): 2020--2038, arxiv:1111.3486 - Ji Hyung Lee and Kyungchul Song, "Stable limit theorems for empirical processes under conditional neighborhood dependence",
Bernoulli
**25**(2019): 1189--1224 - Jean-Francois Marckert, "One more approach to the convergence of the empirical process to the Brownian bridge", arxiv:0710.3296
- D. Marinucci, "The Empirical Process for Bivariate Sequences with
Long Memory", Statistical Inference
for Stochastic Processes
**8**(2005): 205--224 - Song Mei, Yu Bai, Andrea Montanari, "The Landscape of Empirical Risk for Non-convex Losses", arxiv:1607.06534
- Shahar Mendelson, "On weakly bounded empirical processes", arxiv:math/0512554
- Shahar Mendelson, Grigoris Paouris, "On generic chaining and the smallest singular value of random matrices with heavy tails", arxiv:1108.3886 ["We present a very general chaining method which allows one to control the supremum of the empirical process $\sup_{h \in H} |N^{-1}\sum_{i=1}^N h^2(X_i)-\E h^2|$ in rather general situations..."]
- Whitney K. Newey, "Uniform Conference in Probability and Stochastic Equicontinuity", Princeton Econometric Research Program, Research Memorandum No, 342 (1989) [Check for formally-published version?]
- Dragan Radulović, Marten Wegkamp, "Uniform Central Limit Theorems for pregaussian classes of functions", pp. 84--102 in Christian Houdré, Vladimir Koltchinskii, David M. Mason and Magda Peligrad (eds.) High Dimensional Probability V: The Luminy Volume
- Richard Samworth and Oliver Johnson, "The empirical process in Mallows distance, with application to goodness-of-fit tests", math.ST/0504424
- Galen R. Shorack and Jon A. Wellner, Empirical Processes with Applications to Statistics
- Michal Talagrand
- "Majorizing measures: the generic chaining",
Annals of Probability
**24**(1996): 1049--1103 - The Generic Chaining: Upper and Lower Bounds of Stochastic Processes
- Upper and Lower Bounds for Stochastic Processes

- "Majorizing measures: the generic chaining",
Annals of Probability
- Sara van de Geer and Johannes Lederer, "The Bernstein-Orlicz norm and deviation inequalities",
Probability Theory and Related Fields
**157**(2013): 225--250 arxiv:1111.2450 - Aad W. van der Vaart, Jon A. Wellner
- Weak Convergence and Empirical Processes: With Applications to Statistics
- "Empirical processes indexed by estimated functions", arxiv:0709.1013 ["We consider the convergence of empirical processes indexed by functions that depend on an estimated parameter $\eta$ and give several alternative conditions under which the ``estimated parameter'' $\eta_n$ can be replaced by its natural limit $\eta_0$ uniformly in some other indexing set $\Theta$"]
- "A local maximal inequality under uniform entropy",
Electronic Journal of Statistics
**5**(2011): 192--203

- Ramon van Handel
- "Chaining, Interpolation, and Convexity", arxiv:1508.05906
- Probability in High Dimension [PDF lecture notes]

- Vincent Q. Vu and Jing Lei, "Squared-Norm Empirical Process in Banach Space", arxiv:1312.1005
- Chao Zhang, "Bennett-type Generalization Bounds: Large-deviation Case and Faster Rate of Convergence", arxiv:1309.6876
- Chao Zhang and Dachen Tao, "Generalization Bound for Infinitely Divisible Empirical Process", Journal of Machine Learning Research Workshops and Conference Proceedings
**15**(2011): 864--872