Empirical process theory and applications

Objective

Empirical process theory is mainly about extending the law of large numbers (LLN) and central limit theorem (CLT) to uniform LLN's and CLT's. For example, suppose we take a sample of size n from some distribution. Then we know by the law of large numbers that for each set A, the proportion of observations in the set A converges as n tends to infinity, to the probability of the set A. We address questions like: over what collectios of sets A is the convergence uniform? Why would this be an interesting topic for a (theoretical) statistician? The answer is simple: statisticians often model data as being a sample from some unknown distribution. The problem is to estimate certain aspects of the unknown distribution. By some uniform LLN or CLT, we will know that certain averages in the sample will be uniformly close to their expectations. For example, after giving it some thought one sees that a uniform LLN is useful for showing consistency of maximum likelihood estimators. In fact, with empirical process theory, we cannot only make elegant proofs of mathematical statistical results, but also gain good insight into how statistical inference is related to complexity theory.

Content

We will (at least) study the following subjects: - Exponential inequalities for the deviation of averages from their mean. - Vapnik Chervonenkis dimension: a combinatorial concept of the "size" of a collection of sets A. The concept comes from learning theory. - M-estimators, such as maximum likelihood, least squares and other empirical risk minizers. - Consistency, rates of convergence and asymptotic normality of estimators. - Nonparametric theory (+ complexity regularization ?).

Resources