### Evaluating Statistical Models (Advanced Data Analysis from an Elementary Point of View, Lecture 3)

The three big uses of statistical models: as summaries of data; as
predictive instruments; as scientific models. Evaluation depends on the use.
Prediction is the goal which admits of the most definite evaluations; reducing
the evaluation of scientific models to checking predictions (without
necessarily becoming an instrumentalists). Evaluating predictions by their
average errors: in-sample error distinguished from generalization error; the
latter is what really needs to be controlled. A gesture in the direction of
statistical learning theory. Over-fitting defined and illustrated.
Cross-validation for estimating generalization error and for model selection.
Forms of cross-validation; *k*-fold CV generally preferable to
leave-one-out CV.

PDF

R

Advanced Data Analysis from an Elementary Point of View

Posted at February 04, 2011 01:32 | permanent link