The three big uses of statistical models: as summaries of data; as predictive instruments; as scientific models. Evaluation depends on the use. Prediction is the goal which admits of the most definite evaluations; reducing the evaluation of scientific models to checking predictions (without necessarily becoming an instrumentalists). Evaluating predictions by their average errors: in-sample error distinguished from generalization error; the latter is what really needs to be controlled. A gesture in the direction of statistical learning theory. Over-fitting defined and illustrated. Cross-validation for estimating generalization error and for model selection. Forms of cross-validation; k-fold CV generally preferable to leave-one-out CV.
Posted at February 04, 2011 01:32 | permanent link