Notebooks
http://bactra.org/notebooks
Cosma's NotebooksenPost-Model-Selection Inference
http://bactra.org/notebooks/2020/11/07#post-model-selection-inference
<P>Yet Another Inadequate Placeholder
<P><a href="model-selection.html">Model selection</a>, in statistics, means
using your data to pick the correct statistical model, or at least a good one.
Often we're interested in doing statistical inference with the selected model
--- we might want to know confidence sets for parameters (or functions), we
might want to attach measures of uncertainty to its predictions, etc. The
difficulty is that we usually calculate the properties of our inferential
procedures on the assumption of a <em>fixed</em> model, <a href="http://bactra.org/weblog/694.html">as though the
right model were communicated to us by the angels</a>. When instead it's
something we've selected using the data, there are going to be problems.
<P>The easiest way to see this may be to reflect that our data are random
(that's why we're doing statistics), so which model we get from our model
selection is also random (at least a little), and this will create
correlations between the selected model and the outputs of statistical
tests. If we're doing regression and we've used model selection to
<a href="variable-selection.html">pick which variables are included as
regressors</a>, <em>of course</em> the selected variables are going to look
significant on the data we used to pick them! (Thus the classic Freedman,
1983, which never fails to make a mind-blowing assignment for undergrads.) The
whole rest of this subject is essentially refining this basic observation.
<P>One direction of refinement is to try to develop new inferential procedures,
more or less approximate, which can compensate for the fact that our model was
picked in a data-dependent way. This is most of what gets called
"post-selection inference" or "post-model-selection inference" or "selective
inference". There is a lot of intricate theory here, often relying on clever
mathematical understanding of specific selection procedures and how they
interact with specific assumptions about the data-generating process.
<P>The other direction is to attack the problem at its root: using the
same data for selection and inference creates correlations between them, so
<em>use different data for selection and inference</em>. This gets called
"data splitting" or "sample splitting". It's easy to do for IID data ---
divide your data set, at random, into two parts, do your selection on one part,
and then do the inference on the other, with no cross-contamination. (This is
close to, but not quite, <a href="cross-validation.html">cross-validation</a>.)
Because they're independent, the selected model is independent of the contents
of the inference set, hence the usual procedures work with their usual
properties. Problem solved.
<P>Sample splitting is a simple, radical, almost a-theoretical way to solve the
problem of post-selection inference, and as such it appeals to my temperament.
(This is why two of my students wrote their dissertations, in part, on how to
extend it to dependent data, where, alas, theory and subtlety re-enter.) With
all sincere respect to those working heroically on what I called the other
direction, honestly don't know why the sample-splitting approach <em>isn't</em>
the default we all use.
<ul>Recommended (including by reference recommendations listed under <a href="model-selection.html">model selection</a>):
<li>Richard Berk, Lawrence Brown, Andreas Buja, Kai Zhang, and Linda Zhao, "Valid post-selection inference", <a href="http://dx.doi.org/10.1214/12-AOS1077"><cite>Annals of Statistics</cite> <strong>41</strong> (2013): 802--837</a>
<li>Julian J. Faraway
<ul>
<li>"On the Cost of Data Analysis", <a href="http://dx.doi.org/10.1080/10618600.1992.10474582"><cite>Journal of Computational and Graphical Statistics</cite> <strong>1</strong> (1992): 213--229</a> [<a href="http://people.bath.ac.uk/jjf23/papers/cda.pdf">PDF preprint</a>]
<li>"Does Data Splitting Improve Prediction?",
<a href="http://dx.doi.org/10.1007/s11222-014-9522-9"><cite>Statistics and Computing</cite> <strong>26</strong> (2016): 49--60</a>, <a href="http://arxiv.org/abs/1301.2983">arxiv:1301.2983</a>
</ul>
<li>William Fithian, Dennis Sun, Jonathan Taylor, "Optimal Inference After Model Selection", <a href="http://arxiv.org/abs/1410.2597">arxiv:1410.2597</a>
<li>David A. Freedman, "A Note on Screening Regression Equations",
<a href="https://doi.org/10.1080/00031305.1983.1048279"><cite>The American Statistician</cite> <strong>37</strong> (1983): 152--155</a>
<li>Jason D. Lee, Dennis L. Sun, Yuekai Sun, Jonathan E. Taylor, "Exact post-selection inference, with application to the lasso", <a href="http://arxiv.org/abs/1311.6238">arxiv:1311.6238</a>
<li><a href="http://www.stat.yale.edu/~hl284/">Hannes Leeb</a>
<ul>
<li>"Conditional Predictive Inference Post Model
Selection", <cite>Annals of Statistics</cite> <strong>37</strong> (2009):
2838--2876, <a href="http://arxiv.org/abs/0908.3615">arxiv:0908.3615</a>
<li>"Evaluation and selection of models for out-of-sample prediction when the sample size is small relative to the complexity of the data-generating process", <cite>Bernoulli</cite> <strong>14</strong> (2008): 661--690,
<a href="http://arxiv.org/abs/0802.3364">arxiv:0802.3364</a>
</ul>
<li>Hannes Leeb and Benedikt M. Pötscher
<ul>
<li>"Can One Estimate The
Unconditional Distribution of Post-Model-Selection Estimators?", <cite>Annals of Statistics</cite> <strong>34</strong> (2006): 2554--2591,
<a href="http://arxiv.org/abs/0704.1584">arxiv:0704.1584</a>
answer is "No".]
<li>"Model Selection and Inference: Facts and Fiction",
<a href="http://dx.doi.org/10+10170 S0266466605050036"><cite>Econometric
Theory</cite> <strong>21</strong> (2005): 21--59</a>
[<a href="http://www.stat.yale.edu/~hl284/ETAnniv.pdf">PDF reprint</a>]
</ul>
<li>Alessandro Rinaldo, Larry Wasserman, Max G'Sell, Jing Lei, "Bootstrapping and Sample Splitting For High-Dimensional, Assumption-Free Inference", <a href="http://arxiv.org/abs/1611.05401">arxiv:1611.05401</a> [Disclaimer: All
colleagues and friends]
</ul>
<ul>Pride compels me to recommend:
<li>Robert Lunde, <cite>Bootstrapping and Sample Splitting Under Weak Dependence</cite> [Ph.D. thesis, CMU Statistics, 2018]
<li>Lawrence Wang, <cite>Network Comparisons using Sample Splitting</cite> [Ph.D. thesis, CMU Statistics, 2016]
</ul>
<ul>To read:
<li>Alexandre Belloni, Victor Chernozhukov, Ivan Fernández-Val, Christian Hansen, "Program Evaluation and Causal Inference with High-Dimensional Data",
<cite>Econometrica</cite> <strong>85</strong> (2017): 233--298</a>,
<a href="http://arxiv.org/abs/1311.2645">arxiv:1311.2645</a>
<li>Yoav Benjamini, Marina Bogomolov, "Adjusting for selection bias in testing multiple families of hypotheses", <a href="http://arxiv.org/abs/1106.3670">arxiv:1106.3670</a>
<li>Gavin C. Cawley, Nicola L. C. Talbot, "On Over-fitting in Model
Selection and Subsequent Selection Bias in Performance
Evaluation", <a href="http://jmlr.csail.mit.edu/papers/v11/cawley10a.html"><cite>Journal
of Machine Learning Research</cite> <strong>11</strong> (2010): 2079--2107</a>
<li>Victor Chernozhukov, Christian Hansen, Martin Spindler, "Valid Post-Selection and Post-Regularization Inference: An Elementary, General Approach",
<a href="https://doi.org/10.1146/annurev-economics-012315-015826"><cite>Annual Review of Economics</cite> <strong>7</strong> (2015): 649--688</a>, <a href="http://arxiv.org/abs/1501.03430">arxiv:1501.03430</a>
<li>Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, Aaron Roth, "Preserving Statistical Validity in Adaptive Data Analysis", <a href="http://arxiv.org/abs/1411.2664">arxiv:1411.2664</a>
<li>Karl Ewald, Ulrike Schneider, "Uniformly Valid Confidence Sets Based on the Lasso", <a href="https://doi.org/10.1214/18-EJS1425"><cite>Electronic Journal of Statistics</cite> <strong>12</strong> (2018): 1358--1387</a>, <a href="http://arxiv.org/abs/1507.05315">arxiv:1507.05315</a>
<li>Paul Kabaila and Khageswor Giri, "Upper bounds on the minimum coverage probability of confidence intervals in regression after variable selection", <a href="http://arxiv.org/abs/0711.0993">arxiv:0711.0993</a>
<li>Benedikt M. Pötscher [I've heard Prof. Pötscher talk about this work multiple times now at conferences, but I still need to really master it]
<ul>
<li>"The distribution of model averaging
estimators and an impossibility result regarding its estimation", <a href="http://arxiv.org/abs/math/0702781">arxiv:math/0702781</a>
<li>"Confidence sets based on sparse estimators
are necessarily large", <a href="http://arxiv.org/abs/0711.1036">arxiv:0711.1036</a>
</ul>
<li>Yoshikazu Terada, Hidetoshi Shimodaira, "Selective inference after variable selection via multiscale bootstrap", <a href="http://arxiv.org/abs/1905.10573">arxiv:1905.10573</a> [I presume they have an answer to "why not just use sample splitting?"]
<li>Xiaoying Tian, Jonathan Taylor, "Asymptotics of selective inference", <a href="http://arxiv.org/abs/1501.03588">arxiv:1501.03588</a>
</ul>