Conformal Prediction

Last update: 01 Jan 2025 11:01
First version: 25 July 2022

---

Yet Another Inadequate Placeholder. I have tried to explain the basic idea in my review of the (fundamental, but clunky) book by Vovk, Gammerman and Shafer (link below). I strongly recommend Samii's tutorial, and the group paper by Lei et al. 2016 (i.e., my friends and colleagues in the CMU statistics department). Adding this notebook in part to force myself to understand this better by the next time I teach data mining, so I can cover it then. (It has after all been more than a decade since I read Vovk et al.)

See also:
• Confidence Sets

Recommended, big picture:
• Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, Larry Wasserman, "Distribution-Free Predictive Inference For Regression", arxiv:1604.04173
• Cyrus Samii, Conformal Inference Tutorial
• Vladimir Vovk, Alex Gammerman and Glenn Shafer, Algorithmic Learning in a Random World [A brilliant and important, but badly-written, book; see the link for my review]

Recommended, close-ups:
• Jing Lei, Alessandro Rinaldo, Larry Wasserman, "A Conformal Prediction Approach to Explore Functional Data", arxiv:1302.6452
• Jing Lei, James Robins, and Larry Wasserman, "Efficient Nonparametric Conformal Prediction Regions", arxiv:1111.1418
• Jing Lei and Larry Wasserman, "Distribution Free Prediction Bands", arxiv:1203.5422

To read (with thanks to Shubhendu Trivedi's posts for some links):
• Anastasios N. Angelopoulos, Rina Foygel Barber, Stephen Bates, "Theoretical Foundations of Conformal Prediction", arxiv:2411.11824
• Anastasios N. Angelopoulos, Stephen Bates, "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification", arxiv:2107.07511 =? "Conformal Prediction: A Gentle Introduction", Foundations and Trends in Machine Learning 16 (2023): 494--591
• Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, Matteo Sesia, "Testing for Outliers with Conformal p-values", arxiv:2104.08279
• Richard A. Berk, Arun Kumar Kuchibhotla, "Improving Fairness in Criminal Justice Algorithmic Risk Assessments Using Conformal Prediction Sets", arxiv:2008.11664 [Heard the talk]
• Leonardo Cella, Ryan Martin, "Validity, consonant plausibility measures, and conformal prediction", arxiv:2001.09225
• Victor Chernozhukov, Kaspar Wüthrich, Yinchu Zhu, "An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls", arxiv:1712.09089 [i.e., causal inference]
• Leonardo Cella, Ryan Martin, "Valid inferential models for prediction in supervised learning problems", International Journal of Approximate Reasoning 150 (2022): 1--18, arxiv:2112.10234
• Tiffany Ding, Anastasios N. Angelopoulos, Stephen Bates, Michael I. Jordan, Ryan J. Tibshirani, "Class-Conditional Conformal Prediction With Many Classes", arxiv:2306.09335
• Lutz Duembgen, Bernd-Wolfgang Igl, Axel Munk, "P-values for classification", arxiv:0801.2934 [From the abstract, this sounds a lot like conformal prediction!]
• Jianqing Fan, Jiawei Ge, Debarghya Mukherjee, "UTOPIA: Universally Trainable Optimal Prediction Intervals Aggregation", arxiv:2306.16549
• Adam Fisch, Tal Schuster, Tommi Jaakkola, Regina Barzilay, "Conformal Prediction Sets with Limited False Positives", arxiv:2202.07650
• Isaac Gibbs, John J. Cherian, Emmanuel J. Candès, "Conformal Prediction With Conditional Guarantees", arxiv:2305.12616
• Chirag Gupta, Arun K. Kuchibhotla, Aaditya K. Ramdas, "Nested conformal prediction and quantile out-of-bag ensemble methods", arxiv:1910.10562
• Peter Hoff, "Bayes-optimal prediction with frequentist coverage control", arxiv:2105.14045
• Lihua Lei, Emmanuel J. Candès, "Conformal Inference of Counterfactuals and Individual Treatment Effects", arxiv:2006.06138 [Heard the talk...]
• Jordan Lekeufack, Anastasios N. Angelopoulos, Andrea Bajcsy, Michael I. Jordan, Jitendra Malik, "Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions", arxiv:2310.05921
• Tengyuan Liang, "Universal Prediction Band via Semi-Definite Programming", Journal of the Royal Statistical Society B 84 (2022): 1558--1580 [An alternative]
• Zhen Lin, Shubhendu Trivedi, Jimeng Sun
  • "Locally Valid and Discriminative Prediction Intervals for Deep Learning Models", arxiv:2106.00225
  • "Conformal Prediction Intervals with Temporal Dependence", arxiv:2205.12940
• Robert Lunde, "On the Validity of Conformal Prediction for Network Data Under Non-Uniform Sampling", arxiv:2306.07252
• Robert Lunde, Elizaveta Levina, Ji Zhu, "Conformal Prediction for Network-Assisted Regression", arxiv:2302.10095
• Buddhika Nettasinghe, Samrat Chatterjee, Ramakrishna Tipireddy, Mahantesh Halappanavar, "Extending Conformal Prediction to Hidden Markov Models with Exact Validity via de Finetti's Theorem for Markov Chains", arxiv:2210.02271
• Georgy Noarov, Ramya Ramalingam, Aaron Roth, Stephan Xie, "High-Dimensional Prediction for Sequential Decision Making", arxiv:2310.17651 [Framed as an alternative to conformal prediction, rather than an instance of it...]
• Aleksandr Podkopaev, Darren Xu, Kuang-Chih Lee, "Adaptive Conformal Inference by Betting", arxiv:2412.19318
• Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, Regina Barzilay, "Conformal Language Modeling", arxiv:2306.10193
• João Vitor Romano, Conformal Prediction Methods in Finance [MS thesis, Instituto de Matemática Pura a Aplicada, 2022]
• Yusuf Sale, Michele Caprio, Eyke Hüllermeier, "Is the Volume of a Credal Set a Good Measure for Epistemic Uncertainty?", arxiv:2306.09586
• Glenn Shafer and Vladimir Vovk, "A tutorial on conformal prediction", arxiv:0706.3188
• Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candes, Aaditya Ramdas, "Conformal Prediction Under Covariate Shift", arxiv:1904.06019
• Chen Xu, Yao Xie, "Conformal prediction for dynamic time-series", arxiv:2010.09107
• Frédéric Zheng, Alexandre Proutiere, "Conformal Predictions under Markovian Data", arxiv:2407.15277