Independence Tests, Conditional Independence Tests, Measures of Dependence and Conditional Dependence
Last update: 23 Apr 2025 12:54First version: 25 March 2012
My interest in graphical models and Markov models make me very interested in the following related problems:
- Given samples of two (possible high-dimensional) random variables \( X \) and \( Y \), are they independent?
- Given samples of three random variables \( X \), \( Y \) and \( Z \), are \( X \) and \( Y \) conditionally independent given \( Z \)?
- How can we reliably calculate a measure of dependence from sample values?
- Ditto for a measure of conditional dependence.
One obvious approach to both the testing problems is to first calculate a good (conditional) dependence measure, and then see how unlikely that much apparent dependence would be if \( X \) and \( Y \) were really independent. Alternately, if one has a good dependence measure and can give confidence intervals for it (e.g., by some suitable bootstrap), see whether those confidence intervals include 0 (or whatever value indicates independence).
- See also:
- Density Estimation
- Entropy Estimation (mutual information being a fine measure of dependence)
- Statistics
- Two Sample Tests (since if \( X|Y=y_1 \) has a different distribution than \(X | Y=y_2 \), \( X \) and \( Y \) are dependent)
- Recommended (rather miscellaneous):
- Thomas B. Berrett, Richard J. Samworth, "Nonparametric independence testing via mutual information", arxiv:1711.06642
- Jochen Brocker, "A Lower Bound on Arbitrary \( f \)-Divergences in Terms of the Total Variation" arxiv:0903.1765
- Steve Fienberg, The Analysis of Cross-Classified Categorical Data
- Peter Hall, Jeff Racine and Qi Li, "Cross-Validation and the Estimation of Conditional Probability Densities", Journal of the American Statistical Association 99 (2004): 1015--1026 [PDF]
- Jeffrey D. Hart, Nonparametric Smoothing and Lack-of-Fit Tests [comments]
- Solomon W. Kullback, Information Theory and Statistics
- David Lopez-Paz, Philipp Hennig, Bernhard Schölkopf, "The Randomized Dependence Coefficient", arxiv:1304.7717
- Kun Zhang, Jonas Peters, Dominik Janzing, Bernhard Schölkopf, "Kernel-based Conditional Independence Test and Application in Causal Discovery", arxiv:1202.3775
- Modesty forbids me to recommend:
- Daniel J. McDonald, CRS and Mark Schervish
- "Estimating beta-mixing coefficients", AISTATS 2011
- "Estimating Beta-Mixing Coefficients via Histograms", arxiv:1109.5998
- Octavio César Mesner and CRS, "Conditional Mutual Information Estimation for Mixed Discrete and Continuous Variables with Nearest Neighbors", IEEE Transactions on Information Theory 67 (2021): 464--484, arxiv:1912.03387
- To read, dependence measures:
- Sophie Achard, "A quadratic measure of dependence", math.ST/0609259
- Shotaro Akaho, "A kernel method for canonical correlation analysis", cs.LG/0609071
- Sébastien Da Veiga, "Global Sensitivity Analysis with Dependence Measures", arxiv:1311.2483
- Gábor J. Székely and Maria L. Rizzo, "Brownian Distance Covariance", Annals of Applied Statistics 3 (2009): 1236--1265, arxiv:1010.0297 [I heard the talk, but should really read this...]
- Gábor J. Székely and Maria L. Rizzo and Nail K. Bakirov, "Measuring and testing dependence by correlation of distances", Annals of Statistics 35 (2007): 2769--2794, arxiv:0803.4101
- To read, conditional dependence measures:
- Mona Azadkia and Sourav Chatterjee, "A simple measure of conditional dependence", Annals of Statistics 49 (2021): 3070--3102, arxiv:1910.12327
- Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, Bernhard Schölkopf, "Kernel Measures of Conditional Dependence", NIPS 2007
- To read, independence testing:
- Arthur Gretton, László Györfi, "Consistent Nonparametric Tests of Independence", Journal of Machine Learning Research 11 (2010): 1391--1423
- Zhaolu Liu, Robert L. Peach, Felix Laumann, Sara Vallejo Mengod, Mauricio Barahona, "Kernel-based Joint Independence Tests for Multivariate Stationary and Nonstationary Time-Series", arxiv:2305.08529
- Gusztáv Morvai and Benjamin Weiss, "Testing stationary processes for independence", Annales de l'Institut Henri Poincaré, Probabilités et Statistiques 47 (2011): 1219--1225
- Xiaofeng Shao, "A generalized portmanteau test of independence between two stationary time series", arxiv:0810.2276
- To read, conditional independence testing:
- Thomas B. Berrett, Yi Wang, Rina Foygel Barber, Richard J. Samworth, "The conditional permutation test", arxiv:1807.05405
- Zhanrui Cai, Runze Li, Yaowu Zhang, "A Distribution Free Conditional Independence Test with Applications to Causal Discovery", Journal of Machine Learning Research 23 (2022): 85
- Tzee-Ming Huang, "Testing conditional independence using maximal nonlinear conditional correlation", Annals of Statistics 38 (2010): 2047--2091
- Shuai Li, Ziqi Chen, Hongtu Zhu, Christina Dan Wang, Wang Wen, "Nearest-Neighbor Sampling Based Conditional Independence Testing", arxiv:2304.04183
- Xiaotong Lin, Jie Xie, Fangqiao Tian, Dongming Huang, "Testing Multivariate Conditional Independence Using Exchangeable Sampling and Sufficient Statistics", arxiv:2504.06685
- Roman Pogodin, Antonin Schrab, Yazhe Li, Danica J. Sutherland, Arthur Gretton, "Practical Kernel Tests of Conditional Independence", arxiv:2402.13196
- Felipe Maia Polo, Yuekai Sun, Moulinath Banerjee, "Conditional independence testing under model misspecification", arxiv:2307.02520
