Independence Tests, Conditional Independence Tests, Measures of Dependence and Conditional Dependence

Last update: 11 Dec 2024 10:43
First version: 25 March 2012

---

My interest in graphical models and Markov models make me very interested in the following related problems:

• Given samples of two (possible high-dimensional) random variables \( X \) and \( Y \), are they independent?
• Given samples of three random variables \( X \), \( Y \) and \( Z \), are \( X \) and \( Y \) conditionally independent given \( Z \)?
• How can we reliably calculate a measure of dependence from sample values?
• Ditto for a measure of conditional dependence.

See also:
• Density Estimation
• Entropy Estimation (mutual information being a fine measure of dependence)
• Statistics
• Two Sample Tests (since if \( X|Y=y_1 \) has a different distribution than \(X | Y=y_2 \), \( X \) and \( Y \) are dependent)

Recommended (rather miscellaneous):
• Thomas B. Berrett, Richard J. Samworth, "Nonparametric independence testing via mutual information", arxiv:1711.06642
• Jochen Brocker, "A Lower Bound on Arbitrary \( f \)-Divergences in Terms of the Total Variation" arxiv:0903.1765
• Steve Fienberg, The Analysis of Cross-Classified Categorical Data
• Peter Hall, Jeff Racine and Qi Li, "Cross-Validation and the Estimation of Conditional Probability Densities", Journal of the American Statistical Association 99 (2004): 1015--1026 [PDF]
• Jeffrey D. Hart, Nonparametric Smoothing and Lack-of-Fit Tests [comments]
• Solomon W. Kullback, Information Theory and Statistics
• David Lopez-Paz, Philipp Hennig, Bernhard Schölkopf, "The Randomized Dependence Coefficient", arxiv:1304.7717
• Kun Zhang, Jonas Peters, Dominik Janzing, Bernhard Schölkopf, "Kernel-based Conditional Independence Test and Application in Causal Discovery", arxiv:1202.3775

Modesty forbids me to recommend:
• Daniel J. McDonald, CRS and Mark Schervish
  • "Estimating beta-mixing coefficients", AISTATS 2011
  • "Estimating Beta-Mixing Coefficients via Histograms", arxiv:1109.5998
• Octavio César Mesner and CRS, "Conditional Mutual Information Estimation for Mixed Discrete and Continuous Variables with Nearest Neighbors", IEEE Transactions on Information Theory 67 (2021): 464--484, arxiv:1912.03387

To read:
• Sophie Achard, "A quadratic measure of dependence", math.ST/0609259
• Shotaro Akaho, "A kernel method for canonical correlation analysis", cs.LG/0609071
• Barry C. Arnold et al., Conditional Specification of Statistical Models
• Mona Azadkia and Sourav Chatterjee, "A simple measure of conditional dependence", Annals of Statistics 49 (2021): 3070--3102, arxiv:1910.12327
• Thomas B. Berrett, Yi Wang, Rina Foygel Barber, Richard J. Samworth, "The conditional permutation test", arxiv:1807.05405
• Patrice Bertail, Paul Doukhan and Philippe Soulier (eds.), Dependence in Probability and Statistics
• G. Biau and L. Gyorfi, "On the Asymptotic Properties of a Nonparametric $-Test Statistic of Homogeneity", IEEE Transactions on Information Theory 51 (2005): 3965--3973
• Patric Bonnier, Harald Oberhauser, "Signature Cumulants, Ordered Partitions, and Independence of Stochastic Processes", arxiv:1908.06496
• Zhanrui Cai, Runze Li, Yaowu Zhang, "A Distribution Free Conditional Independence Test with Applications to Causal Discovery", Journal of Machine Learning Research 23 (2022): 85
• Cox and Wermuth, Multivariate Dependencies: Models, Analysis and Interpretation
• Sébastien Da Veiga, "Global Sensitivity Analysis with Dependence Measures", arxiv:1311.2483
• Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, Bernhard Schölkopf, "Kernel Measures of Conditional Dependence", NIPS 2007
• Kenji Fukumizu, Le Song, Arthur Gretton, "Kernel Bayes' rule", arxiv:1009.5736
• Arthur Gretton, László Györfi, "Consistent Nonparametric Tests of Independence", Journal of Machine Learning Research 11 (2010): 1391--1423
• Tzee-Ming Huang, "Testing conditional independence using maximal nonlinear conditional correlation", Annals of Statistics 38 (2010): 2047--2091
• Shuai Li, Ziqi Chen, Hongtu Zhu, Christina Dan Wang, Wang Wen, "Nearest-Neighbor Sampling Based Conditional Independence Testing", arxiv:2304.04183
• Zhaolu Liu, Robert L. Peach, Felix Laumann, Sara Vallejo Mengod, Mauricio Barahona, "Kernel-based Joint Independence Tests for Multivariate Stationary and Nonstationary Time-Series", arxiv:2305.08529
• Gusztáv Morvai and Benjamin Weiss, "Testing stationary processes for independence", Annales de l'Institut Henri Poincaré, Probabilités et Statistiques 47 (2011): 1219--1225
• Leandro Pardo, Statistical Inference Based on Divergence Measures
• Roman Pogodin, Antonin Schrab, Yazhe Li, Danica J. Sutherland, Arthur Gretton, "Practical Kernel Tests of Conditional Independence", arxiv:2402.13196
• Felipe Maia Polo, Yuekai Sun, Moulinath Banerjee, "Conditional independence testing under model misspecification", arxiv:2307.02520
• Dino Sejdinovic, Bharath Sriperumbudur, Arthur Gretton, Kenji Fukumizu, "Equivalence of distance-based and RKHS-based statistics in hypothesis testing", Annals of Statistics 41 (2013): 2263--2291, arxiv:1207.6076
• Xiaofeng Shao, "A generalized portmanteau test of independence between two stationary time series", arxiv:0810.2276
• Gábor J. Székely and Maria L. Rizzo, "Brownian Distance Covariance", Annals of Applied Statistics 3 (2009): 1236--1265, arxiv:1010.0297
• Gábor J. Székely and Maria L. Rizzo and Nail K. Bakirov, "Measuring and testing dependence by correlation of distances", Annals of Statistics 35 (2007): 2769--2794, arxiv:0803.4101
• Olivier Thas, Comparing Distributions [mostly about goodness-of-fit tests]