## Factor Models in Statistics

*09 Nov 2017 11:02*

\[ \newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]} \]

Factor models are a specific kind of latent-variable model in multivariate statistics, where the latent variables and the observable variables are both continuous, and the relationship between the two is linear. The basic form is that the latent variable $F$ is a $q$-dimensional vector, the observables form a $p$-dimensional vector $X$, and the relationship is \[ X = F\mathbf{w} + \epsilon \] for a $q\times p$ matrix of "loadings" $\mathbf{w}$, and a $p$-dimensional noise vector $\epsilon$, which is assumed to be independent of, or at least uncorrelated with, the factor vector $F$, and with no correlations between the different coordinates of $\epsilon$. The covariance matrix of $\epsilon$ is thus a diagonal matrix $\mathbf{\psi}$, and the covariance of $X$ is \[ \Var{X} = \mathbf{w}^T \Var{F} \mathbf{w} + \mathbf{\psi} \]

Now, if $\Var{F}$ is anything other than the identity matrix $\mathbf{I}_q$, we can always use principal components analysis / singular value decomposition to define a new $q$-dimensional vector $G$ where $\Var{G} = \mathbf{I}_q$ and $F = G\mathbf{r}$, for a $q\times q$ matrix $\mathbf{r}$. Then, since the relationship of $F$ and $X$ is linear, we also have \[ X=G \mathbf{r} \mathbf{w} +\epsilon = G \mathbf{u} + \epsilon \]

From a purely statistical point of view, therefore, we can always take the factor variables to have unit variance and no correlations. The covariance matrix of $X$ then is $\mathbf{w}^T \mathbf{w} + \mathbf{\psi}$. Since $\mathbf{w}$ has only $q$ rows, this means that $\mathbf{w}^T \mathbf{w}$ has rank $q$, the covariance matrix is of the form often called "low rank plus noise".

Geometrically, as $F$ ranges through $\mathbb{R}^q$, the "structured" part
of $X$, $F\mathbf{w}$, traces out a $q$-dimensional linear subspace in
$\mathbb{R}^p$. Taking each row of $\mathbf{w}$ as a vector, that subspace is
the span of those vectors, and I'll abuse notation to write it
$\mathrm{span}(\mathbf{w})$. The distribution of $F\mathbf{w}$ is the
distribution of values on $\mathrm{span}(\mathbf{w})$; $\epsilon$ is the
distribution of perturbations off of $\mathrm{span}(\mathbf{w})$. The use of a
particular $F$ is the choice of a particular coordinate system on
$\mathrm{span}(\mathbf{w})$, but statistically any coordinate system is equally
good, because it won't change the distribution of $F\mathbf{w}$. In fact, for
any linear transformation of coordinates, where $F = G \mathbf{r}$, we also
have $F\mathbf{w} = G \mathbf{r}\mathbf{w}$, so if we re-define the loading
matrix to $\mathbf{r}\mathbf{w}$, we have another factor model which predicts
exactly the same distribution of observables, and $ \mathrm{span}(\mathbf{w})
= \mathrm{span}(\mathbf{r}\mathbf{w})$. (Nonlinear changes of coordinates,
e.g., from Cartesian to polar, would break the linear relationship between
the *coordinates* of $F$ and those of $X$, even though the linear
relationship between *vectors* would remain.) The
maximal identifiable parameters of
the factor model are thus the distribution of $F\mathbf{w}$ and
$\mathbf{\psi}$, *not* $\mathbf{w}$ or the distribution of $F$. (The
subspace $\mathrm{span}(\mathbf{w})$ is identified because it's implied by a
knowledge of the distribution of $F\mathbf{w}$.) Of course, we might have
reason based on some other source of knowledge to prefer one coordinate system
over another, but this cannot come from the distribution of $X$, the observable
variables. Factor models can thus be seen as an example
of manifold learning, with the special
assumption that the manifold is a linear subspace.

("Confirmatory" factor analysis does not really change that last conclusion.
It essentially tests goodness of fit of a unrestricted estimate of $\mathbf{w}$
against the fit of a restricted model where one or more entries of $\mathbf{w}$
are fixed a priori, usually to 0. But any loading matrix with those
restrictions is observationally equivalent to a matrix $\mathbf{u} =
\mathbf{r}\mathbf{w}$ where $\mathbf{r}$ is any linear coordinate change, and a
continuous infinity of those $\mathbf{u}$ will *not* have the desired
0s. Each zero in some coordinate system really imposes a one-equation
algebraic constraint on $\mathbf{w}^T \mathbf{w}$, and we're really testing
those restrictions.)

I should explain how factor analysis is related to but different from the principal components, but having just been writing about this in my lecture notes, I'll just refer to that (link below).

*Questions:* In large dimensions, how different do random covariance
matrices look from low-rank-plus-noise matrices? For every $q$-factor model
with given means and covariance matrices, there is a mixture model with $q+1$
discrete clusters and the same means and covariances; is the distinction
between them identifiable?

Notoriously, factor analysis comes out of psychology, and the attempt to "construct" general mental attributes from test scores. But I have said more than enough about that elsewhere.

See also: Graphical Models; Manifold Learning; Mixture Models; The Thomson Ability-Sampling Model

- Recommended, big picture:
- David J. Bartholomew, Latent Variable Models and Factor Analysis

- Recommended, close ups:
- J. Scott Armstrong, "Derivation of theory by means of factor analysis or Tom Swift and his electric factor analysis machine", The American Statistician
**21:5**(1967): 17--21 [Reprint] - Dani Gamerman, Hedibert Freitas Lopes, and Esther Salazar, "Spatial
dynamic factor
analysis", Bayesian
Analysis
**3**(2008): 759--792 - Yi-hao Kao and Benjamin Van Roy, "Learning a Factor Model via Regularized PCA", Machine Learning
**91**(2013): 279--303, arxiv:1111.6201 - Wim Krijnen, "Positive Loadings and Factor Correlations from Positive Covariances", Psychometrika
**69**(2004): 655--660 - John C. Loehlin, Latent Variable Models: An Introduction to Factor, Path, and Structural Analysis
- Robert A. Peterson, "A Meta-Analysis of Variance Accounted for and Factor Loadings in Exploratory Factor Analysis", Marketing Letters
**11**(2000): 261--275 - Kristopher J. Preacher and Robert C. MacCallum, "Repairing Tom Swift's Electric Factor Analysis Machine", Understanding Statistics
**2**(2003): 13--43 [The objection about nonlinearity seems deeply unfair to me: if one is actually using factor analysis to*discover*things, when would one be willing to assert the linearity? Similarly, how is one to know which rotation of the factors is better? By definition, the different rotations are all empirically equivalent, so in a genuinely new domain we don't know which ones group related quantities together. Reprint] - Mohamed Saidane, Xavier Bry and Christian Lavergne,
"Generalized Linear Factor Models: A New Local EM Estimation Algorithm",
Communications in Statistics: Theory and
Methods
**42**(2013): 2944--2958

- Recommended, historical:
- Charles Spearman, "``General Intelligence,'' Objectively Determined
and Measured", American Journal of Psychology
**15**(1904): 201--293 [Online] - Godfrey H. Thomson, The Factorial Analysis of Human Ability
- L. L. Thurstone, "The Vectors of Mind", Psychological Review
**41**(1934): 1--32 [Online]

- Modesty forbids me to recommend:
- The chapters on principal components and factor analysis in Advanced Data Analysis from an Elementary Point of View

- To read:
- Umberto Amato, Anestis Antoniadis, Alexander Samarov, Alexander Tsybakov, "Noisy Independent Factor Analysis Model for Density Estimation and Classification", arxiv:0906.2885
- Ery Arias-Castro, Sébastien Bubeck, and Gábor Lugosi,
"Detection of
correlations", Annals of
Statistics
**40**(2012): 412--435 - Jushan Bai and Kunpeng Li, "Statistical analysis of factor models
of high dimension", Annals
of Statistics
**40**(2012): 436--465 - Tony Cai, Zongming Ma, Yihong Wu, "Optimal Estimation and Rank Detection for Sparse Spiked Covariance Matrices", arxiv:1305.3235
- Venkat Chandrasekaran, Pablo A. Parrilo, and Alan S. Willsky,
"Latent variable graphical model selection via convex optimization",
Annals of Statistics
**40**(2012): 1935--1967, arxiv:1008.1290 - John P. Cunningham, Zoubin Ghahramani, "Unifying linear dimensionality reduction", arxiv:1406.0873
- Jianqing Fan, Yuan Liao, and Martina Mincheva,
"High-dimensional covariance matrix estimation in approximate factor models",
Annals of Statistics
**39**(2011): 3320--3356 - Réami Gribonval, Rodolphe Jenatton, Francis Bach, Martin Kleinsteuber, Matthias Seibert, "Sample Complexity of Dictionary Learning and other Matrix Factorizations", arxiv:1312.3790
- Giles Hooker, Steven Roberts, "Maximal Autocorrelation Functions in Functional Data Analysis", arxiv:1407.4578
- V. I. Koltchinskii, "Empirical geometry of multivariate data: a deconvolution approach", Annals of Statistics
**28**(2000): 591--629 - Clifford Lam and Qiwei Yao, "Factor modeling for high-dimensional time series: Inference for the number of factors", Annals of Statistics
**40**(2012): 694--726 - Roderick P. McDonald, "A simple comprehensive model for the analysis of covariance structures: Some remarks on applications",
British
Journal of Mathematical and Statistical Psychology
**33**(1980): 161-–183 - Emanuel Moench, Serena Ng, Simon Potter, "Dynamic Hierarchical Factor Models",
Review of Economics
and Statistics
**95**(2013): 1811--1817 - Janek Musek, "A general factor of personality: Evidence for
the Big One in the five-factor model", Journal of Research in Personality
**41**(2007): 1213--1233 - Patrick O. Perry, Art B. Owen, "A Rotation Test to Verify Latent Structure", Journal of Machine Learning Research
**11**(2010): 603--624 - Yichuan Tang, Ruslan Salakhutdinov, Geoffrey Hinton, "Deep Mixtures of Factor Analysers", arxiv:1206.4635
- Kai Zhang, "Rank-Extreme Association of Gaussian Vectors and Low-Rank Detection", arxiv:1306.0623

- To write:
- CRS, "General Factors in Correlational Psychology: Artifacts and Myths"
- CRS, "Semi-Parametric Generalized-Linear Factor Models"