Notebooks

## Factor Models in Statistics

30 Nov 2019 09:48

$\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}$

Factor models are a specific kind of latent-variable model in multivariate statistics, where the latent variables and the observable variables are both continuous, and the relationship between the two is linear. The basic form is that the latent variable $F$ is a $q$-dimensional vector, the observables form a $p$-dimensional vector $X$, and the relationship is $X = F\mathbf{w} + \epsilon$ for a $q\times p$ matrix of "loadings" $\mathbf{w}$, and a $p$-dimensional noise vector $\epsilon$, which is assumed to be independent of, or at least uncorrelated with, the factor vector $F$, and with no correlations between the different coordinates of $\epsilon$. The covariance matrix of $\epsilon$ is thus a diagonal matrix $\mathbf{\psi}$, and the covariance of $X$ is $\Var{X} = \mathbf{w}^T \Var{F} \mathbf{w} + \mathbf{\psi}$

Now, if $\Var{F}$ is anything other than the identity matrix $\mathbf{I}_q$, we can always use principal components analysis / singular value decomposition to define a new $q$-dimensional vector $G$ where $\Var{G} = \mathbf{I}_q$ and $F = G\mathbf{r}$, for a $q\times q$ matrix $\mathbf{r}$. Then, since the relationship of $F$ and $X$ is linear, we also have $X=G \mathbf{r} \mathbf{w} +\epsilon \equiv G \mathbf{w}^{\prime} + \epsilon$

From a purely statistical point of view, therefore, we can always take the factor variables to have unit variance and no correlations. The covariance matrix of $X$ is thus $\Var{X} = \mathbf{w}^T \mathbf{w} + \mathbf{\psi}$ Since $\mathbf{w}$ has only $q$ rows, this means that $\mathbf{w}^T \mathbf{w}$ has rank $q$, the covariance matrix is of the form often called "low rank plus noise".

Geometrically, as $F$ ranges through $\mathbb{R}^q$, the "structured" part of $X$, $F\mathbf{w}$, traces out a $q$-dimensional linear subspace in $\mathbb{R}^p$. Taking each row of $\mathbf{w}$ as a vector, that subspace is the span of those vectors, and I'll abuse notation to write it $\mathrm{span}(\mathbf{w})$. The distribution of $F\mathbf{w}$ is the distribution of values on $\mathrm{span}(\mathbf{w})$; $\epsilon$ is the distribution of perturbations off of $\mathrm{span}(\mathbf{w})$. The use of a particular $F$ is the choice of a particular coordinate system on $\mathrm{span}(\mathbf{w})$, but statistically any coordinate system is equally good, because it won't change the distribution of $F\mathbf{w}$. In fact, for any linear transformation of coordinates, where $F = G \mathbf{r}$, we also have $F\mathbf{w} = G \mathbf{r}\mathbf{w}$, so if we re-define the loading matrix to $\mathbf{r}\mathbf{w}$, we have another factor model which predicts exactly the same distribution of observables, and $\mathrm{span}(\mathbf{w}) = \mathrm{span}(\mathbf{r}\mathbf{w})$. (Nonlinear changes of coordinates, e.g., from Cartesian to polar, would break the linear relationship between the coordinates of $F$ and those of $X$, even though the linear relationship between vectors would remain.) The maximal identifiable parameters of the factor model are thus the distribution of $F\mathbf{w}$ and $\mathbf{\psi}$, not $\mathbf{w}$ or the distribution of $F$. (The subspace $\mathrm{span}(\mathbf{w})$ is identified because it's implied by a knowledge of the distribution of $F\mathbf{w}$.) Of course, we might have reason based on some other source of knowledge to prefer one coordinate system over another, but this cannot come from the distribution of $X$, the observable variables. Factor models can thus be seen as an example of manifold learning, with the special assumption that the manifold is a linear subspace.

("Confirmatory" factor analysis does not really change that last conclusion. It essentially tests goodness of fit of a unrestricted estimate of $\mathbf{w}$ against the fit of a restricted model where one or more entries of $\mathbf{w}$ are fixed a priori, usually to 0. But any loading matrix with those restrictions is observationally equivalent to a matrix $\mathbf{u} = \mathbf{r}\mathbf{w}$ where $\mathbf{r}$ is any linear coordinate change, and a continuous infinity of those $\mathbf{u}$ will not have the desired 0s. Each zero in some coordinate system really imposes a one-equation algebraic constraint on $\mathbf{w}^T \mathbf{w}$, and we're really testing those restrictions.)

I remarked above that in a factor model, the covariance matrix of the observables $\Var{X}$ always takes the form $\mathbf{w}^T \mathbf{w} + \mathbf{\psi}$. Suppose that all of the entries in $\Var{X}$ are non-negative, i.e., all of the observable variables are positively correlated with each other (or at worst un-correlated). Now, it's evident that the off-diagonal entries of $\Var{X}$ and $\mathbf{w}^T\mathbf{w}$ are equal, because $\mathbf{\psi}$ is a diagonal matrix. But the diagonal entries in $\mathbf{w}^T\mathbf{w}$ are the variances of $F\mathbf{w}$ along the observable coordinates, so they must be non-negative as well. Hence if $\Var{X}$ has only non-negative entries, so does $\mathbf{w}^T\mathbf{w}$. At this point, the Frobenius-Perron theorem of linear algebra tells us that (i) all of the eigenvalues of $\mathbf{w}^T\mathbf{w}$ are non-negative, and (ii) the eigenvector corresponding to the largest eigenvalue has all positive entries. We write $\mathbf{w}^T \mathbf{w} = \mathbf{v}^T \mathbf{\lambda} \mathbf{v}$, with $\mathf{\lambda}$ the diagonal, non-negative matrix of eigenvalues and $\mathbf{v}$ the corresponding matrix of eigenvectors. (In general, Frobenius-Perron allows for different left and right eigenvectors, but $\mathbf{w}^T\mathbf{w}$ is manifestly symmetric.) We can thus choose to set $\mathbf{w} = \mathbf{\lambda}^{1/2} \mathbf{v}$. In context, this means we can choose a coordinate system for $F$ where each latent coordinate (i) "explains" some (positive) share of the variance of the observables, and (ii) the latent coordinate which "explains" the largest share of variance is positively correlated with all observables. If you want to call this latent coordinate a "general factor", feel free, but notice this has to exist, mathematically, as soon as you have both all-positive correlations and decide to use a factor model.

I should explain how factor analysis is related to but different from the principal components, but having just been writing about this in my lecture notes, I'll just refer to that (link below).

Questions: In large dimensions, how different do random covariance matrices look from low-rank-plus-noise matrices? For every $q$-factor model with given means and covariance matrices, there is a mixture model with $q+1$ discrete clusters and the same means and covariances; is the distinction between them identifiable?

Notoriously, factor analysis comes out of psychology, and the attempt to "construct" general mental attributes from test scores. But I have said more than enough about that elsewhere.

Constructively: I am very interested in the possibilities of factor models for data reduction in high-dimensional time-series data, especially spatio-temporal data.

Recommended, big picture:
• David J. Bartholomew, Latent Variable Models and Factor Analysis
Recommended, close ups:
• J. Scott Armstrong, "Derivation of theory by means of factor analysis or Tom Swift and his electric factor analysis machine", The American Statistician 21:5 (1967): 17--21 [Reprint]
• Dani Gamerman, Hedibert Freitas Lopes, and Esther Salazar, "Spatial dynamic factor analysis", Bayesian Analysis 3 (2008): 759--792
• Yi-hao Kao and Benjamin Van Roy, "Learning a Factor Model via Regularized PCA", Machine Learning 91 (2013): 279--303, arxiv:1111.6201
• Wim Krijnen, "Positive Loadings and Factor Correlations from Positive Covariances", Psychometrika 69 (2004): 655--660
• John C. Loehlin, Latent Variable Models: An Introduction to Factor, Path, and Structural Analysis
• Robert A. Peterson, "A Meta-Analysis of Variance Accounted for and Factor Loadings in Exploratory Factor Analysis", Marketing Letters 11 (2000): 261--275
• Kristopher J. Preacher and Robert C. MacCallum, "Repairing Tom Swift's Electric Factor Analysis Machine", Understanding Statistics 2 (2003): 13--43 [The objection about nonlinearity seems deeply unfair to me: if one is actually using factor analysis to discover things, when would one be willing to assert the linearity? Similarly, how is one to know which rotation of the factors is better? By definition, the different rotations are all empirically equivalent, so in a genuinely new domain we don't know which ones group related quantities together. Reprint]
• Mohamed Saidane, Xavier Bry and Christian Lavergne, "Generalized Linear Factor Models: A New Local EM Estimation Algorithm", Communications in Statistics: Theory and Methods 42 (2013): 2944--2958
To write:
• CRS, "General Factors in Correlational Psychology: Artifacts and Myths"
• CRS, "Semi-Parametric Generalized-Linear Factor Models" [Or, a brilliant scheme for replacing the linear dependence on the factors with a generalized additive model, if I can just get it to work...]

#### Update history

(Incomplete)
• Major revision 9 November 2017