Notebooks

Likelihood, and Maximum Likelihood, in Statistics

Last update: 10 Jul 2026 20:05
First version: 24 June 2026

We observe a random variable \( X \). (It may be a big, hairy, high-dimensional beast, with lots of components, but we'll treat it as one object for now.) We also have a probability model with an adjustable parameter \( \theta \). (This may also be an enormous infinite-dimensional object; since I will need a symbol for the parameter space, make that \( \Theta \).) For each \( \theta \) we get a distribution for \( X \), say \( p(x;\theta) \equiv \mathrm{Prob}_{\theta}(X=x) \). That is, the probability model tells us, for each parameter value, the probability of any particular outcome. Ordinarily, we tend to look at how \( p(x;\theta) \) changes with \( x \) fixed, for some particular \( \theta \).

What statisticians have come to call the likelihood function is \[ L(\theta) \equiv p(X; \theta) \] This is the probability of the data as a function of the parameter. That is, it tells us the probability of observing what we did observe, as we consider varying the parameter.

A natural and compelling approach to parameter estimation is then the method of maximum likelihood: guess that the true parameter value is the one which makes the observed data as probable as possible. This is, as I said, natural and compelling, and it works (is consistent/probably-approximately-correct) under a broad range of circumstances, but unfortunately it doesn't always work.

To see a little bit about why it typically works, but doesn't always, notice that \( L(\theta) \) is a random function, i.e., a stochastic process. (It is a process "indexed", as we say in the trade, by the parameter space \( \Theta \), which may be weird, but still a stochastic process.) The method of maximum likelihood looks for the maximum of this random function, and hopes that it converges on the true parameter value. But convergence of stochastic processes is a somewhat delicate business. In many situations, the likelihood function does converge to a sensible, deterministic limiting function which is uniquely maximized at the true parameter value. (When this happy state of affairs applies, the limiting function has nice information-theoretic interpretations.) But there are, alas, times when the convergence just does not work.

The method of maximum likelihood delivers "point" estimates, a single guess at the value of \( \theta \). The is one of the classic statistical tasks. Another is "hypothesis testing": does \( X \) come from one distribution, or family of distributions, or from some other, specified, distribution(s)? (These two alternatives are supposed, ideally, to correspond to two different scientific hypotheses.) In the case of testing one fixed distribution \( p \) against another fixed distribution \( q \), it turns out that the optimal test statistic, in a very strong sense, is the likelihood ratio \( q(X)/p(X) \). (You might ask "in what sense?" and "why the ratio and not \( q(X) - p(X) \), or \( q^2(X)-p^2(X) \) for that matter?", and I'd have an explanation for you but I wrote it out elsewhere long ago.) This suggests that if one wants to test whether \( \theta \) takes some pre-specified value, \( \theta=\theta_0 \) against some broader alternative, \( \theta \neq \theta_0 \), one might do well to also use a ratio: \[ \frac{\max_{\theta \in \Theta}{L(\theta)}}{L(\theta_0)} \] This is most scientifically useful when there is, in fact, some interesting theory which predicts a specific value for the parameter. (Whether that is when such a test is most often used is another story.) This can be generalized: one tests whether \( \theta \in T \subset \Theta \) vs. \( \theta \in \Theta \setminus T \) by the ratio \[ \frac{\max_{\theta \in \Theta}{L(\theta)}}{\max_{\theta \in T}{L(\theta)}} \]

Now, as I said, for the "simple vs. simple", \( p \) or \( q \) situation, likelihood ratio tests are optimal in very strong senses. This is no longer generally true for these more complicated situations, but one can actually show that no test can be optimal in the same way. Likelihood ratio tests are generally consistent (their error probabilities go to zero as the sample becomes more informative) and often efficient (they go to zero about as fast as possible). Moreover, there are useful asymptotic tricks for calculating the distribution of the likelihood ratio under a wide range of circumstances.

Now, I should at this point admit that the way I've defined likelihood above only works when \( X \) is discrete. If \( X \) is continuous, then one needs to work with probability densities rather than mass functions, which I think makes the rhetoric a bit less persuasive. It also opens the way, to those who've learned measure-theoretic probability, to a more general definition.

(For each \( \theta \), say \( P_{\theta} \) is a probability measure on \( \mathcal{X} \), and these are all absolutely continuous with respect to some reference measure \( M \) (not necessarily a probability measure). Then we define \( L(\theta) = \frac{dP_{\theta}}{d M}(X) \), using the Radon-Nikodym derivative. This makes the exact likelihood function relative to the choice of reference measure \( M \), but notice that for any other reference measure \( N \), we'd have \( \frac{dP_{\theta}}{d N}(X) = \frac{d P_{\theta}}{dM}(X) \frac{dM}{dN}(X) \), so changing the reference measure doesn't change relative likelihoods, likelihood ratios, the location of the maximum likelihood estimate, etc.)

I should also admit that the idea that one can simply calculate the probability of a given outcome from a probability model is often rather optimistic. This has opened up a range of pseudo-, quasi-, synthetic, and other likelihoods, which try to retain some of the formal structure, while ditching the full probability calculations. One of my reasons for breaking out this notebook is the hope that it will encourage me to wrap my head around these not-quite-likelihoods. (I think I could define the difference between a pseudo- and a quasi- likelihood if I had to, but it's embarrassing for someone in my position not to be sure.)

[TODO: discussion of the "likelihood principle", and what Birnbaum did or did not prove]

permanent link for this note   RSS feed for this note

Notebooks:   Powered by Blosxom