Notebooks

Predictive Distributions in Statistics

Last update: 31 Jul 2026 16:00
First version: 25 June 2026

Yet Another Inadequate Placeholder

We see some random variable \( X \) and we want to predict another random variable \( Y \). (Pretend in what follows that everything is continuous with well-behaved probability densities, just to simplify notation.) There's some joint distribution \( p(X,Y) \), so a natural way to formulate our prediction is as a conditional distribution, \( p(Y=y|X=x) = p(x,y)/p(x) = \frac{p(x,y)}{\int{p(x,y) dy}} \). This would be the optimal prediction, for many senses of "optimal". Alas, being mere mortal statisticians and not mathematical demi-gods, we do not know the joint distribution, but have to infer it from data. How can we nonetheless give a predictive distribution?

This is one of the places where honesty compels me to admit that a Bayesian approach is indeed vastly more straightforward than a frequentist one. The Bayesian agent will observe a (possibly high-dimensional) \( X \), and has a family of distributions for it, \( m(X|\theta) \), parametrized by a (possibly infinite-dimensional) \( \theta \), and an prior distribution \( \pi_0(\theta) \). (, let's assume that \( m \) and \( \pi \) are probability densities; this is not one of the places where it's worth doing the measure-theoretic version.) This implies that the Bayesian agent predicts the following distribution for \( X \): \[ m(X =x ) = \int{m(X =x|\theta) \pi_0(\theta) d\theta} \] After observing \( X = x_0\), it updates to a posterior distribution, \[ \pi_1(\theta_0) = \frac{m(X=x_0|\theta=\theta_0) \pi_0(\theta_0)}{m(x_0)} \] If \( m \) actually extends to a joint distribution of \( X \) and some not-yet-observed \( Y \), the new predictive distribution of \( Y \) is \[ m(Y=y|X=x_0) = \int{m(Y=y|X=x_0, \theta) \pi_1(\theta) d\theta} \] (This might simplify, if \( Y \) is independent of \( X \) given \( \theta \), but it doesn't have to.) If one believes in origin-myth for \( \pi_0 \) that it represents the agent's initial uncertainty about \( \theta \), then this very neatly captures the change in that uncertainty from observation, and propagates that uncertainty, plus whatever the model distribution is doing, into a distribution for \( Y \).

Again, if there's some true conditional distribution \( p(Y|X) \), that's what we'd like to use for prediction. If the true distribution \( p(Y|X) = m(Y|X, \theta^*) \) for some magic \( \theta^* \), and Bayesian updating happens to be consistent, then \( m(Y|X) \), the Bayesian predictive distribution obtained by integrating over the posterior, will converge on \( p(Y|X) \), the true predictive distribution. (The Bayesian agent is always convinced that it's consistent, but it can be wrong.)

Unfortunately for me, I find that origin-myth incredible, so I am stuck being interested in much more complicated approaches. A simple thing to do would be to come up with some sort of point-estimator \( \hat{\theta}(X) \), and then plug in, using \[ m(Y=y|X=x_0, \hat{\theta}(x_0)) \] as the predictive distribution. This isn't even that unreasonable: if \( \hat{\theta}(X) \) approaches the true \( \theta^* \) (i.e., if the estimator is consistent), and the distributions \( m(Y|X,\theta) \) are continuous in \( \theta \), this will converge on the true predictive distribution \( p(Y|X) = m(Y|X, \theta^*) \). But in the meanwhile, it neglects the fact that the \( X \) we observed is random, it didn't have to be that particular \( x_0 \), so it seems sub-optimal, there's some extra randomness the plug-in distribution does not account for.

I can think of (at least) three responses to this:

  1. Stop worrying and learn to live with plug-ins: What one can (sometimes) show about the Bayesian predictive distribution is that it, too, converges on the true predictive distribution. Whether the extra, pre-asymptotic spread of the former is getting anything right is not at all obvious. It seems more realistic to think of the Bayesian predictive distribution as doing some sort of bias-variance trade-off, away from variance and toward bias. But we might well prefer the opposite trade-off.
  2. Control the badness: We know we're not getting the predictive distribution right, we might try to find some compromise over the set of distributions we consider reasonable, i.e., those of our probability model. One particular way to formulate this is to ask, retrospectively, how much we might have preferred to use a different distribution in our model. That is, we might want to control the "regret" of our predictions. Remarkably enough this can be a tractable optimization problem.
  3. Project confidence: A confidence set for a parameter is a random set which contains the true parameter with a specified probability, say \( \alpha \). More exactly, either the true parameter is in that set, or we were really unlucky (an event whose probability is \( \leq 1 -\alpha \), no matter what the true parameter is, has occurred). Increasing the confidence level \( \alpha \) gives bigger sets; lowering \( \alpha \) gives smaller sets. One can use this to construct a distribution, the "confidence distribution", for the parameter. Now this same logic can sometimes also be applied to another random variable \( Y \): there are ways to build a prediction set which contains \( Y \) with probability \( \alpha \). (Conformal prediction, for example.) Varying \( \alpha \) gives a nested family of such sets, which in turn gives a distribution.

What makes this an inadequate placeholder is that I have not really explained how to do either (2) or (3).

--- Yet another way to tackle this would be to toss away parametric probability models altogether. Non-parametric conditional density estimation is a thing, so one could apply any number of techniques to try to directly estimate the conditional probability distribution \( p(Y|X) \) from enough \( (X, Y ) \) pairs. There are actually quite a few techniques which will converge on the true conditional distribution (if it's not too pathological) as the data grows, without needing to assume any particular mathematical form. You could definitely just learn the conditional probabilities non-parametrically, and use it to make distributional predictions. The issue, however, is that which conditional pdf you learn is still random, because it's a (complicated) function of the random data, and the plug-in approach still doesn't account for that. I know of even less about non-parametric predictive distributions, but that might just be my ignorance.

permanent link for this note   RSS feed for this note

Notebooks:   Powered by Blosxom