Likelihood, and Maximum Likelihood, in Statistics(arxiv.org)

We observe a random variable \( X \). (It may be a big, hairy,
high-dimensional beast, with lots of components, but we'll treat it as one
object for now.) We also have a probability model with an adjustable parameter
\( \theta \). (This may also be an enormous infinite-dimensional object.) For
each \( \theta \) we get a distribution for \( X \), say \( p(x;\theta) \equiv
\mathrm{Prob}_{\theta}(X=x) \). That is, the probability model tells us, for
each parameter value, the probability of any particular outcome.
Ordinarily, we tend to look at how \( p(x;\theta) \) changes with \( x \) fixed, for some particular \( \theta \).

What statisticians have come to call the likelihood function is

\[
L(\theta) \equiv p(X; \theta)
\]

This is the probability of the data as a function of the parameter.
That is, it tells us the probability of observing what we did observe, as we
consider varying the parameter.

A natural and compelling approach to parameter estimation is then
the method of maximum likelihood: guess that the true
parameter value is the one which makes the observed data as probable as
possible. This is, as I said, natural and compelling, and it works (is
consistent/probably-approximately-correct)
under a broad range of circumstances, but unfortunately it doesn't
always work.

To see a little bit about why it typically works, but doesn't always, notice
that \( L(\theta) \) is a random function, i.e.,
a stochastic process. (It is a process
"indexed", as we say in the trade, by the parameter space, which may be weird,
but still a process.) The method of maximum likelihood looks for the maximum
of this random function, and hopes that it converges on the true parameter
value. But convergence of
stochastic processes is a somewhat delicate business. In many situations,
the likelihood function does converge to a sensible, deterministic
limiting function which is uniquely maximized at the true parameter value.
(When this happy state of affairs applies, the limiting function has
nice information-theoretic
interpretations.) But there are, alas, times when the convergence just does
not work.

Now, I should at this point admit that the way I've defined likelihood above
only works when \( X \) is discrete. If \( X \) is continuous, then one needs
to work with probability densities rather than mass functions, which I think
makes the rhetoric a bit less persuasive. It also opens the way, to those
who've learned measure-theoretic probability, to a more general definition.

(For each \( \theta \), say \( P_{\theta} \) is a probability measure on \( \mathcal{X} \), and these are all absolutely continuous with respect to some reference measure \( M \) (not necessarily a probability measure). Then we define \( L(\theta) = \frac{dP_{\theta}}{d M}(X) \), using the Radon-Nikodym derivative. This makes the exact likelihood function relative to the choice of reference measure \( M \), but notice that for any other reference measure \( N \), we'd have \( \frac{dP_{\theta}}{d N}(X) = \frac{d P_{\theta}}{dM}(X) \frac{dM}{dN}(X) \), so changing the reference measure doesn't change relative likelihoods, the location of the maximum likelihood estimate, etc.)

I should also admit that the idea that one can simply calculate the
probability of a given outcome from a probability model is often rather
optimistic. This has opened up a range of pseudo-, quasi-, synthetic, and
other likelihoods, which try to retain some of the formal structure, while
ditching the full probability calculations. One of my reasons for breaking out
this notebook is the hope that it will encourage me to wrap my head around
these not-quite-likelihoods. (I think I could define the difference
between a pseudo- and a quasi- likelihood if I had to, but it's embarrassing
for someone in my position not to be sure.)

permanent link for this
note   RSS feed for this
note