Advanced – Maximum Likelihood Estimation

Posted on by clbmath

In statistical inference, one often encounters a dataset X = \{x_1, x_2, \ldots, x_k\} \subset \mathbb{R}^n and seek to characterize it by estimating the parameters \theta of a chosen probability distribution p(X | \theta). A prevalent technique for achieving this is Maximum Likelihood Estimation (MLE). At its core, MLE is the method of finding the parameter values \theta that makes the observed data most probable under a specified statistical model. To do this, we construct a likelihood function for the dataset by assuming each data point x_i contributes independently to the total likelihood. Thus, the overall likelihood is the product of individual probabilities, which we aim to maximize with respect to \theta:

\mathcal{L}(\theta; X) = p(X | \theta) = \prod_{i=1}^k p(x_i | \theta)

The goal of MLE is to find the parameter values, \theta_{ML}, that maximize this likelihood function:

\theta_{ML} = \text{argmax}_{\theta} \mathcal{L}(\theta; X)

However directly maximizing the likelihood can be computationally challenging, especially with a large number of observations. Instead, it is often more practical to maximize the logarithm of the likelihood, known as the log-likelihood.

\theta_{ML} = \text{argmax}_{\theta} \mathcal{L}(\theta; X) = \text{argmax}_{\theta} \ln \mathcal{L}(\theta; X) = \text{argmax}_{\theta} \sum_{i=1}^{k} \ln p(x_i | \theta)

This transformation is helpful due to the log function’s property of being monotonically increasing. This means that the parameters that maximize the likelihood also maximize the log-likelihood, allowing us to work with sums instead of products, which is mathematically and computationally more convenient.

Once we have the log-likelihood function, we can find the maximum likelihood estimates of the parameters by taking the derivative of the log-likelihood with respect to the parameters, setting these derivatives equal to zero, and solving for the parameters. Lets illustrate this concept with an example. Imagine we’re dealing with a multivariate Gaussian distribution over X. Our objective here is to estimate the mean \mu_{ML} and covariance \Sigma_{ML} that most effectively represent the characteristics of the data.

Estimating the Mean

Having laid the foundational concepts, and again assuming x_i \sim N(\mu, \Sigma) for all i = 1, 2, ..., k, we can proceed first with a best estimate of the mean, \mu_{ML}:

\mu_{ML} = \text{argmax}_{\mu} \sum_{i=1}^{k} \{\ln \frac{1}{\sqrt{ (2 \pi )^{n} |\Sigma| }} e^{- \frac{1}{2} (x_i - \mu)^{T} \Sigma^{-1} (x_i - \mu)} \}

= \text{argmax}_{\mu} k \ln \frac{1}{\sqrt{ (2 \pi )^{n} |\Sigma| }} - \frac{1}{2} \sum_{i=1}^{k} (x_i - \mu)^{T} \Sigma^{-1} (x_i - \mu),

Considering that the first term, k \ln \frac{1}{\sqrt{ (2 \pi )^{n} |\Sigma| }}, is independent of \mu, we may disregard it for our present purposes, thus simplifying our expression to

= \text{argmax}_{\mu} - \frac{1}{2} \sum_{i=1}^{k} (x_i - \mu)^{T} \Sigma^{-1} (x_i - \mu)

and, noting the presence of the negative factor -1/2, we find it expedient to replace finding the argument of the maximum with that of the minimum, thereby simplifying our endeavor.

= \text{argmin}_{\mu} \sum_{i=1}^{k} (x_i - \mu)^{T} \Sigma^{-1} (x_i - \mu)

The optimal solution for this expression can be unveiled by equating its gradient to naught. It is at this value, \mu_{ML}, where the function’s gradient vanishes that we find its minimum argument.

0 = \nabla_{\mu} \{\sum_{i=1}^{k} (x_i - \mu)^{T} \Sigma^{-1} (x_i - \mu)\}

= \sum_{i=1}^{k} \nabla_{\mu} (x_i^{T} \Sigma^{-1} x_i - 2 \mu^{T} \Sigma ^{-1} x_i + \mu^{T} \Sigma ^{-1} \mu)

= -2 \sum_{i=1}^{k} (\Sigma^{-1}x_i - \Sigma^{-1} \mu) = -2 \Sigma^{-1} \sum_{i=1}^{k} (x_i - \mu)

Since \Sigma is positive-definite, we now have

0 = \sum_{i=1}^{k} (x_i - \mu) = (\sum_{i=1}^{k} x_i) - k \mu

or

\mu_{ML} = \sum_{i=1}^{k} x_i /k = \mathbb{E}[X]

The observant reader will, no doubt, recognize this formula as none other than the arithmetic mean of X. It is quite natural, then, to question the necessity of this extensive exposition for a result so well-trodden within the annals of statistics. To this, I offer two-fold reasoning. Firstly, the MLE serves to illuminate that the arithmetic mean does not merely emerge by happenstance; rather, it is the very culmination of an optimization process, aiming to maximize the likelihood across our dataset. Furthermore, the framework we’ve now established paves the way for our ensuing explorations. For example, when dealing with Maximum A Posteriori methods, we’ll find ourselves substituting our familiar likelihood function with the intricate products of posterior distributions.

Estimating Covariance

Applying the same reasoning we did for finding \mu_{ML}, we can estimate \Sigma_{ML}:

\Sigma_{ML} = \text{argmax}_{\Sigma}\sum_{i=1}^{k} \ln p(x_{i} | \mu_{ML}, \Sigma)

=\text{argmax}_{\Sigma} \{\sum_{i=1}^{k} \{\ln \frac{1}{\sqrt{ (2 \pi )^{n} |\Sigma| }} e^{- \frac{1}{2} (x_i - \mu_{ML})^{T} \Sigma^{-1} (x_i - \mu_{ML})} \}\}

= \text{argmax}_{\Sigma} \{ k \ln \frac{1}{\sqrt{ (2 \pi )^{n} |\Sigma| }} - \frac{1}{2} \sum_{i=1}^{k} (x_i - \mu_{ML})^{T} \Sigma^{-1} (x_i - \mu_{ML})\}.

Lets observe the second term here. A nice property of trace, is that for any scalar q \in \mathbb{R}, we have q = \text{trace}(q) . We can expand on this, by saying for any two vectors x, y \in \mathbb{R}^n, and for matrix M \in \mathbb{R}^{n \times n}, it is clear

x^{T}My = \text{trace}(x^{T}My) = \text{trace}(Myx^{T}).

Thus, we can rewrite our equation in the following manner:

= \text{argmax}_{\Sigma} \{ k \ln \frac{1}{\sqrt{ (2 \pi )^{n} |\Sigma| }} - \frac{1}{2} \sum_{i=1}^{k} \text{trace} (\Sigma^{-1} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T}) \}.

and since the trace is linear, we can move our summand in the second term,

= \text{argmax}_{\Sigma} \{ k \ln \frac{1}{\sqrt{ (2 \pi )^{n} |\Sigma| }} - \frac{1}{2} \text{trace} (\Sigma^{-1} \sum_{i=1}^{k} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T}) \}.

Now we’ll expand on the first term. Observe there is a fraction inside of the logarithm, that can be simplified:

k \ln \frac{1}{\sqrt{(2 \pi)^{n} | \Sigma|}} = - \frac{kn}{2} \ln 2 \pi - \frac{k}{2} \ln | \Sigma |

therefore by substitution, it is clear that

\Sigma_{ML} = \text{argmax}_{\Sigma} \{ - \frac{kn}{2} \ln 2 \pi - \frac{k}{2} \ln | \Sigma | - \frac{1}{2} \text{trace} (\Sigma^{-1} \sum_{i=1}^{k} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T}) \}.

Again, the first term is not a function of \Sigma, and can therefore be eliminated

\Sigma_{ML} = \text{argmax}_{\Sigma} \{ - \frac{k}{2} \ln | \Sigma | - \frac{1}{2} \text{trace} (\Sigma^{-1} \sum_{i=1}^{k} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T}) \}

= \text{argmax}_{\Sigma} \{ - \frac{1}{2} \{ k \ln | \Sigma | + \text{trace} (\Sigma^{-1} \sum_{i=1}^{k} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T}) \}\}

= \text{argmin}_{\Sigma} \{ k \ln | \Sigma | + \text{trace} (\Sigma^{-1} \sum_{i=1}^{k} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T}) \}.

Finally, we have simplified it enough to solve for \Sigma:

0 = \nabla_{\Sigma} \{ k \ln | \Sigma | + \text{trace} (\Sigma^{-1} \sum_{i=1}^{k} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T}) \}

= k \nabla_{\Sigma} \ln | \Sigma | + \nabla_{\Sigma} \text{trace} (\Sigma^{-1} \sum_{i=1}^{k} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T})

= k \Sigma^{-1} - \Sigma^{-2} \sum_{i=1}^{k} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T},

which implies

k \Sigma^{-1} = \Sigma^{-2} \sum_{i=1}^{k} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T},

or

\Sigma_{ML} = \frac{\sum_{i=1}^{k} (x_i - \mu_{ML})(x_i - \mu_{ML})^{T}}{k} = \mathbb{E}[(X - \mu_{ML})(X - \mu_{ML})^{T}]

One can’t help but discern that just as \mu_{ML} corresponds to the arithmetic mean of X, the matrix \Sigma_{ML} is the conventional formula representing the variance of X. This is important, particularly when navigating situations devoid of any prior assumptions regarding \mu and \Sigma. As we progress to MAP estimation, it will become evident that such situations are not always useful… but where might maximum likelihood be useful?

There are many potential realms of applicability for the MLE technique. While the standard pedagogical trajectory often couples MLE to linear regression, the scope of MLE stretches well beyond, encompassing many analytical techniques and applications.

Allow me to put forth a thought experiment (one both intriguing and yet grounded in historical trajectory). Picture, if you will, the discernment patterns in the terminal phases of Intercontinental Ballistic Missile (ICBM) launches, tracing the arc of history back to its advent in 1957. Such a vast repository of data, with each entry denoted by altitude-latitude-longitude coordinates, naturally adopts the guise of spherical coordinates, symbolically represented as (r, \theta, \phi).

For the purposes of a more tractable analysis, a transition to Cartesian coordinates, represented as (x, y, z), would prove invaluable. Here’s where the underlying mathematical beauty unveils itself: one could postulate that a sample of the data adheres to a Von Mises-Fisher distribution (or otherwise known as “spherical normal”), succinctly denoted as v(\mu, k). Within this distribution, \mu embodies the mean direction while k serves as a testament to the concentration parameter.

The challenge, thus, metamorphoses into leveraging the formidable prowess of MLE to discern the optimal parameters, \mu and k, that best encapsulate the historical distribution of these launches. Not only would this endeavor provide a mathematical framework to understand past trajectories, but it also lays the groundwork for predictive insights into future launch patterns.

In synthesizing such abstract mathematical principles with tangible, historical datasets, one witnesses the confluence of theory and application. As our journey through this realm progresses, we shall uncover even more intricate tapestries woven from the threads of mathematical abstraction and real-world applicability.

Share Your Feedback

Leave a Reply

Your email address will not be published. Required fields are marked *