In the preceding sections, we covered the intricacies of Linear Regression, explored the concept of Maximum Likelihood Estimation (MLE), and further dissected the statistical properties of the OLS estimator. Having laid some groundwork with earlier topics, the next step involves a thorough examination of MAP estimation. Both MLE and MAP are referred to as point estimation methods, utilizing sample data to compute a singular, specific value as an estimate. However MAP estimation introduces a Bayesian perspective to parameter estimation, where prior knowledge about the parameters is explicitly considered. This approach differs from MLE’s frequentist perspective, which depends only on observed data. By incorporating prior probabilities, MAP estimation offers a more holistic view, effectively merging our pre-existing beliefs with insights gained from the data. We will unravel the theoretical underpinnings of MAP, and delineate its advantages particularly in comparison to the methodologies previously discussed.

The core idea of MLE was beautifully simple: we wanted to find the best parameters (denoted as ) for our model to make our observed data as likely as possible. Mathematically, this was represented as:

where represented the likelihood of given . However this approach has its limitations. Notably, MLE doesn’t factor in any assumptions about the underlying distribution of the parameters, whereas MAP does. Mathematically, we express MAP estimation as follows:

In this model, the prior is integrated to serve as a form of regularization, influencing the optimization towards more plausible parameter values. Additionally, the marginal likelihood does not affect the parameter optimization process. As a result, it can be excluded from the equation for simplicity and efficiency in computation.

Various methods exist to approximate the parameters that maximize in MLE. Analogous to the common practice of using the logarithm trick for simplification in optimization, applying the logarithm to the posterior distribution often proves to be mathematically convenient when solving analytically. This technique can streamline the calculation process, making complex optimization tasks more manageable:

Later, we’ll explore the application of this concept in Ridge Regression. However, determining isn’t limited to just one method. For cases where the distributions don’t lend themselves to analytical solutions, yet require identifying a global maximum, optimization algorithms come into play. A prime example is simulated annealing, an approach inspired by the metallurgical process of annealing. In the physical world, annealing involves heating and then slowly cooling a material to alter its physical properties. In the world of optimization, this concept is used metaphorically to solve complex problems that are otherwise resistant to straightforward solutions.