Understanding how our estimators behave is crucial for making accurate predictions. In my blog about Linear Regression, we covered the OLS estimator, which characterizes the weight vector of our linear regression model in terms of and :

In many practical situations, we assume that is drawn from a normal distribution. The mean of the distribution is represented by , where denotes the weights or coefficients assigned to our model. The covariance is given by , where represents the variance of the errors and is the identity matrix.

Given that adheres to a specific distribution, it logically follows that also possesses its own distribution. Unraveling the parameters defining this distribution for could provide valuable insights into the fundamental principles at play. To unpack this concept, let’s calculate the mean and variance of .

Evaluating the Mean:

We calculate the expected value of using the formula

utilizing the linearity of expectation, this simplifies to:

and because , we can substitute with :

what this tells us, is that the mean of the MLE (i.e. OLS estimator) of is indeed itself — the very parameter that represents the mean of our response variable in the model . In other words, under the assumption of a normal distribution for , the MLE not only fits our data but also aligns with the expected value of our response variable.

But this insight is only half of a larger picture, and an important question remains: can we realistically expect to obtain a result close to this expected value in practical scenarios? To address this, let’s consider the variance.

Evaluating the Variance:

Again, we understand , and therefore . This can also be expressed in terms of expectation:

Since , we can adjust this formula to:

Rearranging the terms gives an expression for :

This calculation plays a crucial role in understanding the behavior of our model. Specifically, it helps in calculating the variance of our OLS estimator, , which is given by the formula:

From earlier discussion, we know that , simplifying our variance formula to:

Next, let’s substitute our OLS estimator formula into the variance equation:

Applying the linearity of expectation, we have

Now, recall the expression we derived earlier, substituting this into our variance equation gives us:

Expanding this and simplifying, we find

This formula highlights how the variance of depends on the variance of errors in the regression model () and X.

When is nearly singular due to multicollinearity, the eigenvalues become very small. Consequently the eigenvalues of its inverse become very large. Remember, the eigenvalues of the inverse are the reciprocals of the eigenvalues of the original matrix. Since , these large eigenvalues would imply that the variance of our OLS estimator would be inflated. This inflation in variance leads to less reliable predictions, as it indicates a high level of uncertainty or instability in our coefficient estimates. Essentially, the high correlation among our predictors causes our model to be less certain about the true effect of each individual predictor.