Ordinary Least Squares (OLS) Estimation

Visualizing the SSE

To further understand the sum of squared error, we can examine a visual representation of the SSE for Model A. Recall that visually, the residual is the vertical distance between an observation and the fitted value (which lie on the fitted line). The residual indicates how different these two quantities are on the Y-metric. In the formula we squared each of the residuals. Visually, this is equivalent to producing the area of a square that has side length equal to the absolute value of the residual.

Figure 5: Scatterplot of the observed toy data and the OLS fitted regression line for Model A. The left-hand plot visually displays the residual values as line segments with negative residuals shown as dashed lines. The right-hand plot visually shows the squared residuals as the area of a square with side length equal to the absolute value of the residual.

The SSE is simply the total area encompassed by all of the squares. Note that the observation that is directly on the line has a residual of 0 and thus does not contribute an quantity to the SSE. If you computed the SSE for a line with different intercept or slope values, the SSE will be different. The plot below shows what this might look like for the flat line produced by $\widehat{Y_i}=50$.

Figure 10: Scatterplot of the observed toy data and the fitted flat line with Y-intercept of 50. The plot visually shows the squared residuals as the area of a square with side length equal to the absolute value of the residual.

Powell & Lehe (2015) created an interactive website to help understand how the SSE is impacted by changing the intercept or slope of a line. You can also see how indicidual observations impact the SSE value.

Mathematical Optimization

Finding the intercept and slope that give the lowest SSE is referred to as an optimization problem in the field of mathematics. Optimization is such an important (and sometimes difficult) probelm that there have been several mathematical and computational optimization methods that have been developed over the years. You can read more about mathematical optimization on Wikipedia if you are interested.

One common mathematical method to find the minimum SSE involves calculus. We would write the SSE as a function of${\beta }_0$ and ${\beta }_1$, compute the partial derivatives (w.r.t. each of the coefficients), set these equal to zero, and solve to find the values of the coefficients. The lm() function actually uses an optimization method called QR decomposition to obtain the regression coefficients. The actual mechanics and computation of these methods are beyond the scope of this course. We will just trust that the lm() function is doing things correctly in this course.

Evaluating the Impact of a Predictor Using SSE

Consider again the general equation for the statistical model that includes a single predictor,

$$Y_i={\beta }_0+{\beta }_1(X_i)+{ϵ}_i$$

One way that statisticians evaluate a predictor is to compare a model that includes that predictor to the same model that does not include that predictor. For example, comparing the following two models allows us to evaluate the impact of $X_i$.

$$\begin{array}{rl}{Y}_i& ={\beta }_0+{\beta }_1(X_i)+{ϵ}_i\\ Y_i& ={\beta }_0+{ϵ}_i\end{array}$$

The second model, without the effect of $X$, is referred to as the intercept-only model. This model implies that the value of $Y$ is not a function of $X$. In our example it suggests that the mean income is not conditional on education level. The fitted equation,

$${\hat{Y}}_i={\hat{\beta }}_0$$

indicates that the predicted $Y$ would be the same (constant) regardless of what $X$ is. In our example, this would be equivalent to saying that the mean income is the same, regardless ofemployee education level.

Fitting the Intercept-Only Model

To fit the intercept-only model, we just omit the prediter term on the right-hand side of the lm() formula.

lm.0 = lm(income ~ 1, data = city)
lm.0

Call:
lm(formula = income ~ 1, data = city)

Coefficients:
(Intercept)  
      53.74  

The fitted regression equation for the intercept-only model can be written as,

$$\widehat{{\mathrm{Income}}_i}=53.742$$

Graphically, the fitted line is a flat line crossing the $y$-axis at 53.742 (see plot below).

Figure 11: Scatterplot of employee incomes versus education levels. The OLS fitted regression line for the intercept-only model is also displayed.

Does the estimate for ${\beta }_0$, 53.742, seem familiar? If not, go back to the exploration of the response variable in the Simple Linear Regression—Description chapter. The estimated intercept in the intercept-only model is the marginal mean value of the response variable. This is not a coincidence.

Remember that the regression model estimates the mean. Here, since the model is not a conditional model (no $X$ predictor) the expected value (mean) is the marginal mean.

$$\begin{array}{rl}{\mu }_Y& ={\beta }_0\end{array}$$

Plotting this we get,

Figure 12: Plot displaying the OLS fitted regression line for the intercept-only model. Histogram showing the marginal distributon of incomes is also shown.

The model itself does not consider any predictors, so on the plot, the $X$ variable is superfluous; we could just collapse it to its margin. This is why the mean of all the $Y$ values is sometimes referred to as the marginal mean.

Yet another way to think about this is that the model is choosing a single income (${\hat{\beta }}_0$) to be the predicted income for all the employees. Which value would be a good choice? Remember the lm() function chooses the “best” value for the parameter estimate based on minimizing the sum of squared errors. The marginal mean is the value that minimizes the squared deviations (errors) across all of the observations, regardless of education level. This is one reason the mean is often used as a summary measure of a set of data.

SSE for the Intercept-Only Model

Since the intercept-only model does not include any predictors, the SSE for this model is a quantification of the total variation in the outcome variable. It can be used as baseline measure of the error variation in the data. Below we compute the SSE for the intercept-only model (if you need to go through the steps one-at-a-time, do so.)

city %>%
  mutate(
    y_hat = 53.742,
    errors = income - y_hat,
    sq_errors = errors ^ 2
  ) %>%
  summarize(
    SSE = sum(sq_errors)
  )

# A tibble: 1 × 1
    SSE
  <dbl>
1 6566.

Proportion Reduction in Error

The SSE for the intercept-only model represents the total amount of variation in the sample incomes. As such we can use it as a baseline for comparing other models that include predictors. For example,

• SSE (Intercept-Only): 6566
• SSE (w/Education Level Predictor): 2418

Once we account for education in the model, we reduce the SSE. Moreover, since the only difference between the intercept-only model and the preictor model was the inclusion of the effect of education level, any difference in the SSE is attributable to including education in the model. Since the SSE is smaller after we include education level in the model it implies that improving the data–model fit (smaller error).

How much did the amount of error improve? The SSE was reduced by 4148 after including education level in the model. Is this a lot? To answer that question, we typically compute and report this reduction as a proportion of the total variation; called the proportion of the reduction in error, or PRE.

$$\mathrm{PRE}=\frac{{\mathrm{SSE}}_{\mathrm{Intercept}\text{-}\mathrm{Only}}-{\mathrm{SSE}}_{\mathrm{Predictor}\text{-}\mathrm{Model}}}{{\mathrm{SSE}}_{\mathrm{Intercept}\text{-}\mathrm{Only}}}$$

For our particular example,

$$\begin{array}{rl}\mathrm{PRE}& =\frac{6566-2418}{6566}\\ & =\frac{4148}{6566}\\ & =0.632\end{array}$$

Including education level as a predictor in the model reduced the error by 63.2%.

Variation Accounted For

It is often convenient to express these values as proportions of the total variation. To do this we can divide each term in the partitioning by the total sum of squares.

$$\frac{{\mathrm{SS}}_{\mathrm{Total}}}{{\mathrm{SS}}_{\mathrm{Total}}}=\frac{{\mathrm{SS}}_{\mathrm{Model}}}{{\mathrm{SS}}_{\mathrm{Total}}}+\frac{{\mathrm{SS}}_{\mathrm{Error}}}{{\mathrm{SS}}_{\mathrm{Total}}}$$

Using the values from our example,

$$\begin{array}{rl}\frac{6566}{6566}& =\frac{4148}{6566}+\frac{2418}{6566}\\ 1& =0.632+0.368\end{array}$$

The first term on the right-hand side of the equation, $\frac{{\mathrm{SS}}_{\mathrm{Model}}}{{\mathrm{SS}}_{\mathrm{Total}}}$, is 0.632. This is the PRE value we computed earlier. Since the ${\mathrm{SS}}_{\mathrm{Model}}$ represents the model-explained variation, many researchers interpret this value as the percentage of variation explained or accounted for by the model. They might say,

The model accounts for 63.2% of the variation in incomes.

Since the only predictor in the model is education level, an alternative interpretation of this value is,

Differences in education level account for 63.2% of the variation in incomes.

Better models explain more variation in the outcome. They also have small errors. Aside from conceptually making some sense, this is also shown in the mathematics of the partitioning of variation.

$$1=\frac{{\mathrm{SS}}_{\mathrm{Model}}}{{\mathrm{SS}}_{\mathrm{Total}}}+\frac{{\mathrm{SS}}_{\mathrm{Error}}}{{\mathrm{SS}}_{\mathrm{Total}}}$$

Since the denominator is the same on both terms, and the sum of the two terms must be one, this implies that the smaller the amount of error, the smaller the last term (proportion of unexplained variation ) must be and the larger the first term (the proportion of explained variation) has to be.

R-Squared

Another way to think about measuring the quality of a model is that ‘good’ models should reproduce the observed outcomes, after all they explain variation in the outcome. How well do the fitted (predicted) values from our model match with the outcome values? To find out, we can compute the correlation between the model fitted values and the observed outcome values. To compute a correlation, we will use the correlate() function from the corrr package.

# Load library
library(corrr)

# Create fitted values and correlate them with the outcome
city %>%
  mutate(
    y_hat = 11.321 + 2.651*education
  ) %>%
  select(y_hat, income) %>%
  correlate()

# A tibble: 2 × 3
  term    y_hat income
  <chr>   <dbl>  <dbl>
1 y_hat  NA      0.795
2 income  0.795 NA    

The correlation between the observed and fitted values is 0.795. This is a high correlation indicating that the model fitted values and the observed values are similar. We denote this value using the uppercase Roman letter $R$.

$$R=0.795$$

Now square this value.

$$R^2={0.795}^2=0.632$$

Again we get the PRE value! All three ways of expressing this metric of model quality are equivalent:

• $\frac{{\mathrm{SSE}}_{\mathrm{Intercept}\text{-}\mathrm{Only}}-{\mathrm{SSE}}_{\mathrm{Predictor}\text{-}\mathrm{Model}}}{{\mathrm{SSE}}_{\mathrm{Intercept}\text{-}\mathrm{Only}}}$
• $\frac{{\mathrm{SS}}_{\mathrm{Model}}}{{\mathrm{SS}}_{\mathrm{Total}}}$
• $R^2$

Although these indices seem to measure different aspects of model quality—reduction in error variation, model explained variation, and alignment of the model fitted and observed values—with OLS fitted linear models, these values are all equal. This will not necessarily be true when we estimate model parameters using a different estimation method (e.g., maximum likelihood). Most of the time this value will be reported in applied research as $R^2$, but as you can see, there are many interpretations of this value under the OLS framework.

Back to Partitioning

Using the fact that $R^2=\frac{{\mathrm{SS}}_{\mathrm{Model}}}{{\mathrm{SS}}_{\mathrm{Total}}}$, we can substitute this into the partitioning equation from earlier.

$$\begin{array}{rl}\frac{{\mathrm{SS}}_{\mathrm{Total}}}{{\mathrm{SS}}_{\mathrm{Total}}}& =\frac{{\mathrm{SS}}_{\mathrm{Model}}}{{\mathrm{SS}}_{\mathrm{Total}}}+\frac{{\mathrm{SS}}_{\mathrm{Error}}}{{\mathrm{SS}}_{\mathrm{Total}}}\\ 1& =R^2+\frac{{\mathrm{SS}}_{\mathrm{Error}}}{{\mathrm{SS}}_{\mathrm{Total}}}\\ 1-R^2& =\frac{{\mathrm{SS}}_{\mathrm{Error}}}{{\mathrm{SS}}_{\mathrm{Total}}}\end{array}$$

This suggests that the last term in the partitioning, $\frac{{\mathrm{SS}}_{\mathrm{Error}}}{{\mathrm{SS}}_{\mathrm{Total}}}$ is simply the difference between 1 and $R^2$. In our example,

• $R^2=0.632$, and
• $1-R^2=0.368$

Remember that one interpretation of $R^2$ is that 63.2% of the variation in incomes was explained by the model. Alternatively, 36.8% of the variation in income is not explained by the model; it is residual variation. If the unexplained variation is too large, it suggests to an applied analyst that she could include additional predictors in the model. We will explore this in future chapters.

For now, recognize that OLS estimation gives us the ‘best’ model in terms of minimizing the sum of squared residuals, which in turn maximizes the explained variation. But, the ‘best’ model may not be a ‘good’ model. One way to measure quality of the model is through the metric of $R^2$. Understanding whether $R^2$ is large or small is based on the domain science. For example in some areas of educational research, an $R^2$ of 0.4 might indicate a really great model, whereas the same $R^2$ of 0.4 in some areas of biological research might be quite small and indicate a poor model.