Simple Linear Regression—Description

Marginal Distribution of Income

To begin this exploration, we will examine the marginal distribution of employee incomes. We can plot a marginal distribution using functionality from the ggplot2 package.

ggplot(data = city, aes(x = income)) +
  stat_density(geom = "line") +
  theme_bw() +
  xlab("Income (in thousands of dollars)") +
  ylab("Probability density")

Figure 1: Density plot of employee incomes.

This plot suggests that the distribution of employee incomes is unimodal and most of the incomes are between roughly $50,000 and $70,000. The smallest income in the sample is about $25,000 and the largest income is over $80,000. This indicates there is a fair amount of variation in the data.

To further summarize the distribution, it is typical to compute and report summary statistics such as the mean and standard deviation. One way to compute these values is to use functions from the dplyr library.

city %>%
  summarize(
    M = mean(income),
    SD = sd(income)
    )

# A tibble: 1 × 2
      M    SD
  <dbl> <dbl>
1  53.7  14.6

Describing this variable we might write,

The marginal distribution of income is unimodal with a mean of approximately $53,700. There is variation in employees’ salaries (SD = $14,500).

Marginal Distribution of Education Level

We will also examine the marginal distribution of the education level variable.

# Plot
ggplot(data = city, aes(x = education)) +
  stat_density(geom = "line") +
  theme_bw() +
  xlab("Education level") +
  ylab("Probability density")

Figure 2: Density plot of employee education levels.

# Summary statistics
city %>%
  summarize(
    M = mean(education),
    SD = sd(education)
    )

# A tibble: 1 × 2
      M    SD
  <dbl> <dbl>
1    16  4.36

Again, we might write,

The marginal distribution of education is unimodal with a mean of 16 years. There is variation in employees’ level of education (SD = 4.4).

Relationship Between Variables

Although examining the marginal distributions is an important first step, those descriptions do not help us directly answer our research question. To better understand any relationship between income and education level we need to explore how the distribution of income differs as a function of education. To do this, we will create a scatterplot of incomes versus education.

ggplot(data = city, aes(x = education, y = income)) +
  geom_point() +
  theme_bw() +
  xlab("Education (in years)") +
  ylab("Income (in U.S. dollars)")

Figure 3: Scatterplot displaying the relationship between employee education levels and incomes.

The plot suggests a relationship (at least for these employees) between level of education and income. When describing the relationship we want to touch on four characteristics of the relationship:

• Functional form of the relationship
• Direction
• Strength
• Observations that do not fit the trend (outliers)

Systematic Part of the Statistical Model

The systematic (fixed) part of the equation gives the mean value of \(Y\) given a particular \(X\)-value. The mean of \(Y\) given a particular \(X\)-value is referred to as a conditional mean of \(Y\). The notation for the conditional mean of \(Y\) given a particular \(X\) value is \(\mu_{Y \vert X_i}\). We express the systematic part of the linear regression model mathematically as,

\[ \mu_{Y \vert X_i} = \beta_0 + \beta_1(X_i) \]

Note that the systematic part of the equation does not include the error term. The error term is a part of the random component of the model. Statisticians also use \(\hat{Y_i}\) to indicate the conditional mean of \(Y\) at a particular \(X\). Thus, the systematic part of the linear regression model can also be written as,

\[ \hat{Y_i} = \beta_0 + \beta_1(X_i) \]

The terms \(\beta_0\) and \(\beta_1\) in the systematic part of the model are referred to as the regression parameters. One of the primary goals of a regression analysis is to estimate the values of the regression parameters (i.e., the intercept and slope terms). Thus the systematic part of the regression model is a description, in mathematical form, of how the conditional mean \(Y\) is related to \(X\). The equation here indicates that the conditional mean of \(Y\) (\(\hat{Y_i}\)) is a linear function of \(X\). This implies that the conditional mean value of \(Y\) differs by a constant amount for a constant difference in \(X\).

For example, the difference between the mean income for those employees who have 10 years of education and those that have 11 years of education is the same as the difference between the mean income for those employees who have 17 years of education and those that have 18 years of education. Or, the difference between the mean income for those employees who have 4 years of education and those that have 8 years of education is the same as the difference between the mean income for those employees who have 20 years of education and those that have 24 years of education.

Visual Representation of the Regression Model

To help better understand the model, consider the following plot:

Figure 4: Plot displaying conditional distribution of \(Y\) at several \(X\) values. The OLS fitted regression line (dotted) is also shown. The red points show the mean value of \(Y\) for each conditional distribution.

At each value of \(X\) there is a distribution of \(Y\). For example, there would be a distribution of incomes for the employees with an education level of 10 years (in the population). There would be another distribution of incomes for the employees with an education level of 11 years (in the population). And so on. These are the conditional distributions of \(Y\).

Each conditional distribution of \(Y\) has a mean; the conditional mean or \(\hat{Y_i}\). These conditional means are can be connected using a line. This is what it means to be able to express the conditional mean of \(Y\) as a linear function of \(X\), or to say that the relationship between \(X\) and \(Y\) is linear.

Random Part of the Statistical Model

From the visual representation of the model, we can see that there is a distribution of \(Y\)-values at each \(X\)-value; this is represented by the normal distributions in the picture. In our example, there are many employees who have the same education level, but have different incomes. The error term in the statistical model accounts for this variation in \(Y\) for those cases that have the same \(X\)-value. Mathematically we can understand this by re-writing the statistical model, substituting \(\mu_{Y \vert X_i}\) in to the systematic part of the model.

\[ \begin{split} Y_i &= \beta_0 + \beta_1(X_i) + \epsilon_i \\ Y_i &= \mu_{Y \vert X_i} + \epsilon_i \end{split} \]

This equation implies that each observed \(Y\)-value is the sum of the conditional mean value of \(Y\) (which is based on the \(X\)-value) and some residual (or error) term. Re-arranging the terms, we can mathematically express the residual term as,

\[ \epsilon_i = Y_i - \mu_{Y \vert X_i} \]

Or, using the \(\hat{Y_i}\) notation,

\[ \epsilon_i = Y_i - \hat{Y_i} \]

To compute an observation’s residual, we compute the difference between the observation’s \(Y\)-value and its conditional mean value. When the observed value of \(Y\) is larger than the conditional mean value of \(Y\) the residual term will be positive (underprediction). If the observed value of \(Y\) is smaller than the conditional mean value of \(Y\) the residual term will be negative (overprediction).

To further understand the residual term, consider the plot below. This figure shows the relationship between education and income we plotted earlier, and also includes the regression line.

Figure 5: Scatterplot displaying the relationship between employee education levels and incomes. The OLS fitted regression line is also displayed.

Consider the three employees that have an education level of 10 years. The conditional mean income for these three employees is approximately $37,800. This is denoted by the blue point. Remember, the conditional means are on the regression line. The error (residual) term allows for discrepancy between the conditional mean of \(Y\) and the observed \(Y\). In other words, none of these three employees have an actual income of $37,800. The residual represents the difference between the employee’s observed income and the conditional mean income based on their education level.

Graphically, the residual is represented by the vertical distance between the line and a given point on the scatterplot. Some of those points are above the line (they have a positive residual) and some are below the line (they have a negative residual). Also note that for some observations the error term is smaller than for others.

Estimating Residuals

Note that we can also use the estimated regression coefficients to obtain estimates for the residuals, often referred to as the observed residuals. Here we make use of the earlier idea that the residual term was the difference between the observed value of \(Y\) and the conditional mean of \(Y\) for a given \(X\)-value. Mathematically,

\[ \epsilon_i = Y_i - \mu_{Y \vert X_i} \]

Once we use data to obtain estimates for the intercept and slope (\(\hat\beta_0\) and \(\hat\beta_1\)) we can substitute those into the fitted equation and obtain an estimate for the conditional mean (\(\hat{\mu}_{Y \vert X_i}\)). This value can then be used to obtain an estimate for the residual.

\[ \hat\epsilon_i = Y_i - \hat{\mu}_{Y \vert X_i} \]

Or, using the “Y-hat” notation,

\[ \hat\epsilon_i = Y_i - \hat{Y_i} \]

Remember, the hat on the residual indicates it is an estimate based on values obtained from the data!

Intercept Interpretation

The estimate for the intercept was 11.321. Graphically, this value indicates the value where the line passes through the \(y\)-axis (i.e., \(y\)-intercept). As such, it gives the \(\hat{Y}_i\) or predicted value when \(X = 0\). Algebraically we get the same thing if we substitute 0 in for \(X_i\) in the fitted regression equation.

\[ \begin{split} \hat{Y}_i &= \hat{\beta}_0 + \hat{\beta}_1(0) \\ \hat{Y}_i &= \hat{\beta}_0 \end{split} \]

To interpret this value, remember that \(\hat{Y}_i\) is a conditional mean. In this case, it represents the model predicted mean income for all employees that have an education level of 0 years. Graphically this looks like the following.

Figure 6: Plot displaying conditional distribution of \(Y\) at \(X=0\). The OLS fitted regression line (dotted) is also shown. The red points show the mean value of \(Y\) for this conditional distribution—which corresponfds to the intercept value of the regression line.

Interpreting the intercept coefficient from our example,

The model predicted mean income for all employees that have an education level of 0 years is $11,321.

Slope Interpretation

Recall from algebra that the slope of a line describes the change in \(Y\) versus the change in \(X\). In regression, the slope describes the predicted change in \(\hat{Y}\) for a one-unit difference in \(X\). In our example,

\[ \hat{\beta}_1 = \frac{\Delta\hat{Y}}{\Delta X} = \frac{2.651}{1} \]

Again, because \(\hat{Y}\) is a conditional mean, the slope represents the difference in predicted mean incomes for each one-year difference in education level. Graphically,

Figure 7: Plot displaying conditional distribution of \(Y\) at \(X=0\) and \(X=1\). The OLS fitted regression line (dotted) is also shown. The red points show the mean value of \(Y\) for these conditional distributions—the relative change which corresponds to the slope value of the regression line.

Interpreting the slope coefficient in our example,

Each one-year difference in education level is associated with a model predicted difference of $2,651 in mean income.

To better understand this, consider the mean income for city employees with three different education levels. The first set of employees have an education level of 10 years. The second set have an education level of 11 years, and the third set have an education level of 12 years. Now let’s compute each set of employees’ mean income using the estimated regression coefficients.

\[ \begin{split} \mathbf{Education=10:~}\hat{\mathrm{Income}} &= 11.321 + 2.651(10) \\ &= 37.831 \end{split} \]

\[ \begin{split} \mathbf{Education=11:~}\hat{\mathrm{Income}} &= 11.321 + 2.651(11) \\ &= 40.482 \end{split} \]

\[ \begin{split} \mathbf{Education=12:~}\hat{\mathrm{Income}} &= 11.321 + 2.651(12) \\ &= 43.133 \end{split} \]

Each set of employees’ education levels differ by one year (10 to 11 to 12). The difference in predicted mean incomes for these employees differs by 2.651 thousand dollars.

Estimating Residuals

Consider the 25th case in the data frame.

city %>%
  filter(row_number() == 25)

# A tibble: 1 × 5
  education income seniority gender party      
      <dbl>  <dbl>     <dbl> <chr>  <chr>      
1        20   54.7        12 female Independent

This employee (Employee 25) has an education level of 20 years (\(X_{25}=20\)). Her income is 54.672 thousand dollars (\(Y_{25}=54.672\)). Using the fitted regression equation, we can compute the model predicted mean income for employees with 20 years of education as,

# Y_hat = b0 + b1 * X
11.321 + 2.651 * 20

[1] 64.341

\[ \hat{Y}_{25} = 64.341 \]

We could also report this using the conditional mean notation,

\[ \hat\mu_{Y \vert X=20} = 64.341 \]

Now we can use the estimated conditional mean to also compute Employee 25’s residual.

# e = Y - Y_hat
54.672 - 64.341

[1] -9.669

Using mathematical notation,

\[ \hat\epsilon_{25} = -9.669 \]

The negative residual, \(\hat\epsilon_{25} = -9.669\), suggests that this employee earns $9,669 less than the average predicted income for employees with 20 years of education. We can also represent these values graphically.

Figure 8: Plot displaying the OLS fitted regression line (blue) between employee education levels and incomes. The 25th employee’s observed data (black dot) is plotted, and a visual representation of the employee’s residual (red line) is also displayed.