1.1 A Silly Example: Predicting Visits to the Rest Room

Let’s use a silly example - because it is memorable.

Say we are interested in knowing how many time a patron of a bar or pub or night club visits the rest room in an evening.

And let’s also suppose that we have some ethical way to measure this and collect the data on 100 people who are in this busy bar on a saturday night.

In this case, our dependent variable is ‘visits to the rest room’, and it might vary between 0 and 10 for our 100 people.

And the question we ask ourselves is - what cause or predicts the number of visits?

Well we might think that there are two important variables

1. the number of hours the person has been in the bar; and
2. the number of drinks the person has had at the bar.

To express this as a line we could say that, on average:

\(\begin{aligned} \text{ rest_room_visits } = &b_1 \text{ hours_at_bar } + b_2 \text{ number_of_drinks } + c\\ \\ \text{Where:} \\ b_1 = &\text{ slope of relationship between hours_at_bar and rest_room_visits } \\ b_2 = &\text{ slope of relationship between number_of_drinks and rest_room_visits} \\ c = &\text{ the constant or y-intercept, which is the number of rest_room_visits (on average) } \\ &\text{ for someone with zero for the two other variables, i.e. they just arrived at bar } \\ &\text{ (hours_at_bar = 0) and has not had a drink yet (number_of_drinks = 0) } \\ \end{aligned}\)

Now if we want to try to work out what B1, B2, and C are we would first need to get a whole lot of data. Well we are lucky, because we have collected data from 100 people on a Saturday night at a local Irish Pub.

The data looks like this, with four columns

… AND SO ON FOR 100 people …

We can ignore the first column (patron) which is just an identifing column.

Now we could, if we wanted to, plot each person on a chart, where the y axis is visits, and the x axis is one of the predictor/independent variables (e.g. hours).

If we did this we would have a very messy graph but there would be a general tendency for visits to go up as the number of hours at the pub went up, and the number of drinks goes up.

In a linear regression, the statistical software calculates the ‘line of best fit’ - meaning the line that is as close as possible to all persons - all the dots on our graph.

This line of best fit can then be described with the three coefficients discussed earlier: \(b_1\), \(b_2\), and \(c\).

B1 and B2 are ‘slopes’ - which you would have learnt about in year 8 mathematics. Remember the equation \(y = bx + c\) Well \(b\) is the slope, and \(c\) is the intercept, and it is the same in linear regression models.

Exactly how the statistical packages calcuate the line of best fit is not our concern, and it involves some complex maths. What matters to us as social scientists is that - so long as basic assumptions about the nature of the data are true, and the statistical package is sound - these models can estimate the relationship between various predictors (independent variables) - and an outcome (dependent variable) we care about.

So let’s say we did analyse our data, and R told us the values for the coefficients were:

• \(b_1 = 0.5\)
• \(b_2 = 0.8\)
• \(c = 0.2\)

What does this mean?

It means that - on average:

• for each one hour a person spends at the bar the will visit the rest room 0.5 times;
• for each drink they have, they will visit the rest room 0.8 times; and
• for someone who stays for zero hours and has no drinks, they will visit the rest room 0.2 times.

2. Linear Regression Example: Political Knowledge

Let’s run a model in R ourselves now, and see how it is done with a real dataset - the dataset from McAllister 2016.

Let’s start with a very simple model of political knowledge, looking at the impact of being 18-24 years of age (or not) on political knowledge.

Now remember, our political knowledge scale goes from 0 to 10, and each one point represents one quiz question the person go right (out of a total of 10 questions) The independent variable we are using to use is a binary variable (also called a dummy variable) which is equal to 1 if the person in aged 18-24 years of age or zero otherwise.

To run a linear regression model, we need to run two lines of code. The first run’s the model, with the command ‘lm()’. Inside the brackets we put the dependent variable and then a tilde (~) and then the independent variable/s (if we have multiple independent variables, we separate them with plus (+) sign). We put the name of the dataset at the end of the set of arguments.

NOTE: In this case, we are using piping with %>%. The meaning of this piping is explained after the hash symbols on each line

We then send the output of this line to a summary command to give us a very useful summary of the model.

When you are ready, run the following command:

elect_2013 %>%          # Take this data frame and PUT it...
    stats::lm(pol_knowledge ~ age_18_24, data = .) %>%     # Into this model. Data frame will appear where the full stop is (.)
    base::summary()         # and then we put the output of lm() into summary()

## 
## Call:
## stats::lm(formula = pol_knowledge ~ age_18_24, data = .)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -4.855 -2.541  0.145  2.145  6.459 
## 
## Coefficients:
##             Estimate Std. Error t value       Pr(>|t|)    
## (Intercept)   4.8545     0.0479  101.31        < 2e-16 ***
## age_18_24    -1.3140     0.1996   -6.58 0.000000000052 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.89 on 3849 degrees of freedom
##   (104 observations deleted due to missingness)
## Multiple R-squared:  0.0111, Adjusted R-squared:  0.0109 
## F-statistic: 43.3 on 1 and 3849 DF,  p-value: 0.000000000052

Now this code might look confusing, but it will become easy - with some practice reading it.

Remember the rule “First look at the p-value”.

However, for models (e.g. regression models like this), we aren’t that interested in the p-value for the model overall. What you should look at is the p-value for the coefficients. and for some reason in R these p-values are listed under the heading “Pr(>|t|)”. Why? I don’t know.

Anyway, so we look at the two numbers under “Pr” and they are both very small numbers (with 11 and 16 zeros after the decimal point), which means that p < 0.05, and so they are significant and we can interpret the coefficients.

What do the coefficients say? Well lets start with the first that is just called “(Intercept)”. This is the C which we discussed in the earlier section, as in the value of y, when all the x’s are equal to zero.

In this case, the intercept has a value of 4.85, which means that people who have a zero for the variable 18-24 year old (i.e. everyone else) have a mean score on the political knowledge quiz of 4.85.

And now let’s look at the second coefficient - the first variable - which is -1.31. So this means persons who are 18-24 years of age, get, on average, 1.31 less questions correct in the quiz than persons who are older.

So, in essence, this model says that 25+ year olds get on average 4.85 quiz questions right, and 18-24 year olds get 3.54 question right.

3. R-squared and Adjusted R-squared

What else can this model tell us? Well towards the bottom of the model you can see that it gives a value for the R-squared and the Adjusted R-squared. We were introduced to the notion of an R-squared in the last set of notes (Week 2, Part 2 - correlation), because R-squared is literally the square of Pearson’s R. We learnt in that that R-squared is an measure of effect size, and that it can it represents the proportion (percentage) of variation in the dependent variable that MIGHT be able to be explained by the independent variable/s.

We also said that roughly speaking we can look at R-squared and say that: small effect size is 0.01-0.04 (i.e 1%-4%) medium effect size is 0.09-0.16 (i.e.9%-16%) large effect size is 0.25+ (ie. 25%+)

So given this, what does the output for our regression model say? It says that the multiple R-squared is 0.01. So basically the difference between 18-24 year olds and the rest of the population explains about 1% of the variation in political knowledge - at best!

elect_2013 %>%
    stats::lm(pol_knowledge ~ age_18_24 
  + internet_skills
  + female
  + tertiary_ed
  + income_quintiles
  + aust_born
  + interest_pol, data = .) %>%
    base::summary()

## 
## Call:
## stats::lm(formula = pol_knowledge ~ age_18_24 + internet_skills + 
##     female + tertiary_ed + income_quintiles + aust_born + interest_pol, 
##     data = .)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -7.052 -1.819  0.014  1.749  8.047 
## 
## Coefficients:
##                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)        7.2572     0.1675   43.34  < 2e-16 ***
## age_18_24         -0.7244     0.1876   -3.86  0.00011 ***
## internet_skills    0.1172     0.0358    3.27  0.00107 ** 
## female            -0.7485     0.0835   -8.97  < 2e-16 ***
## tertiary_ed        0.8749     0.0988    8.85  < 2e-16 ***
## income_quintiles   0.1304     0.0334    3.90 0.000098 ***
## aust_born          0.0431     0.0952    0.45  0.65064    
## interest_pol      -1.5622     0.0537  -29.11  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.42 on 3423 degrees of freedom
##   (524 observations deleted due to missingness)
## Multiple R-squared:  0.293,  Adjusted R-squared:  0.291 
## F-statistic:  202 on 7 and 3423 DF,  p-value: <2e-16

elect_2013 %>%
  stats::lm(pol_knowledge ~ age_18_24 
     + internet_skills
     + female
     + tertiary_ed
     + income_quintiles
     + aust_born, data = .) %>%
  base::summary()

## 
## Call:
## stats::lm(formula = pol_knowledge ~ age_18_24 + internet_skills + 
##     female + tertiary_ed + income_quintiles + aust_born, data = .)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -6.898 -2.134  0.091  2.145  6.669 
## 
## Coefficients:
##                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)        4.2579     0.1472   28.94  < 2e-16 ***
## age_18_24         -1.5296     0.2071   -7.39  1.9e-13 ***
## internet_skills    0.1680     0.0399    4.22  2.6e-05 ***
## female            -1.0899     0.0921  -11.83  < 2e-16 ***
## tertiary_ed        1.1527     0.1097   10.51  < 2e-16 ***
## income_quintiles   0.1630     0.0372    4.38  1.2e-05 ***
## aust_born          0.1016     0.1059    0.96     0.34    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.7 on 3435 degrees of freedom
##   (513 observations deleted due to missingness)
## Multiple R-squared:  0.118,  Adjusted R-squared:  0.116 
## F-statistic: 76.5 on 6 and 3435 DF,  p-value: <2e-16

4. Standardised Coefficients - Beta

The other method that is commonly used to measure effect size and to compare effect size across independent variables in a regression model is ‘standardised regression coefficients’.

At some level these are quite simple to understand. To estimate standardised coefficients, all of the variables are standardised (also called normalised) so that their mean is zero and their standard deviation is 1. And then they are put in the regression model.

The effect of this is to make the coefficients all measured in the same units: Each coefficient is a number that represents “The number of standard deviations change in y (the dependent variables) are predicted/caused by a one standard deviation change in each X (each independent variable).”

Let’s estimate the standardised regression coefficients - which are also called ‘standardized betas’ or just ‘betas’ - for model 2.

To do this we need to install and then load the package “lm.beta”.

if(!require(lm.beta)) {install.packages("lm.beta", repos='https://cran.csiro.au/', dependencies=TRUE)}
base::library(lm.beta)

and then the command is simple: we just run lm.beta() before we run summary (and after we have run lm())

elect_2013 %>%
  stats::lm(pol_knowledge ~ age_18_24 
     + internet_skills
     + female
     + tertiary_ed
     + income_quintiles
     + aust_born
     + interest_pol, data = .) %>%
  lm.beta::lm.beta() %>%
  base::summary()

## 
## Call:
## stats::lm(formula = pol_knowledge ~ age_18_24 + internet_skills + 
##     female + tertiary_ed + income_quintiles + aust_born + interest_pol, 
##     data = .)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -7.052 -1.819  0.014  1.749  8.047 
## 
## Coefficients:
##                  Estimate Standardized Std. Error t value Pr(>|t|)    
## (Intercept)       7.25724      0.00000    0.16747   43.34  < 2e-16 ***
## age_18_24        -0.72437     -0.05817    0.18756   -3.86  0.00011 ***
## internet_skills   0.11722      0.05497    0.03581    3.27  0.00107 ** 
## female           -0.74846     -0.13042    0.08348   -8.97  < 2e-16 ***
## tertiary_ed       0.87490      0.14094    0.09883    8.85  < 2e-16 ***
## income_quintiles  0.13042      0.06266    0.03345    3.90 0.000098 ***
## aust_born         0.04310      0.00655    0.09517    0.45  0.65064    
## interest_pol     -1.56222     -0.43175    0.05367  -29.11  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.42 on 3423 degrees of freedom
##   (524 observations deleted due to missingness)
## Multiple R-squared:  0.293,  Adjusted R-squared:  0.291 
## F-statistic:  202 on 7 and 3423 DF,  p-value: <2e-16

stats::sd(elect_2013$age, na.rm = TRUE)

## [1] 17

stats::sd(elect_2013$pol_knowledge, na.rm = TRUE)

## [1] 2.94

This information is identitical to the output from a summary of a normally linear regression model, except that it has added the standardised beta coefficients, as the second set of numbers (under the heading ‘Standardized’).

If we look at these, we can see that the highest beta is for interest in politics, which is -0.43. When assessing effect size, we can ignore the sign (+/-), and a beta of 0.4 is very large.

When thinking about effect size, and beta, a rough rule of thumb is

• small = ~ 0.05
• medium = ~ 0.1-0.2
• large ,= 0.3+

Reading the beta’s we can see that the three variables in our model with the largest impact on political knowledge are, in rank order:

1. Interest in politics
2. Tertiary education
3. Gender