Posthoc Testing in Two-Way ANOVA

Introduction

  • Recall that ANOVA allows us to compare the means of three or more groups.

  • In one-way ANOVA, we are only considering one factor (grouping variable).

  • In this module, we discuss two-way ANOVA, which allows us to consider a second factor (grouping variable).

  • We now will discuss performing posthoc tests in the two-way ANOVA setting.

R Setup

library(tidyverse)
library(ssstats)
exam_scores <- read_csv("https://raw.githubusercontent.com/samanthaseals/SDSI/refs/heads/main/files/data/lectures/5-1-exam_scores.csv")
fatigue_scores <- read_csv("https://raw.githubusercontent.com/samanthaseals/SDSI/refs/heads/main/files/data/lectures/5-2-fatigue_scores.csv")

Posthoc Testing

  • We can follow up two-way ANOVA tests with posthoc tests.

  • Recall our hypotheses for main effects,

    • H_0: \mu_1 = \mu_2 = ... = \mu_a
    • H_1: at least one \mu_i is different

  • We will again turn to posthoc tests to determine pairwise differences.

Posthoc Testing (R)

  • We will again use the posthoc() function,
dataset_name |> posthoc(outcome = response_variable,
                        A = factor_A,
                        B = factor_B,
                        interaction = TRUE or FALSE,
                        adjust = TRUE or FALSE)

Example 1

  • Recall the two-way ANOVA results for the fatigue_scores data,

  • Does the effect of shift on fatigue depend on which unit an officer works in?

fatigue_scores |> anovas(outcome = fatigue_score,
                         A = shift,
                         B = unit)
Two-way ANOVA (with interaction)

Source       df          SS        MS       F  p-value 
───────────────────────────────────────────────── 
shift         1    998.2845  998.2845  10.274    0.002 
unit          2     51.7080   25.8540   0.266    0.767 
shift:unit    2    729.3173  364.6587   3.753    0.025 
Residuals   174  16906.4077   97.1633                  
───────────────────────────────────────────────── 


Hypotheses (shift:unit):
H₀: There is no interaction between shift and unit
H₁: There is an interaction between shift and unit
Test statistic: F(2, 174) = 3.753
p-value: 0.025
Conclusion: Reject the null hypothesis (p = 0.025 < α = 0.050)

Example 1

  • We can look at the posthoc test of the interaction term.

  • Treating it as exploratory,

fatigue_scores |> posthoc(outcome = fatigue_score,
                       A = shift,
                       B = unit,
                       interaction = TRUE,
                       adjust = FALSE)
Post-hoc analysis: Fisher's LSD

Following: two-way ANOVA (fatigue_score ~ shift * unit)

Main effect: shift 
Comparison    Difference  p-value   
──────────────────────────────────── 
Night vs Day      4.7100    0.002  * 
──────────────────────────────────── 

Main effect: unit 
Comparison      Difference  p-value   
────────────────────────────────────── 
K9 vs Desk          1.2200    0.499   
Patrol vs Desk      1.0300    0.568   
Patrol vs K9       -0.1900    0.916   
────────────────────────────────────── 

Interaction: shift × unit 
Comparison                  Difference  p-value   
────────────────────────────────────────────────── 
Night:Desk vs Day:Desk          4.4833    0.080   
Day:K9 vs Day:Desk              3.5133    0.169   
Day:K9 vs Night:Desk           -0.9700    0.704   
Night:K9 vs Day:Desk            3.4100    0.182   
Night:K9 vs Night:Desk         -1.0733    0.674   
Night:K9 vs Day:K9             -0.1033    0.968   
Day:Patrol vs Day:Desk         -1.6033    0.530   
Day:Patrol vs Night:Desk       -6.0867    0.018  * 
Day:Patrol vs Day:K9           -5.1167    0.046  * 
Day:Patrol vs Night:K9         -5.0133    0.050   
Night:Patrol vs Day:Desk        8.1467    0.002  * 
Night:Patrol vs Night:Desk      3.6633    0.152   
Night:Patrol vs Day:K9          4.6333    0.070   
Night:Patrol vs Night:K9        4.7367    0.064   
Night:Patrol vs Day:Patrol      9.7500  < 0.001  * 
────────────────────────────────────────────────── 

* p < α = 0.050

Example 2

  • Recall the two-way ANOVA results for the exam_scores data,

  • Does the effect of training method depend on experience level?

exam_scores |> anovas(outcome = exam_score,
                      A = training_method,
                      B = experience_level,
                      interaction = TRUE)
Two-way ANOVA (with interaction)

Source                             df          SS        MS      F  p-value 
────────────────────────────────────────────────────────────────────── 
training_method                     2   1339.2191  669.6096  7.584  < 0.001 
experience_level                    1     88.9014   88.9014  1.007    0.317 
training_method:experience_level    2      4.8084    2.4042  0.027    0.973 
Residuals                         174  15362.8583   88.2923                 
────────────────────────────────────────────────────────────────────── 


Hypotheses (training_method:experience_level):
H₀: There is no interaction between training_method and experience_level
H₁: There is an interaction between training_method and experience_level
Test statistic: F(2, 174) = 0.027
p-value: 0.973
Conclusion: Fail to reject the null hypothesis (p = 0.973 ≥ α = 0.050)

Example 2

  • Recall the two-way ANOVA results for the exam_scores data,

  • Are there main effects of training method or experience level?

exam_scores |> anovas(outcome = exam_score,
                      A = training_method,
                      B = experience_level,
                      interaction = FALSE)
Two-way ANOVA (no interaction)

Source             df          SS        MS      F  p-value 
────────────────────────────────────────────────────── 
training_method     2   1339.2191  669.6096  7.669  < 0.001 
experience_level    1     88.9014   88.9014  1.018    0.314 
Residuals         176  15367.6668   87.3163                 
────────────────────────────────────────────────────── 


Hypotheses (training_method):
H₀: All means across levels of training_method are equal
H₁: At least one mean across levels of training_method differs
Test statistic: F(2, 176) = 7.669
p-value: < 0.001
Conclusion: Reject the null hypothesis (p = < 0.001 < α = 0.050)

Hypotheses (experience_level):
H₀: All means across levels of experience_level are equal
H₁: At least one mean across levels of experience_level differs
Test statistic: F(1, 176) = 1.018
p-value: 0.314
Conclusion: Fail to reject the null hypothesis (p = 0.314 ≥ α = 0.050)

Example 2

  • We can look at the posthoc tests of the main effects.

  • Treating it as confirmatory,

exam_scores |> posthoc(outcome = exam_score,
                      A = training_method,
                      B = experience_level,
                      interaction = FALSE,
                      adjust = TRUE)
Post-hoc analysis: Tukey's HSD

Following: two-way ANOVA (exam_score ~ training_method + experience_level)

Main effect: training_method 
Comparison               Difference  p-value   
─────────────────────────────────────────────── 
Peer vs Led vs Enhanced     -4.6300    0.020  * 
Standard vs Enhanced        -6.4867  < 0.001  * 
Standard vs Peer vs Led     -1.8567    0.523   
─────────────────────────────────────────────── 

Main effect: experience_level 
Comparison         Difference  p-value   
───────────────────────────────────────── 
Veteran vs Rookie      1.4056    0.314   
───────────────────────────────────────── 

* p < α = 0.050

Wrap Up

  • In this lecture, we have covered posthoc testing in two-way ANOVA.

    • Interactions.

    • Main effects.

  • In our following lectures, we will discuss next steps.

    • profile plots