Posthoc Testing in One-Way Analyses

Introduction

• Previous: one sample, two samples

• Now: more than two groups, continuous outcome

  • One-way ANOVA

  • Kruskal-Wallis

  • posthoc testing

• This lecture: posthoc testing

R Setup

library(tidyverse)
library(ssstats)
commute_times <- read_csv("https://raw.githubusercontent.com/samanthaseals/SDSI/refs/heads/main/files/data/lectures/4-2-commute_times.csv")
citations <- read_csv("https://raw.githubusercontent.com/samanthaseals/SDSI/refs/heads/main/files/data/lectures/4-4-citations.csv")
academy_scores <- read_csv("https://raw.githubusercontent.com/samanthaseals/SDSI/refs/heads/main/files/data/lectures/4-1-academy_scores.csv")
response_times <- read_csv("https://raw.githubusercontent.com/samanthaseals/SDSI/refs/heads/main/files/data/lectures/4-3-response_times.csv")

Posthoc Testing

• Recall our hypotheses in one-way ANOVA,

  • H_0: \mu_1 = \mu_2 = ... = \mu_k
  • H_1: at least one \mu_i is different

• The F test does not tell us which mean is different… only that a difference exists.

• In theory, we could perform repeated t tests to determine pairwise differences.

  • Recall that ANOVA is an extension of the t test… or that the t test is a special case of ANOVA.
  • However, this will increase the Type I error rate (\alpha).

Posthoc Testing

• Recall that the Type I error rate, \alpha, is the probability of incorrectly rejecting H_0.

  • i.e., we are saying there is a difference between the means when there is actually not a difference.

• Suppose we are comparing 5 groups.

  • This is 10 pairwise comparisons!!

    • 1-2, 1-3, 1-4, 1-5, 2-3, 2-4, 2-5, 3-4, 3-5, 4-5

  • If we perform repeated t tests under \alpha=0.05, we are inflating the Type I error to 0.40! 😵

Posthoc Testing

• When performing posthoc comparisons, we can choose one of two paths:

  • Control the Type I (familywise) error rate.
  • Do not control the Type I error rate.

• Note that controlling the Type I error rate is more conservative than when we do not control it.

  • “Conservative” = more difficult to reject.

• Generally, statisticians:

  • do not control the Type I error rate if examining the results of pilot/preliminary studies that are exploring for general relationships.

  • do control the Type I error rate if examining the results of confirmatory studies and are attempting to confirm relationships observed in pilot/preliminary studies.

Posthoc Testing

• The posthoc tests we will learn:

  • Tukey’s test

    • Performs all pairwise tests and controls the Type I error rate

  • Fisher’s least significant difference

    • Performs all pairwise tests but does not control the Type I error rate

• Caution: we should only perform posthoc tests if we have determined that a general difference exists!

  • i.e., when we reject when looking at either ANOVA or Kruskal-Wallis

Posthoc Testing

• We will be performing pairwise comparisons.

• For each test performed, we are testing:

  • If using an ANOVA-based method:

    • H_0: \ \mu_i = \mu_j
    • H_1: \ \mu_i \ne \mu_j

  • If using a KW-based method:

    • H_0: \ M_i = M_j
    • H_1: \ M_i \ne M_j

Example 4

• Okonkwo again, “Dispatch keeps flagging Ashford as slow on 911 response, but Ashford’s also our biggest district by square footage, so maybe that’s just geography. Let’s compare response times across districts.”

• We will use the response_times dataset to answer the following question:

  • Do 911 response times differ across the five districts?

• First, let’s check assumptions.

Example 4

response_times |> anova_check(outcome = response_minutes,
                              grouping = district)

Example 4

• Examining the variance assumption further,

response_times |> multiple_variances(outcome = response_minutes,
                                     grouping = district)

Brown-Forsythe-Levene test for equality of variances

s²[Ashford] = 35.4302 (s = 5.9523)
s²[Coldharbor] = 10.9707 (s = 3.3122)
s²[Fairview] = 20.3763 (s = 4.514)
s²[Meadowbrook] = 10.6601 (s = 3.265)
s²[Northgate] = 8.6518 (s = 2.9414)

Hypotheses:
H₀: All group variances are equal (no effect of district on variance)
H₁: At least one group variance differs (effect of district on variance exists)
Test statistic: F(4, 145) = 2.421
p-value: 0.051
Conclusion: Fail to reject the null hypothesis (p = 0.051 ≥ α = 0.050)

Example 4

• Now, looking at the omnibus test (ANOVA),

response_times |> anovas(outcome = response_minutes,
                         grouping = district)

One-way ANOVA

Source      df         SS        MS       F  p-value 
─────────────────────────────────────────────── 
district     4   857.2949  214.3237  12.448  < 0.001 
Residuals  145  2496.5840   17.2178                  
─────────────────────────────────────────────── 


Hypotheses (district):
H₀: All group means are equal (no effect of district)
H₁: At least one group mean differs (effect of district exists)
Test statistic: F(4, 145) = 12.448
p-value: < 0.001
Conclusion: Reject the null hypothesis (p = < 0.001 < α = 0.050)

Posthoc Testing: Generic (R)

• We will use the posthoc() function from library(ssstats) to perform Tukey’s posthoc test (resulting p are adjusted for multiple comparisons).

1dataset_name |> posthoc(outcome = continuous_variable,
2                        grouping = grouping_variable,
3                        adjust = TRUE or FALSE,
4                        nonparametric = TRUE or FALSE,
5                        alpha = alpha_level)

Posthoc Testing: Tukey’s Test (ANOVA)

• Tukey’s test allows us to do all pairwise comparisons while controlling \alpha.

• Hypotheses

  • H_0: \ \mu_i = \mu_j
  • H_1: \ \mu_i \ne \mu_j

• Test Statistic

Q = \frac{|\bar{y}_i - \bar{y}_j|}{\sqrt{ \frac{\text{MS}_{\text{E}}}{2} \left( \frac{1}{n_i} + \frac{1}{n_j} \right) }}

Posthoc Testing: Tukey’s (R)

• To use posthoc() for Tukey’s test,

1dataset_name |> posthoc(outcome = continuous_variable,
2                        grouping = grouping_variable,
3                        adjust = TRUE,
4                        nonparametric = FALSE,
5                        alpha = alpha_level)

Example 4

• Examining Tukey’s for our example,

response_times |> posthoc(outcome = response_minutes,
                          grouping = district,
                          adjust = TRUE,
                          nonparametric = FALSE)

Post-hoc analysis: Tukey's HSD

Following: one-way ANOVA (response_minutes ~ district)

Comparison                 Difference  p-value   
───────────────────────────────────────────────── 
Coldharbor vs Ashford         -6.4567  < 0.001  * 
Fairview vs Ashford           -4.9967  < 0.001  * 
Meadowbrook vs Ashford        -6.0833  < 0.001  * 
Northgate vs Ashford          -5.8833  < 0.001  * 
Fairview vs Coldharbor         1.4600    0.652   
Meadowbrook vs Coldharbor      0.3733    0.997   
Northgate vs Coldharbor        0.5733    0.984   
Meadowbrook vs Fairview       -1.0867    0.848   
Northgate vs Fairview         -0.8867    0.922   
Northgate vs Meadowbrook       0.2000    1.000   
───────────────────────────────────────────────── 

* p < α = 0.050

Posthoc Testing: Fisher’s Test (ANOVA)

• Fisher’s allows us to test all pairwise comparisons but does not control the \alpha.

• Hypotheses:

  • H_0: \ \mu_i = \mu_j
  • H_1: \ \mu_i \ne \mu_j

• Test Statistic:

t = \frac{|\bar{y}_i - \bar{y}_j|}{\sqrt{ \text{MS}_{\text{E}} \left( \frac{1}{n_i} + \frac{1}{n_j} \right) }}

Posthoc Testing: Fisher’s (R)

• To use posthoc() for Fisher’s test,

1dataset_name |> posthoc(outcome = continuous_variable,
2                        grouping = grouping_variable,
3                        adjust = FALSE,
4                        nonparametric = FALSE,
5                        alpha = alpha_level)

Example 4

• Examining Fisher’s for our example,

response_times |> posthoc(outcome = response_minutes,
                          grouping = district,
                          adjust = FALSE,
                          nonparametric = FALSE)

Post-hoc analysis: Fisher's LSD

Following: one-way ANOVA (response_minutes ~ district)

Comparison                 Difference  p-value   
───────────────────────────────────────────────── 
Coldharbor vs Ashford         -6.4567  < 0.001  * 
Fairview vs Ashford           -4.9967  < 0.001  * 
Fairview vs Coldharbor         1.4600    0.175   
Meadowbrook vs Ashford        -6.0833  < 0.001  * 
Meadowbrook vs Coldharbor      0.3733    0.728   
Meadowbrook vs Fairview       -1.0867    0.312   
Northgate vs Ashford          -5.8833  < 0.001  * 
Northgate vs Coldharbor        0.5733    0.593   
Northgate vs Fairview         -0.8867    0.409   
Northgate vs Meadowbrook       0.2000    0.852   
───────────────────────────────────────────────── 

* p < α = 0.050

Example 2

• Recall our second question: do assessment scores differ across the three training academies (Meadowbrook, Coldharbor, and Northgate)?

Example 2

• Do assessment scores differ across the three training academies (Meadowbrook, Coldharbor, and Northgate)?

academy_scores |> kruskal_wallis(outcome = score,
                                 grouping = academy)

Kruskal-Wallis test

M[Coldharbor] = 80.9 (n = 32)
M[Meadowbrook] = 74.75 (n = 32)
M[Northgate] = 82.6 (n = 32)

Hypotheses:
H₀: All group distributions are equal (no effect of academy)
H₁: At least one group distribution differs in location (effect of academy exists)
Test statistic: H(2) = 9.801
p-value: 0.007
Conclusion: Reject the null hypothesis (p = 0.007 < α = 0.050)

Posthoc Testing: Dunn’s Test

• We can also perform posthoc testing in the Kruskal-Wallis setting using Dunn’s test.

  • Rather than compare pairwise means, it compares pairwise average ranks.

• Hypotheses:

  • H_0: \ M_{i} = M_{j}
  • H_1: \ M_{i} \ne M_{j}

• Test Statistic:

z_0 = \frac{|\bar{R}_i - \bar{R}_j|}{\sqrt{ \frac{n(n+1)}{12} \left( \frac{1}{n_i} + \frac{1}{n_j} \right) }}

Posthoc Testing: Adjusted Dunn’s (R)

• To use posthoc() for Dunn’s test adjusting for the Type I error rate,

1dataset_name |> posthoc(outcome = continuous_variable,
2                        grouping = grouping_variable,
3                        adjust = TRUE,
4                        nonparametric = TRUE,
5                        alpha = alpha_level)

Example 2

• Examining adjusted Dunn’s for our example,

academy_scores |> posthoc(outcome = score,
                          grouping = academy,
                          adjust = TRUE,
                          nonparametric = TRUE)

Post-hoc analysis: Dunn's test (Bonferroni adjustment)

Following: Kruskal-Wallis (score ~ academy)

Comparison                      z  p-value   
───────────────────────────────────────────── 
Coldharbor vs Meadowbrook   2.455    0.021  * 
Coldharbor vs Northgate    -0.455    0.973   
Meadowbrook vs Northgate   -2.910    0.005  * 
───────────────────────────────────────────── 

* p < α = 0.050

Posthoc Testing: Unadjusted Dunn’s (R)

• To use posthoc() for Dunn’s test without adjusting for the Type I error rate,

1dataset_name |> posthoc(outcome = continuous_variable,
2                        grouping = grouping_variable,
3                        adjust = FALSE,
4                        nonparametric = TRUE,
5                        alpha = alpha_level)

Example 2

• Examining unadjusted Dunn’s for our example,

academy_scores |> posthoc(outcome = score,
                          grouping = academy,
                          adjust = FALSE,
                          nonparametric = TRUE)

Post-hoc analysis: Dunn's test (no adjustment)

Following: Kruskal-Wallis (score ~ academy)

Comparison                      z  p-value   
───────────────────────────────────────────── 
Coldharbor vs Meadowbrook   2.455    0.007  * 
Coldharbor vs Northgate    -0.455    0.324   
Meadowbrook vs Northgate   -2.910    0.002  * 
───────────────────────────────────────────── 

* p < α = 0.050

Example 1

• Recall our first question: does commute time differ across the four districts?

commute_times |> anovas(outcome = commute_minutes,
                        grouping = district)

One-way ANOVA

Source      df         SS       MS      F  p-value 
───────────────────────────────────────────── 
district     3   120.0100  40.0033  0.780    0.507 
Residuals  116  5945.7900  51.2568                 
───────────────────────────────────────────── 


Hypotheses (district):
H₀: All group means are equal (no effect of district)
H₁: At least one group mean differs (effect of district exists)
Test statistic: F(3, 116) = 0.78
p-value: 0.507
Conclusion: Fail to reject the null hypothesis (p = 0.507 ≥ α = 0.050)

Example 1

• Halt!

• p = 0.507 \to FTR \to we did not detect a difference in the means.

• We do not perform posthoc testing when ANOVA or Kruskal-Wallis did not show an overall difference.

• Remember that we said ANOVA and Kruskal-Wallis were omnibus tests.

  • If the omnibus test does not suggest a difference exists, we do not further poke around.

Wrap Up

• This lecture introduced posthoc testing, the follow-up to ANOVA and Kruskal-Wallis.

  • If ANOVA \to Tukey’s (adjusted) or Fisher’s (unadjusted)

  • If Kruskal-Wallis \to adjusted Dunn’s or unadjusted Dunn’s

• We only perform posthoc testing when the omnibus test (ANOVA or Kruskal-Wallis) has determined there is an overall difference in the means or medians.

• We also discussed the familywise error rate (FWE), \alpha.

  • When we adjust to preserve the FWE, we maintain \alpha = 0.05, regardless of the number of tests performed.

  • When we do not adjust to preserve the FWE, we allow \alpha to inflate; the inflation depends on number of pairwise comparisons.