Lecture 10: Two-way ANOVA II

Statistics in 2-way ANOVA

Jihong Zhang*, Ph.D

Educational Statistics and Research Methods (ESRM) Program*

University of Arkansas

2025-03-07

Overview of Lecture 09 & Lecture 10

  1. The rest of the semester
  2. Advantage of Factorial Design
  3. Two-way ANOVA:
    • Steps for conducting 2-way ANOVA
    • Hypothesis testing
    • Assumptions for 2-way ANOVA
    • Visualization
    • Difference between 1-way and 2-way ANOVA

Question: How about the research scenario when more than two independent variables?

1 Hypothesis testing for two-way ANOVA

Hypothesis testing for two-way ANOVA (I)

  • For two-way ANOVA, there are three distinct hypothesis tests :
    • Main effect of Factor A
    • Main effect of Factor B
    • Interaction of A and B

Definition

F-test

Three separate F-tests are conducted for each!

Main effect

Occurs when there is a difference between levels for one factor

Interaction

Occurs when the effect of one factor on the DV depends on the particular level of the other factor

Said another way, when the difference in one factor is moderated by the other

Said a third way, if the difference between levels of one factor is different, depending on the other factor

Hypothesis testing for two-way ANOVA (II)

  1. Hypothesis test for the main effect with more than 2 levels

    • Mean differences among levels of one factor:

      1. Differences are tested for statistical significance
      2. Each factor is evaluated independently of the other factor(s) in the study

    • Factor A’s Main effect: “Controlling Factor B, are there differences in the DV across Factor A?”

      \(H_0\): \(\mu_{A_1}=\mu_{A_2}=\cdots=\mu_{A_k}\)

      \(H_1\): At least one \(\mu_{A_i}\) is different from the control group in Factor A

    • Factor B’s Main effect : “Controlling Factor A, are there differences in the DV across Factor B?”

      \(H_0\): \(\mu_{B_1}=\mu_{B_2}=\cdots=\mu_{B_k}\)

      \(H_1\): At least one \(\mu_{B_i}\) is different from the control group in Factor A

    • Interaction effect between A and B:

      \(H_0:\mu_{A_1B_2}-\mu_{A_1B_1}=\mu_{A_2B_2}-\mu_{A_2B_1}\)

      or \(H_0: \mu_{A_1B_1}-\mu_{A_2B_1}=\mu_{A_1B_2}-\mu_{A_2B_2}\)

    B’s group differences do not change at different levels of A. OR A’s group differences do not change at different levels of B.

Hypothesis testing for two-way ANOVA (III)

Example

Background: A researcher is investigating how study method (Factor A: Lecture vs. Interactive) and test format (Factor B: Multiple-Choice vs. Open-Ended) affect student performance (dependent variable: test scores). The study involves randomly assigning students to one of the two study methods and then assessing their performance on one of the two test formats.

  • Main Effect of Study Method (Factor A):
    • H0: There is no difference in test scores between students who used the Lecture method and those who used the Interactive method.
    • H1: There is a significant difference in test scores between the two study methods.
  • Main Effect of Test Format (Factor B):
    • H0: There is no difference in test scores between students taking a Multiple-Choice test and those taking an Open-Ended test.
    • H1: There is a significant difference in test scores between the two test formats.
  • Interaction Effect (Study Method × Test Format):
    • H0: The effect of study method on test scores is the same regardless of test format.
    • H1: The effect of study method on test scores depends on the test format (i.e., there is an interaction).

Hypothesis testing for two-way ANOVA (IV)

Example: Tutoring Program and Types of Schools on Grades

  • IVs:
    1. Tutoring Programs: (1) No tutor; (2) Once a week; (3) Daily
    2. Types of schools: (1) Public (2) Private-secular (3) Private-religious
  • Research purpose: to examine the effect of tutoring program (no tutor, once a week, and daily) AND types of school (e.g., public, private-secular, and private-religious) on the students’ grades
  • Question: What are the null and alternative hypotheses for the main effects in the example?:

    • Factor A’s Main effect: “Controlling school types, are there differences in the students’ grade across three tutoring programs?”

      \(H_0\): \(\mu_{\mathrm{no\ tutor}}=\mu_{\mathrm{once\ a\ week}}=\mu_{\mathrm{daily}}\)

    • Factor B’s Main effect : “Controlling tutoring programs, are there differences in the students’ grades across three school types?”

      \(H_0\): \(\mu_{\mathrm{public}}=\mu_{\mathrm{private-religious}}=\mu_{\mathrm{private-secular}}\)

Model 1: Two-Way ANOVA without interaction

  • The main-effect only ANOVA with no interaction has a following statistical form:

\[ \mathrm{Grade} = \beta_0 + \beta_1 \mathrm{Toturing_{Once}} + \beta_2 \mathrm{Toturing_{Daily}} \\ + \beta_3 \mathrm{SchoolType_{PvtS}} + \beta_4 \mathrm{SchoolType_{PvtR}} \]

Data Generation 
library(tidyverse)
# Set seed for reproducibility
set.seed(123)

# Define sample size per group
n <- 30  

# Define factor levels
tutoring <- rep(c("No Tutor", "Once a Week", "Daily"), each = 3 * n)
school <- rep(c("Public", "Private-Secular", "Private-Religious"), times = n * 3)

# Simulate student grades with assumed effects
grades <- c(
  rnorm(n, mean = 75, sd = 5),  # No tutor, Public
  rnorm(n, mean = 78, sd = 5),  # No tutor, Private-Secular
  rnorm(n, mean = 76, sd = 5),  # No tutor, Private-Religious
  rnorm(n, mean = 80, sd = 5),  # Once a week, Public
  rnorm(n, mean = 83, sd = 5),  # Once a week, Private-Secular
  rnorm(n, mean = 81, sd = 5),  # Once a week, Private-Religious
  rnorm(n, mean = 85, sd = 5),  # Daily, Public
  rnorm(n, mean = 88, sd = 5),  # Daily, Private-Secular
  rnorm(n, mean = 86, sd = 5)   # Daily, Private-Religious
)
# Create a dataframe
data <- data.frame(
  Tutoring = factor(tutoring, levels = c("No Tutor", "Once a Week", "Daily")),
  School = factor(school, levels = c("Public", "Private-Secular", "Private-Religious")),
  Grades = grades
)
  • Main effect of Tutoring Programs
    • Collapsing across School Type
    • Ignoring the difference levels of School Type
    • Averaging DV regarding Tutoring Programs across the levels of School Type
  • Main effect of School Type
    • Collapsing across Tutoring Program
    • Ignoring the difference levels of Tutoring Program
    • Averaging DV regarding School Type across the levels of Tutoring Programs

Ignoring the effect of school types, we can tell the main effect of tutor programs on students’ grades (Daily tutoring has the highest grade, followed by once a week).

Code 
# Compute mean and standard error for each tutoring group
tutoring_summary <- data |>
  group_by(Tutoring) |>
  summarise(
    Mean_Grade_byTutor = mean(Grades),
    School = factor(c("Public","Private-Secular", "Private-Religious"), 
                    levels = c("Public","Private-Secular", "Private-Religious"))
  ) 

school_summary <- data |>
  group_by(School) |>
  summarise(
    Mean_Grade_bySchool = mean(Grades)
  ) 

total_summary <- tutoring_summary |> 
  left_join(school_summary, by = "School")

# Plot main effect of tutoring
ggplot(total_summary, aes(x = School)) +
  geom_point(aes(y = Mean_Grade_byTutor, color = Tutoring), size = 5) +
  geom_hline(aes(yintercept = Mean_Grade_byTutor, color = Tutoring), linewidth = 1.3) +
  scale_y_continuous(limits = c(70, 90), breaks = seq(70, 90, 5)) +
  labs(title = "Main Effect of Tutoring on Student Grades",
       x = "School Types",
       y = "Mean Grade") +
  theme_minimal() 

distinct(tutoring_summary[, c("Tutoring", "Mean_Grade_byTutor")])
# A tibble: 3 × 2
# Groups:   Tutoring [3]
  Tutoring    Mean_Grade_byTutor
  <fct>                    <dbl>
1 No Tutor                  76.6
2 Once a Week               81.1
3 Daily                     86.3
  • The x-axis represents three school types.
  • The y-axis represents the mean student grades for the different tutoring programs: No Tutor, Once a Week, and Daily.
  • If the main effect of tutoring is significant, we expect to see noticeable differences in mean grades across tutoring conditions.

You turn: Visualize Main Effect of School Type

Combined Visualization for no-interaction model

  • There is the LARGE effect of Factor Tutoring and very small effect of School Type:
    • Effect of Factor Tutoring: the LARGE vertical distance
    • Effect of Factor School Type: the very small horizontal distance

Real-World Examples of Interaction Effects

Research Question: Does the effectiveness of a new pain medication depend on patient age?

Factors:

  • Factor A: Medication Type (Drug A vs. Placebo)
  • Factor B: Age Group (Young Adults vs. Elderly)

Observed Pattern (Interaction Present):

  • Young Adults: Drug A shows moderate improvement over placebo (Δ = 15 points)
  • Elderly: Drug A shows LARGE improvement over placebo (Δ = 40 points)

Why the interaction?

  • Elderly patients may have different metabolism rates
  • Age-related changes in pain receptors may make them more responsive to the drug
  • Elderly may have more severe baseline pain, allowing more room for improvement

Implication: You cannot simply say “Drug A works” - you must specify “Drug A works especially well for elderly patients”

Visualization:

Research Question: Does the best teaching method depend on students’ learning style?

Factors:

  • Factor A: Teaching Method (Lecture-Based vs. Problem-Based Learning)
  • Factor B: Learning Style (Visual vs. Auditory Learners)

Observed Pattern (Interaction Present):

  • Visual Learners: Problem-Based Learning >> Lecture-Based (80 vs. 65)
  • Auditory Learners: Lecture-Based >> Problem-Based Learning (85 vs. 70)

Why the interaction?

  • Visual learners benefit from hands-on activities with diagrams and visual aids in PBL
  • Auditory learners benefit from verbal explanations and discussions in lectures
  • One-size-fits-all teaching ignores individual differences in cognitive processing

Implication: Educational policy should consider matching teaching methods to learning styles, not universally adopting one method.

Visualization:

Research Question: Does the effect of exercise intensity on weight loss depend on biological sex?

Factors:

  • Factor A: Exercise Intensity (Low vs. High Intensity)
  • Factor B: Gender (Male vs. Female)

Observed Pattern (Interaction Present):

  • Males: High intensity >> Low intensity (12 kg vs. 6 kg lost)
  • Females: High intensity ≈ Low intensity (8 kg vs. 7 kg lost)

Why the interaction? - Males typically have higher muscle mass and testosterone, enabling better response to high-intensity training - Females may have hormonal differences affecting metabolism response to exercise intensity - High-intensity exercise may increase cortisol more in females, potentially interfering with weight loss - Different fat distribution patterns between sexes respond differently to exercise intensity

Implication: Exercise prescriptions should be sex-specific, not generic.

Visualization:

Research Question: Does the effectiveness of therapy type depend on depression severity?

Factors:

  • Factor A: Therapy Type (CBT vs. Medication)
  • Factor B: Depression Severity (Mild vs. Severe)

Observed Pattern (Interaction Present):

  • Mild Depression: CBT >> Medication (75% vs. 60% improvement)
  • Severe Depression: Medication >> CBT (80% vs. 55% improvement)

Why the interaction?

  • Mild depression patients can engage effectively with cognitive strategies in CBT
  • Severe depression patients may lack energy/motivation for CBT homework
  • Severe cases need biological intervention (medication) to restore neurochemical balance
  • CBT requires cognitive capacity that may be impaired in severe depression

Implication: Treatment selection should be severity-matched, not preference-based alone.

Visualization:

Research Question: Does the effect of study time on exam performance depend on intelligence level?

Factors:

  • Factor A: Study Time (Low: 2 hrs vs. High: 10 hrs)
  • Factor B: IQ Level (Below Average vs. Above Average)

Observed Pattern (Interaction Present):

  • Below Average IQ: High study time >> Low study time (75 vs. 50 points)
  • Above Average IQ: High study time ≈ Low study time (85 vs. 82 points)

Why the interaction?

  • High IQ students can learn material efficiently with less study time
  • Below average IQ students need more repetition and practice (more study time helps significantly)
  • High IQ students may reach a “ceiling effect” - already performing well with minimal study
  • Different cognitive abilities create different learning curves

Implication: Educational interventions (increasing study time) may be most beneficial for students who struggle, not top performers.

Visualization:

Research Question: Does caffeine’s effect on alertness depend on time of day?

Factors:

  • Factor A: Caffeine Dose (0 mg vs. 200 mg)
  • Factor B: Time of Day (Morning vs. Evening)

Observed Pattern (Interaction Present):

  • Morning: Small caffeine effect (alertness: 65 vs. 75)
  • Evening: Large caffeine effect (alertness: 35 vs. 70)

Why the interaction?

  • Morning: Natural cortisol levels are already high (circadian rhythm), so caffeine adds less
  • Evening: Cortisol levels drop naturally, so caffeine has a larger stimulating effect
  • Adenosine (sleepiness molecule) accumulates throughout the day - caffeine blocks adenosine receptors
  • Baseline alertness differs dramatically by time of day

Implication: Timing caffeine intake strategically (avoiding morning when cortisol is high) may be more effective.

Visualization:

Common Themes: Why Interactions Occur

Underlying Mechanisms for Interactions

  1. Biological Differences: Age, sex, genetics create different physiological responses
  2. Baseline/Ceiling Effects: Different starting points or maximum capacities across groups
  3. Mechanism Differences: Treatments work through different pathways in different populations
  4. Resource Allocation: Limited cognitive/physical resources are used differently across contexts
  5. Moderating Variables: Third variables (not measured) that correlate with factors
  6. Compensatory Strategies: Different groups adapt differently to interventions
  7. Dose-Response Variation: Sensitivity to “dosage” varies across groups

Critical Insight

Ignoring interactions leads to incorrect conclusions!

If you only report main effects in these examples, you would conclude:

  • “Drug A works” (missing that it works mainly for elderly)
  • “Problem-based learning is better” (missing that it depends on learning style)
  • “High intensity exercise is better” (missing that it mainly benefits males)

Always test and visualize interactions before interpreting main effects!

Model 2: Two-Way ANOVA with Interaction

\[ \mathrm{Grade} = U_1(\mathrm{Tutoring}) + U_2(\mathrm{School}) + U_3(\mathrm{TutoringxSchool}) \]

There variance components (random effects) included in a full two-way ANOVA with two Factors.

Model 2: Visualization of Two-Way ANOVA with Interaction

In the graph:

  • There is the LARGE effect of Factor A and very small effect of Factor B:
  • Effect of Factor A: the LARGE distance between two green stars
  • Effect of Factor B: the very small distance between two red stars

In the graph, there is an intersection between two lines → This is an interaction effect between two factors!

  • The level of Factor B depends upon the level of Factor A – B1 is higher than B2 for A1, but lower for A2 – B2 is lower than B1 for A1, but higher for A2

2 F-statistics

F-statistics for Two-Way ANOVA

F-statistics for Two-Way ANOVA: R code

Source df MS F
Main Effect of Tutoring 2 \(2120.1 = 4240 / 2\) 89.065***
Main Effect of School 2 \(10.8 = 22 / 2\) 0.636
Interaction Effect of School x Tutoring 4 \(46.5 = 186 / 4\) 0.102
Residual 261 \(23.8 = 6213 / 261\) —

F-statistics for Two-Way ANOVA: Degree of Freedom

  • IVs:
    1. Tutoring Programs: (1) No tutor; (2) Once a week; (3) Daily
    2. Types of schools: (1) Public (2) Private-secular (3) Private-religious
  • Degree of freedom (DF):
    • \(df_{tutoring}\) = (Number of levels of Tutoring) - 1 = 3 - 1 = 2
    • \(df_{school}\) = (Number of levels of School) - 1 = 3 - 1 = 2
    • \(df_{interaction}\) = \(df_{tutoring} \times df_{school}\) = \(2 \times 2\) = 4
    • \(df_{total}\) = 270 - 1 = 269
    • \(df_{residual}\) = \(df_{total}-df_{tutoring}-df_{school} - df_{interaction}\) = 269 - 2 - 2 - 4= 261

Calculation of p-values

Based on DF and observed F statistics, we can locate observed F-statistic onto the F-distribution can calculate the p-values.

F-statistics for Two-Way ANOVA: SS and DF

Why residual DF is calculated by subtracting the total DFs with effects’ DFs?

Because DF is linked to SS.

Source SS df
Main Effect of Tutoring 4240 2
Main Effect of School 22 2
Interaction Effect of School x Tutoring 186 4
Residual 6213 261
Total 4240 + 22 + 186 + 6213 2 + 2 + 4 + 261 
sum((data$Grades - mean(data$Grades))^2) # manually calculate total Sum of squares
nrow(data) - 1 # nrow() calculate the number of rows / observations
[1] 10660.77
[1] 269
4240 + 22 + 186 + 6213
2 + 2 + 4 + 261
[1] 10661
[1] 269

Difference between one-way and two-way ANOVA

One-way ANOVA

Total = Between + Within

Two-way ANOVA

Total = FactorA + FactorB + A*B + Residual

Within-factor effects (one-way ANOVA) and Residuals (two-way ANOVA) have same meaning, both of which are unexplained part of DV by the model.

  • In Factor ANOVA Design, the overall between-group variability is divided up into variance terms for each unique source of the factors.
Source 2 IVs 3 IVs 4 IVs
Main effect A, B A, B, C A, B, C, D
Interaction effect AB AB, BC, AC, ABC AB, AC, AD, BC, BD, CD, ABC, ABD, BCD, ACD, ABCD

Calculate the number of interaction effects \[ \mathrm{N_{interaction}} = \sum_{j=2}^J C_J^j \]

##Number of interaction for 4 IVs
choose(4, 2) + choose(4, 3) + choose(4, 4)
[1] 11

F-statistics for Two-way ANOVA: F values

  • F-observed values:

    • Main effect of factor A: \(F_A = \frac{SS_A/df_A}{SS_E/df_E} = \frac{MS_A}{MS_E}\)
      • \(F_{tutoring} = \frac{MS_{tutoring}}{MS_{residual}} = \frac{2120.1}{23.8} = 89.065\)
    • Main effect of factor B: \(F_B = \frac{MS_B}{MS_E}\)
      • \(F_{School} = \frac{MS_{School}}{MS_{residual}} = \frac{10.8}{23.8} = 0.453\)
    • Interaction effect of AxB: \(F_{AB} = \frac{MS_{AB}}{MS_E}\)
      • \(F_{SxT} = \frac{MS_{SxT}}{MS_{residual}} = \frac{46.5}{23.8} = 1.953\)

  • Degree of freedom:

    • \(df_{tutoring}\) = (Number of levels of Tutoring) - 1
    • \(df_{school}\) = (Number of levels of School) - 1
    • \(df_{interaction}\) = \(df_{tutoring} \times df_{school}\)
    • \(df_{total}\) = N - 1
    • \(df_{residual}\) = \(df_{total}-df_{tutoring}-df_{school} - df_{interaction}\)

Exercise

  • A new samples with N = 300
    • Tutoring still has three levels,
    • School types only have two levels: Public vs. Private.

Based on the SS provide below, calculate DF and F values for three effects.

F-statistics for Two-way ANOVA: Sum of squres

Note

In a old-fashion way, you can calculate Sum of Squares based on marginal means and sample size for each cell.

Interpretation

Source Df Sum Sq Mean Sq F value P
Tutoring 2 4240 2120.1 89.065 <2e-16 ***
School 2 22 10.8 0.453 0.636
Tutoring:School 4 186 46.5 1.953 0.102
Residuals 261 6213 23.8
  • Main Effect A: Ignoring type of school, are there differences in grades across type of tutoring?
    • Under alpha=.05 level, because F-observed (F_observed=89.065) exceeds the critical value (p < .001), we reject the null that all means are equal across tutoring type (ignoring the effect of school type).
    • There is a significant main effect of tutoring type on grade.
  • Main Effect B: Ignoring tutoring type, are there differences in grades across type of school?
    • Under alpha=.05 level, because F-observed (F_observed=95.40) exceeds the critical value (p = .453), we retain (or fail to reject) the null hypothesis that all means are equal across school type (ignoring the effect of tutoring type).
    • There is no evidence suggesting a significant main effect of school type on grade.
  • Interaction Effect: Does the effect of tutoring type on grades depend on the type of school?
    • Under alpha=.05 level, because F-observed (F_observed=2.21) does not exceed the critical value (p = 0.102), we fail to reject the null that the effect of Factor A depends on Factor B.
    • There is not a significant interaction between tutoring type and school type on grades.

Assumptions for conducting 2-way ANOVA

  • In order to compare our sample to the null distribution, we need to make sure we are meeting some assumptions for each CELL:
    1. Variance of DV in each cell is about equal. → Homogeneity of variance
    2. DV is normally distributed within each cell. → Normality
    3. Observations are independent. → Independence
  • Robustness of assumption violations:
    1. Violations of independence assumption: bad news! → Not robust to this!
    2. Having a large N and equal cell sizes protects you against violations of the normality assumption → Rough suggestion: have at least 15 participants per cell → If you don’t have large N or equal groups, check cell normality 2 ways: (1) skew/kurtosis values, (2) histograms
    3. Use Levene’s test to check homogeneity of cell variance assumption → If can’t assume equal variances, use Welch or Brown-Forsyth. → However, F is somewhat robust to violations of HOV as long as within-cell standard deviations are approximately equal.

Two-way ANOVA: Calculation based on Grand & Marginal & Cell Means

This research study is an adaptation of Gueguen (2012) described in Andy Field’s text. Specifically, the researchers hypothesized that people with tattoos and piercings were more likely to engage in riskier behavior than those without tattoos and piercings. In addition, the researcher wondered whether this difference varied, depending on whether a person was male or female.

Question: How many IVs? How many levels for each.

Answer: 2 IVs: (1) Whether or not having Tattos and Piercings (2) Male of Female. DV: Frequency of risk behaviors

Step 1: Data importion

Either you can import a CSV file, or manually import the data points (for small samples).

Imagine you have a data with N = 16.

tatto_piercing = rep(c(TRUE, FALSE), each = 8)
gender = rep(c("Male", "Female", "Male", "Female"), each = 4)
outcome = c(5.4, 6.7, 1.8, 6.1, 5.9, 4.6, 2.7, 3.8,
            2.6, 5.8, 1.5, 2.1, .6, .7, .7, 1.8)

dat <- data.frame(
  tatto_piercing = tatto_piercing,
  gender = gender,
  outcome = outcome
)
dat
   tatto_piercing gender outcome
1            TRUE   Male     5.4
2            TRUE   Male     6.7
3            TRUE   Male     1.8
4            TRUE   Male     6.1
5            TRUE Female     5.9
6            TRUE Female     4.6
7            TRUE Female     2.7
8            TRUE Female     3.8
9           FALSE   Male     2.6
10          FALSE   Male     5.8
11          FALSE   Male     1.5
12          FALSE   Male     2.1
13          FALSE Female     0.6
14          FALSE Female     0.7
15          FALSE Female     0.7
16          FALSE Female     1.8

Step 2: Sum of Squares of main effects

\[ SS_A = \sum n_a (M_{Aa} - M_T)^2 \]

  • \(M_{Aa}\) marginal mean of main effect A at each level
  • \(n_a\): sample size for factor A at each level
  • \(M_T\): grand mean of outcome

Obtain sum of square of tatto piercing

Manual way

M_A_table <-  dat |> 
  group_by(tatto_piercing) |>  
  summarise(
    N = n(),
    Mean = mean(outcome)
  ) 
M_T = mean(dat$outcome)
M_A_table
# A tibble: 2 × 3
  tatto_piercing     N  Mean
  <lgl>          <int> <dbl>
1 FALSE              8  1.98
2 TRUE               8  4.62
M_A_table$N[1] * (M_A_table$Mean[1] - M_T)^2 + 
  M_A_table$N[2] * (M_A_table$Mean[2] - M_T)^2
[1] 28.09

or

sum(M_A_table$N * (M_A_table$Mean - M_T)^2)
[1] 28.09

Your turn: Obtain sum of squares of gender

dat object has already been loaded into R. Please, calculate the gender’s sum of squares:

Step 3: Sum of Squares of interaction effect

\[ SS_{AB} = (\sum n_{ab} (M_{AaBb} - M_T)^2) - SS_A - SS_B \]

cell_means <- dat |> 
  group_by(tatto_piercing, gender) |> 
  summarise(
    N = n(),
    Mean = mean(outcome)
  )
cell_means
# A tibble: 4 × 4
# Groups:   tatto_piercing [2]
  tatto_piercing gender     N  Mean
  <lgl>          <chr>  <int> <dbl>
1 FALSE          Female     4  0.95
2 FALSE          Male       4  3   
3 TRUE           Female     4  4.25
4 TRUE           Male       4  5

Your turn

The data frame — cell_means is already available to you.

Note: Sum of squares of interaction effects is the leftover of total sum of squares after removing SS of gender and SS of tutoring program.

\(SS_{total} = \Sigma (N_{ab} * (M_{ab} - M_T))\)

where \(N_{ab}\) is the cell sample size for a-level of factor A and b-level of factor B.

\(M_{ab}\) is the cell means of outcome; \(M_T\) is the grand mean of outcome.

Step 4: Calculate Degree of freedom and F-statistics

Interpretation

#                       Df Sum Sq Mean Sq F value Pr(>F)   
# gender                 1   7.84   7.840   2.942  0.112   
# tatto_piercing         1  28.09  28.090  10.540  0.007 **
# gender:tatto_piercing  1   1.69   1.690   0.634  0.441   
# Residuals             12  31.98   2.665

  • Main Effect of Tattoo-Piercing: Reject the null → Ignoring the gender, there is a significant main effect of tattoo on risky behavior.

  • Main Effect of Gender: Retain the null → Ignoring the tattoo, there is NO significant main effect of gender on risky behavior.

  • Interaction: Retain the null → Under alpha=.05 level, because F-observed (F=0.63) does not exceed the critical value (F=4.75), we fail to reject the null that the effect of Factor A depends on Factor B. → “There is NO significant interaction between gender and tattoo on risky behavior.”

Final discussion

In this example,

  • The interaction effect shows several issues:

    • Violation of independency assumption
    • Violation of normality assumption in the condition of “Male” and “Tattoo=Yes”

  • We can guess/think about one possibility that it might be cause by several reasons. (e.g., too small sample sizes, not randomized assignments of the samples, not a significant interaction effect, etc.)

    • From the two-way ANOVA results, we found that the interaction effect is not significant.

  • Thus, in this case, it is more reasonable to conduct

    1. one-way ANOVA for “Group” variable, and
    2. one-way ANOVA for “Gender” variable, separately.
    • For each of one-way ANOVAs, we should check the assumptions, conduct one-way ANOVA, and post-hoc test separately as well.

Type I Sum of Squares

  • Type I / Type II / Type III sum of squares?
    • Effect sums of squares (\(SS_A\), \(SS_B\), \(SS_{AB}\)) are a decomposition of the total sum of squared deviations from the overall mean (SST).
    • How the \(SS_T\) is decomposed depends on characteristics of the data as well as the hypotheses of interest to the researcher.
  • Type I sums of squares (SS) are based on a sequential decomposition.
    • For example, if your ANOVA model statement is “MODEL Y = A B A*B”, then, the sum of squares are considered in effect order A, B, A*B, with each effect adjusted for all preceding effects in the model.
    • Thus, any variance that is shared between the various effects will be sub-summed by the variable entered earlier.

Pros:

  1. Nice property: balanced or not, SS for all the effects add up to the total SS, a complete decomposition of the predicted sums of squares for the whole model. This is not generally true for any other type of sums of squares.

  2. Preferable when some factors (such as nesting) should be taken out before other factors. For example, with unequal number of male and female, factor “gender” should precede “subject” in an unbalanced design.

Cons: 1. Order matters! 2. Not appropriate for factorial designs, but might be ok for Blocking designs.

Type II Sum of Squares

  • With Type II SS, each main effect is considered as though it were added after the other main effects but before the interaction.
    • Any interaction effects are calculated based on a model already containing the main effects.
    • Any variance that is shared between A and B is not considered part of A or B.
    • Thus, interaction variance that is shared with A or with B will be counted as part of the main effect, and not as part of the interaction effect.

Pros:

  1. appropriate for model building, and natural choice for regression
  2. most powerful when there is no interaction
  3. invariant to the order in which effects are entered into the model

Cons:

  1. For factorial designs with unequal cell samples, Type II sums of squares test hypotheses that are complex functions of the cell so that ordinarily are not meaningful.
  2. Not appropriate for factorial designs.

Type III Sum of Squares

  • Type III SS considers all effects as though they are added last.
    • Any shared variance ends up not being counted in any of the effects.
    • In ANOVA, when the data are balanced (equal cell sizes) and the factors are orthogonal and all three types of sums of squares are identical.
    • Orthogonal, or independent, indicates that there is no variance shared across the various effects, and the separate sums of squares can be added to obtain the model sums of squares.

Pros:

Not sample size dependent: effect estimates are not a function of the frequency of observations in any group (i.e., for unbalanced data, where we have unequal numbers of observations in each group).

Cons:

Not appropriate for designs with missing cells: for ANOVA designs with missing cells, Type III sums of squares generally do not test hypotheses about least squares means, but instead test hypotheses that are complex functions of the patterns of missing cells in higher-order containing interactions and that are ordinarily not meaningful.

In general, for the factorial design, we usually report Type III SS, unless you have missing cells in your model.

Exercise: Two-Way ANOVA with Significant Interaction

A researcher is investigating the effect of learning environment (Online vs. In-Person) and feedback type (Immediate vs. Delayed) on student test scores. The researcher hypothesizes that:

  1. Students in in-person classes will perform better than online students
  2. Immediate feedback will lead to better performance than delayed feedback
  3. The benefit of immediate feedback will be stronger in online environments (interaction effect)

The study randomly assigned 20 students to one of four conditions:

  • Online + Immediate Feedback (n=5)
  • Online + Delayed Feedback (n=5)
  • In-Person + Immediate Feedback (n=5)
  • In-Person + Delayed Feedback (n=5)

Your Task: Calculate the sum of squares for both main effects and the interaction effect manually, then compute F-statistics.

Formula: \(SS_A = \sum n_a (M_{Aa} - M_T)^2\)

where: - \(M_{Aa}\) = marginal mean of Learning Environment at each level - \(n_a\) = sample size for each level of Learning Environment - \(M_T\) = grand mean

Formula: \(SS_B = \sum n_b (M_{Bb} - M_T)^2\)

Formula: \(SS_{AB} = (\sum n_{ab} (M_{AaBb} - M_T)^2) - SS_A - SS_B\)

Key Findings:

  1. Main Effect of Learning Environment: The effect depends on whether it’s significant (check your F-statistic and p-value)

  2. Main Effect of Feedback Type: The effect depends on whether it’s significant (check your F-statistic and p-value)

  3. Interaction Effect: This should be significant! The interaction plot shows that:

    • In Online environments, Immediate feedback leads to MUCH higher scores than Delayed feedback
    • In In-Person environments, the difference between Immediate and Delayed feedback is smaller (and reversed!)
    • This crossing pattern indicates a strong interaction: the effect of feedback type depends on the learning environment

Lecture Summary: Key Takeaways

1. Conceptual Understanding

Two-way ANOVA allows us to:

  • Examine the effects of two independent variables simultaneously
  • Test for three distinct effects: Main effect A, Main effect B, and Interaction A×B
  • Investigate whether the effect of one factor depends on the level of another factor (interaction)
  • Achieve greater statistical power and efficiency compared to multiple one-way ANOVAs

Main Effects vs. Interaction:

  • Main Effect: The average effect of one factor, ignoring (collapsing across) the other factor
  • Interaction Effect: When the effect of one factor changes depending on the level of the other factor
    • Look for non-parallel lines in interaction plots
    • Crossing lines indicate strong interactions
    • When interaction is significant, main effects may be misleading!

2. Statistical Calculations

Partitioning Variance in Two-Way ANOVA:

\[ SS_{Total} = SS_A + SS_B + SS_{AB} + SS_{Residual} \]

Key Formulas:

  • Main Effect A: \(SS_A = \sum n_a (M_{Aa} - M_T)^2\)
  • Main Effect B: \(SS_B = \sum n_b (M_{Bb} - M_T)^2\)
  • Interaction: \(SS_{AB} = (\sum n_{ab} (M_{AaBb} - M_T)^2) - SS_A - SS_B\)

F-statistics:

\[ F = \frac{MS_{effect}}{MS_{residual}} = \frac{SS_{effect}/df_{effect}}{SS_{residual}/df_{residual}} \]

Degrees of Freedom:

  • \(df_A = k_A - 1\) (number of levels of Factor A minus 1)
  • \(df_B = k_B - 1\) (number of levels of Factor B minus 1)
  • \(df_{AB} = df_A \times df_B\) (interaction df)
  • \(df_{residual} = N - 1 - df_A - df_B - df_{AB}\)

3. Practical Considerations

Assumptions for Two-Way ANOVA:

  1. Independence: Observations must be independent (most critical!)
  2. Normality: DV should be normally distributed within each cell
  3. Homogeneity of Variance: Equal variances across all cells (Levene’s test)

Robustness:

  • F-test is fairly robust to normality violations with large N and equal cell sizes
  • Recommend at least 15 participants per cell
  • Use Welch or Brown-Forsythe tests if variances are unequal
  • Never compromise on independence assumption

Type I, II, III Sum of Squares:

  • Type I: Sequential (order matters) - appropriate for nested/blocking designs
  • Type II: Each main effect adjusted for other main effects - good for model building
  • Type III: Each effect adjusted for all others - recommended for factorial designs with balanced data
  • With balanced designs (equal cell sizes), all three types give identical results!

4. Interpretation Guidelines

When Interaction is Significant:

  • Interpret interaction first before discussing main effects
  • Main effects may be misleading or uninterpretable when interaction is present
  • Focus on simple effects: the effect of one factor at each level of the other
  • Use interaction plots to visualize and explain the pattern

When Interaction is NOT Significant:

  • Proceed to interpret main effects
  • If main effect is significant with >2 levels, conduct post-hoc tests (e.g., Tukey HSD)
  • Report effect sizes (partial η²) alongside p-values

Reporting Results:

Include in your report:

  1. Descriptive statistics (means, SDs) for all cells
  2. ANOVA table with F-values, df, and p-values
  3. Effect sizes
  4. Interaction plot (even if not significant)
  5. Post-hoc tests if needed
  6. Assumption checks

5. Common Pitfalls to Avoid

  1. Don’t ignore significant interactions! Main effects alone don’t tell the whole story
  2. Don’t forget assumption checks - especially for small samples
  3. Don’t confuse interaction with correlation - they are different concepts
  4. Don’t over-interpret non-significant results - absence of evidence ≠ evidence of absence
  5. Don’t use multiple one-way ANOVAs instead of two-way - you’ll miss interactions and inflate Type I error
  6. Don’t forget to report effect sizes - statistical significance ≠ practical significance

6. Next Steps in Your Learning

Extending Two-Way ANOVA:

  • Three-way ANOVA: Adding a third factor (exponentially more complex!)
  • Mixed ANOVA: Combining between-subjects and within-subjects factors
  • ANCOVA: Adding continuous covariates to control for confounding variables
  • MANOVA: Multiple dependent variables simultaneously
  • Repeated Measures ANOVA: Same subjects measured multiple times

When to Use Two-Way ANOVA:

✓ Two categorical independent variables (factors) ✓ One continuous dependent variable ✓ Independent observations (between-subjects design) ✓ Interest in both main effects and interaction ✓ Adequate sample size (≥15 per cell recommended)

Alternatives to Consider:

  • Regression with interactions: More flexible, can handle continuous IVs
  • Mixed-effects models: Better for unbalanced designs or nested data
  • Non-parametric tests: When assumptions are severely violated (e.g., Aligned Rank Transform ANOVA)

Fin

Thank you for your attention!