Lecture 05: ANOVA Comparisons and Contrasts

Experimental Design in Education

Jihong Zhang*, Ph.D

Educational Statistics and Research Methods (ESRM) Program*

University of Arkansas

2025-02-17

Class Outline

  • Planned Contrast
  • Example: Group Comparison - STEM vs. Non-STEM Groups
    • ANOVA-style t-statistics and Regression-style Coding Schema in R
  • Effect sizes.

1 Planned Contrasts

Definition: Planned Contrasts

  • Pre-defined:

    • Unlike post-hoc tests, planned contrasts are determined before looking at the data, meaning the researcher has a specific hypothesis about which groups to compare.
    • Definition: Planned contrasts are hypothesis-driven comparisons made before data collection.

  • Weights assigned:

    • To perform a planned contrast, each group is assigned a numerical “weight” which reflects its role in the comparison, with the weights usually summing to zero.

    \[ D = weights * means \]

    • D: unscalled group differences given coding scheme

Example of Planned Contrasts

  • Imagine a study comparing the effects of three different study methods (A, B, C) on test scores.

    • One planned contrast might be to compare the average score of method A (considered the “experimental” method) against the combined average of methods B and C (considered the “control” conditions),

    • Testing the hypothesis that method A leads to significantly higher scores than the traditional methods.

    • \(H_0: \mu_{A} = \frac{\mu_B+\mu_C}{2}\), we also call this complex contrast

  • When to use planned contrasts:

    • When you have a clear theoretical basis for predicting specific differences between groups in your study.
    • When you are only interested in a few specific comparisons, not all possible pairwise comparisons.

What Does Each Contrast Tell Us?

  • Each contrast is a mean comparison (via t-test).
  • Simple contrast (pairwise) compares two individual group means.
  • Complex contrast compares a combination of group means.
  • Must be theoretically justified for meaningful interpretation.

Simple vs. Complex Comparisons

  • Simple Comparison: Two groups directly compared.
    • Example: \(H_0: \mu_2 = \mu_3\)
  • Complex Comparison: Combines means of multiple groups.
    • Example: \(H_0: \frac{(\mu_1 + \mu_2)}{2} = \mu_3\)
    • Example: \(H_0: \frac{(\mu_1 + \mu_2 + \mu_3)}{3} = \frac{(\mu_4 + \mu_5)}{2}\)

Note

We should not test all possible combinations of groups. Instead, justify your comparison plan before performing statistic analysis.

Today’s focus: Complex Comparisons

  • We perform ominous test in last lecture, which gives us simple contrasts
    • which provides all pairwise group comparisons (simple contrasts)
  • Today we focus more on complex contrasts.
    • Helmert contrast: Compares each mean to the mean of subsequent groups.
    • Sum (Deviation) contrast: each group compared to grand mean
    • Polynomial contrast: Tests for trends in ordered data.
  • By default, R uses Treatment contrasts: each group compared to the reference group
    • Question: is “Treatment constrast” simple or complex constrast
    • G1 vs. Treatment (reference)
    • G2 vs. Treatment (reference)
    • ….

Orthogonal vs. Non-Orthogonal Contrasts

  • Orthogonal Contrasts: Independent from each other, sum of product of weights equals zero.

  • Non-Orthogonal Contrasts: Not independent, lead to inflated Type I error rates.

    Note

    Orthogonal contrasts allow clear interpretation without redundancy.

  • Orthogonal contrasts follows a series of group comparisons that does not overlap variances.

Orthogonal contrasts from variances: no redundency

Hermert contrast for example

  • With a logical control group, a good first contrast compares all treatment groups to the one control group.
  • To get each level of IV alone you should have one less contrast than your number of IV levels (3 levels = 2 contrasts)
  • Once an IV level appears by itself, it shouldn’t reappear in subsequent contrasts

VarianceDiagram A Total Variance Explained B Variance for G1 and G2 A->B E Variance for G3 A->E C Variance for G1 B->C D Variance for G2 B->D

Example of Orthogonal Contrasts

  • Contrast 1: g3 vs. (g1, g2)
  • Contrast 2: g1 vs. g2

Orthogonal Planned Contrasts

  • If the same exact combination of means is not found in more than one contrast, the contrasts are independent (orthogonal)
    • Check this by ensuring that the product of the weights across all contrasts sum to zero
  • For a orthogonal comparison, contrasts are independent with each other:
    • We weight the means included on each side of the contrast
    • Each contrast has a sum of weights as 0
    • Groups not in the contrast get a weight of 0
  • Why does independence matter?
    • Type I error rate is unaffected by independent (orthogonal) contrasts
    • Interpretation of contrasts is cleaner because contrasts aren’t related (you’ve isolated effects)
Group Contrast 1 Contrast 2 Product
G1 +1 -1 -1
G2 +1 +1 +1
G3 -2 0 0
Sum 0 0 0

Contrasts’ Independence checking in R

contras <- matrix(
  c(1, 1, -2,
    -1, 1, 0), ncol = 2
)
contras
     [,1] [,2]
[1,]    1   -1
[2,]    1    1
[3,]   -2    0
t(contras[,1]) %*% contras[,2] ## the cross-product of two constrasts should be zero
     [,1]
[1,]    0
crossprod(contras) ## if non-diagnonal elements are zero = orthogonal
     [,1] [,2]
[1,]    6    0
[2,]    0    2
cor(contras) ## or correlation matrix is identity matrix
     [,1] [,2]
[1,]    1    0
[2,]    0    1

Computing Planned Contrasts

  • Formula for contrast value: \(C = c_1\mu_1 + c_2\mu_2 + \dots + c_k\mu_k\)
  • Test statistic: \(t = \frac{C}{\sqrt{MSE \sum \frac{c_i^2}{n_i}}}\)
    • \(MSE\): Mean Square Error from ANOVA
    • \(c_i\): Contrast coefficients
    • \(n_i\): Sample size per group

2 Example - STEM vs. Non-STEM Groups

Background

  • Hypothesis: STEM students have different growth mindset scores(score) than non-STEM students.
  • Weights assigned:
    • STEM (Engineering, Chemistry): \(+\frac{1}{2}\)
    • Non-STEM (Education, Political Sci, Psychology): \(-\frac{1}{3}\)
  • Compute contrast value and test using t-statistic.

Set Contrasts in R

Code 
library(tidyverse)
library(kableExtra)
library(here)
# Set seed for reproducibility
set.seed(42)
dt <- read.csv(here("teaching/2025-01-13-Experiment-Design/Lecture05","week5_example.csv"))
options(digits = 5)
summary_tbl <- dt |> 
  group_by(group) |> 
  summarise(
    N = n(),
    Mean = mean(score),
    SD = sd(score),
    shapiro.test.p.values = shapiro.test(score)$p.value
  )
kable(summary_tbl)
group N Mean SD shapiro.test.p.values
g1 28 4.2500 3.15054 0.07759
g2 28 2.7589 2.19478 0.07605
g3 28 3.5446 2.86506 0.00623
g4 28 3.8568 0.58325 0.03023
g5 28 2.0243 1.30911 0.06147
  • HOV Assumption: Levene’s Test
aov_fit <- aov(score ~ group, data = dt)
car::leveneTest(aov_fit) |> as.data.frame() |> kable()
Df F value Pr(>F)
group 4 12.966 0
135 NA NA

Even though assumption checkings did not pass using original categorical levels, we may be still interested in different group contrasts.

Complex Contrast Matrix

  • There are multiple “canned” contrasts: Helmert, Sum (Effective Coding), Treatment

For example, Helmert Four contrasts:

  1. g1 vs. g2: \(\mu_{Engineering} = \mu_{Education}\)
  2. \(\frac{g1+g2}{2}\) vs. g3: \(\mu_{non-Chemistry} = \mu_{Chemistry}\)
  3. \(\frac{g1+g2+g3}{3}\) vs. g4: \(\mu_{non-Political} = \mu_{Political}\)
  4. \(\frac{g1+g2+g3+g4}{4}\) vs. g5: \(\mu_{non-Psychology} = \mu_{Psychology}\)

Summary Statistics:

Code 
dt$group <- factor(dt$group, levels = c("g1", "g2", "g3", "g4", "g5"))
groups <- levels(dt$group)
cH <- contr.helmert(groups) # pre-defined four contrasts
colnames(cH) <- paste0("Ctras", 1:4)
summary_ctras_tbl <- cbind(summary_tbl, cH)
kable(summary_ctras_tbl)
group N Mean SD shapiro.test.p.values Ctras1 Ctras2 Ctras3 Ctras4
g1 g1 28 4.2500 3.15054 0.07759 -1 -1 -1 -1
g2 g2 28 2.7589 2.19478 0.07605 1 -1 -1 -1
g3 g3 28 3.5446 2.86506 0.00623 0 2 -1 -1
g4 g4 28 3.8568 0.58325 0.03023 0 0 3 -1
g5 g5 28 2.0243 1.30911 0.06147 0 0 0 4

Orthogonal contrast matrix

apply(cH, 2, sum)
Ctras1 Ctras2 Ctras3 Ctras4 
     0      0      0      0 
crossprod(cH) # diagonal -- columns are orthogonal
       Ctras1 Ctras2 Ctras3 Ctras4
Ctras1      2      0      0      0
Ctras2      0      6      0      0
Ctras3      0      0     12      0
Ctras4      0      0      0     20
summary(aov(score ~ group, dt))
             Df Sum Sq Mean Sq F value Pr(>F)   
group         4     89    22.3    4.47  0.002 **
Residuals   135    675     5.0                  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Planed Contrasts and Coding Schema

  • The relationship between planned contrasts in ANOVA and coding in regression lies in how categorical variables are represented and interpreted in statistical models.

  • Both approaches aim to test specific hypotheses about group differences, but their implementation varies based on the framework

    • ANOVA focuses on partitioning variance,
    • while regression interprets categorical predictors through coding schemes.

ANOVA: t-value formula for Defined Contrast Matrix

\(t = \frac{C}{\sqrt{MSE \sum \frac{c_i^2}{n_i}}}\)

Sum_C2_n <- colSums(cH^2 / summary_tbl$N)
C <- crossprod(summary_tbl$Mean, cH)
MSE <- 5.0
t <- as.numeric(C / sqrt(MSE * Sum_C2_n))
t
[1] -2.495040  0.077632  0.694597 -3.340639
tibble(
  t_value = t,
  p_value = pt(t, df = 135) ## p-values
)
# A tibble: 4 × 2
  t_value  p_value
    <dbl>    <dbl>
1 -2.50   0.00690 
2  0.0776 0.531   
3  0.695  0.756   
4 -3.34   0.000541

  • g1 vs. g2: We reject the null and determine that the mean of the Education is different from the mean of Engineering in their growth mindset scores (p = 0.531).

  • \(\frac{g1+g2}{2}\) vs. g3: We retain the null and determine that the mean of the Chemistry is not significant different from the mean of Education and Engineering in their growth mindset scores (p = 0.531).

Helmert Contrast

Remember that Planned Contrast: g1 vs. g2 from Helmert Contrast:

  • t-value: -2.495
  • p-value: 0.0069
  • df: 134
contrasts(dt$group) <- "contr.helmert"
fit_helmert <- lm(score ~ group, dt)
contr.helmert(levels(dt$group))
   [,1] [,2] [,3] [,4]
g1   -1   -1   -1   -1
g2    1   -1   -1   -1
g3    0    2   -1   -1
g4    0    0    3   -1
g5    0    0    0    4
summary(fit_helmert)$coefficients |> round(3)
            Estimate Std. Error t value Pr(>|t|)
(Intercept)    3.287      0.189  17.391    0.000
group1        -0.746      0.299  -2.495    0.014
group2         0.013      0.173   0.078    0.938
group3         0.085      0.122   0.695    0.489
group4        -0.316      0.095  -3.340    0.001

Planned Contrast Connected to Linear Regression

  • Planned contrast can be done using linear regression + contrasts

  • Let’s look at the default contrasts plan: treatment contrasts == dummy coding

## treatment contrast matrix 
attributes(C(dt$group, treatment, 4))$contrasts
   g2 g3 g4 g5
g1  0  0  0  0
g2  1  0  0  0
g3  0  1  0  0
g4  0  0  1  0
g5  0  0  0  1
## sum contrast matrix 
attributes(C(dt$group, sum, 4))$contrasts
   [,1] [,2] [,3] [,4]
g1    1    0    0    0
g2    0    1    0    0
g3    0    0    1    0
g4    0    0    0    1
g5   -1   -1   -1   -1
attributes(C(dt$group, helmert, 4))$contrasts
   [,1] [,2] [,3] [,4]
g1   -1   -1   -1   -1
g2    1   -1   -1   -1
g3    0    2   -1   -1
g4    0    0    3   -1
g5    0    0    0    4
crossprod(attributes(C(dt$group, treatment, 4))$contrasts)
   g2 g3 g4 g5
g2  1  0  0  0
g3  0  1  0  0
g4  0  0  1  0
g5  0  0  0  1

Treatment Contrasts

  • For treatment contrasts, four dummy variables are created to compared:

    • G1 (ref) vs. G2
    • G1 (ref) vs. G3
    • G1 (ref) vs. G4
    • G1 (ref) vs. G5
  • Intercept: G1’s mean
  • group2: G2 vs. G1
  • group3: G3 vs. G1
  • group4: G4 vs. G1
  • group5: G5 vs. G1
library(multcomp)
contrasts(dt$group) <- "contr.treatment"
fit <- lm(score ~ group, dt)
unique(cbind(model.matrix(fit), group = dt$group))
    (Intercept) groupg2 groupg3 groupg4 groupg5 group
1             1       0       0       0       0     1
29            1       1       0       0       0     2
57            1       0       1       0       0     3
85            1       0       0       1       0     4
113           1       0       0       0       1     5
summary(fit)$coefficients
            Estimate Std. Error t value   Pr(>|t|)
(Intercept)  4.25000    0.42262 10.0562 4.2275e-18
groupg2     -1.49107    0.59768 -2.4948 1.3810e-02
groupg3     -0.70536    0.59768 -1.1802 2.4001e-01
groupg4     -0.39321    0.59768 -0.6579 5.1172e-01
groupg5     -2.22571    0.59768 -3.7239 2.8718e-04

Sum Contrasts

  • Another type of coding is effect coding. In R, the corresponding contrast type are the so-called sum contrasts.

  • A detailed post about sum contrasts can be found here

  • With sum contrasts the reference level is in fact the grand mean.

    • \(\frac{g1+g2+g3+g4+g5}{5}\) vs. g1/g2/g3/g4: the difference between mean score of g1 with grand mean across all five groups
contrasts(dt$group) <- "contr.sum"
fit2 <- lm(score ~ group, dt)
contr.sum(levels(dt$group))
   [,1] [,2] [,3] [,4]
g1    1    0    0    0
g2    0    1    0    0
g3    0    0    1    0
g4    0    0    0    1
g5   -1   -1   -1   -1
summary(fit2)$coefficients
            Estimate Std. Error  t value   Pr(>|t|)
(Intercept)  3.28693    0.18900 17.39087 2.8188e-36
group1       0.96307    0.37801  2.54777 1.1962e-02
group2      -0.52800    0.37801 -1.39680 1.6476e-01
group3       0.25771    0.37801  0.68177 4.9655e-01
group4       0.56986    0.37801  1.50753 1.3401e-01
mean(dt$score) # (Intercept) grand mean
[1] 3.2869
tibble(
  Label = paste0("group", 1:4),
  Estimate = summary_tbl$Mean[1:4] - mean(dt$score) 
)
# A tibble: 4 × 2
  Label  Estimate
  <chr>     <dbl>
1 group1    0.963
2 group2   -0.528
3 group3    0.258
4 group4    0.570

Effect Coding (Deviation Coding)

  • In modern statistics, Regression-style coding is statistically equivalent as ANOVA-style contrast matrix.
    • Equivalent to ANOVA-style contrasts. (we will use this in R to reproduce ANOVA-style contrast matrix)
  • Compares each level to the grand mean.

Note

Effect coding is a method of encoding categorical variables in regression models, similar to dummy coding, but with a different interpretation of the resulting coefficients. It is particularly useful when researchers want to compare each level of a categorical variable to the overall mean rather than to a specific reference category.

1. Definition and Representation

In effect coding, categorical variables are transformed into numerical variables, typically using values of -1, 0, and 1. The key difference from dummy coding is that the reference category is represented by -1 instead of 0, and the coefficients indicate deviations from the grand mean.

For a categorical variable with k levels, effect coding requires k-1 coded variables. If we have a categorical variable X with three levels: \(A, B, C\), the effect coding scheme could be:

Category \(X_1\) \(X_2\)
A 1 0
B 0 1
C (reference) -1 -1

The last category (\(C\)) is the reference group, coded as -1 for all indicator variables.

2. Interpretation of Coefficients

When effect coding is used in a regression model:

\[ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \epsilon \]

  • \(X_1\) and \(X_2\) are coded varaibles. They have no much meaning, but their coefficients are important
  • \(\beta_0\) represents the grand mean of \(Y\) across all categories.
  • \(\beta_1\) and \(\beta_2\) represent the deviation of categories \(A\) and \(B\) from the grand mean.
  • The reference group (\(C\)) does not have a separate coefficient; instead, its deviation can be inferred as \(-(\beta_1 + \beta_2)\).
Code 
library(ggplot2)
# Create a data frame for text labels
text_data <- data.frame(
  x = rep(0.25, 3),  # Repeating the same x-coordinate
  y = c(0.3, 0.7, 0.9),  # Different y-coordinates
  label = c("C: beta[0] - beta[1] - beta[2]", 
            "A: beta[0] + 1*'×'*beta[1] + 0*'×'*beta[2]", 
            "B: beta[0] + 0*'×'*beta[1] + 1*'×'*beta[2]")  # Labels
)

# Create an empty ggplot with defined limits
ggplot() +
  geom_text(data = text_data, aes(x = x, y = y, label = label), parse = TRUE, size = 11) +
  # Add a vertical line at x = 0.5
  # geom_vline(xintercept = 0.5, color = "blue", linetype = "dashed", linewidth = 1) +
  # Add two horizontal lines at y = 0.3 and y = 0.7
  geom_hline(yintercept = c(0.35, 0.75, 0.95), color = "red", linetype = "solid", linewidth = 1) +
  geom_hline(yintercept = 0.5, color = "grey", linetype = "solid", linewidth = 1) +
  geom_text(aes(x = .25, y = .45, label = "grand mean of Y"), color = "grey", size = 11) +
  # Set axis limits
  xlim(0, 1) + ylim(0, 1) +
  labs(y = "Y", x = "") +
  # Theme adjustments
  theme_minimal() +
  theme(text = element_text(size = 20))

3. Comparison to Dummy Coding

  • Dummy Coding: Compares each category to a specific reference category (e.g., comparing A and B to C).
Category \(X_1\) \(X_2\)
A 1 0
B 0 1
C (reference) 0 0
  • Effect Coding: Compares each category to the grand mean rather than a single reference category.

4. Use Cases

Effect coding is beneficial when:

  • There is no natural baseline category, and comparisons to the overall mean are more meaningful.
  • Researchers want to maintain sum-to-zero constraints for categorical variables in linear models.
  • In ANOVA-style analyses, where main effects and interaction effects are tested under an equal-weight assumption.

5. Implementation in R

Effect coding can be set in R using the contr.sum function:

X <- factor(c("A", "B", "C"))
contrasts(X) <- contr.sum(3) # set up effect coding in R
model <- lm(Y ~ X, data = mydata) # use linear regression to mimic ANOVA-style results
summary(model)

Self-defined Contrast

  • Extended Example 2 : Assume now that I think the average of the STEM groups is different than the average of the non-STEM groups

Method 1: Calculation by Hand

  group  N   Mean      SD shapiro.test.p.values Contrasts
1    g1 28 4.2500 3.15054             0.0775874   0.50000
2    g2 28 2.7589 2.19478             0.0760542  -0.33333
3    g3 28 3.5446 2.86506             0.0062253   0.50000
4    g4 28 3.8568 0.58325             0.0302312  -0.33333
5    g5 28 2.0243 1.30911             0.0614743  -0.33333

\[ H_0: \frac{\mu_{Engineering}+\mu_{Chemistry}}{2} = \frac{\mu_{Education}+\mu_{PoliSci}+\mu_{Psychology}}{3} \]

weighted mean difference:

\[ C = c_1\mu_{Eng}+c_2\mu_{Edu}+c_3\mu_{Chem}+c_4\mu_{PoliSci}+c_5\mu_{Psych}\\ = \frac{1}{2}*4.25+(-\frac13)*2.75+(\frac12)*3.54+(-\frac13)*3.85+(-\frac13)*2.02\\ = 1.0173 \]

(C <- sum(summary_tbl_ext$Contrasts*summary_tbl_ext$Mean))
[1] 1.0173

\[ \sum\frac{c^2}{n} = \frac{(\frac12)^2}{28}+\frac{(-\frac13)^2}{28}+\frac{(\frac12)^2}{28}+\frac{(-\frac13)^2}{28}+\frac{(-\frac13)^2}{28} \]

(Sum_C2_n <- sum(summary_tbl_ext$Contrasts^2 / summary_tbl$N))
[1] 0.029762
(MSE = sum((residuals(aov(score ~ group, dt)))^2) / (nrow(dt) - 5))
[1] 5.0011
(t = as.numeric(C / sqrt(MSE * Sum_C2_n)))
[1] 2.6369

\[ t = \frac{C}{\sqrt{MSE*\sum\frac{c^2}{n} }} = \frac{1.0173}{\sqrt{5.0011*0.029762}}=2.6368 \]

pt(t, df = 135, lower.tail = FALSE) * 2
[1] 0.0093476

Method 2: Linear Regression Contrasts by R

# set first contrast
contrasts(dt$group) <- matrix(
  c(1/2, -1/3, 1/2, -1/3, -1/3)
)
fit_extended <- lm(score ~ group, dt)
unique(model.matrix(fit_extended))[, 1:2]
    (Intercept)   group1
1             1  0.50000
29            1 -0.33333
57            1  0.50000
85            1 -0.33333
113           1 -0.33333
summary(fit_extended)$coefficient |> round(3)
            Estimate Std. Error t value Pr(>|t|)
(Intercept)    3.287      0.189  17.391    0.000
group1         1.221      0.463   2.637    0.009
group2        -0.518      0.423  -1.227    0.222
group3         0.948      0.423   2.243    0.027
group4        -0.885      0.423  -2.093    0.038

3 Effect Sizes

What is Effect Sizes

  • Effect size measures the magnitude of an effect beyond statistical significance.
    • Put simply: a p-value is partially dependent on sample size and does not give us any insight into the strength of the relationship
    • Lower p-value → just increase sample size
  • Provides context for interpreting practical significance.
    • In scientific experiments, it is often useful to know not only whether an experiment has a statistically significant effect, but also the size (magnitude) of any observed effects.
  • Common measures: Eta squared (\(\eta^2\)), Omega squared (\(\omega^2\)), Cohen’s d.

Note

Many psychology journals require the reporting of effect sizes

Eta Squared

  • \(\eta^2\): Proportion of total variance explained by the independent variable.
  • Formula: \(\eta^2 = \frac{SS_{Model}}{SS_{Total}}\)
  • Interpretation:
    • Small: 0.01, Medium: 0.06, Large: 0.14
(F_table <- as.data.frame(anova(fit)))
           Df  Sum Sq Mean Sq F value    Pr(>F)
group       4  89.368 22.3420  4.4674 0.0020173
Residuals 135 675.149  5.0011      NA        NA
(eta_2 <- F_table$`Sum Sq`[1] / sum(F_table$`Sum Sq`))
[1] 0.11689

Interpretation: 11.69% of variance in the DV is due to group differences.

Drawbacks of Eta Squared

  1. As you add more variables to the model, the proportion explained by any one variable will automatically decrease.
    • This makes it hard to compare the effect of a single variable in different studies.
    • Partial Eta Squared solves this problem. There, the denominator is not the total variation in Y, but the unexplained variation in Y plus the variation explained just by that IV.
      • Any variation explained by other IVs is removed from the denominator.
    • In a one-way ANOVA, Eta Squared and Partial Eta Squared will be equal, but this isn’t true in models with more than one independent variable (factorial ANOVA).
  2. Eta Squared is a biased measure of population variance explained (although it is accurate for the sample).
    • It always overestimates it. This bias gets very small as sample size increases, but for small samples an unbiased effect size measure is Omega Squared.

Omega Square

  • Omega Squared (\(\omega^2\)) has the same basic interpretation but uses unbiased measures of the variance components.
    • Because it is an unbiased estimate of population variances, Omega Squared is always smaller than Eta Squared. ## Omega Squared (\(\omega^2\))
  • Unbiased estimate of effect size, preferred for small samples.
  • Formula: \(\omega^2 = \frac{SS_{Model} - df_{Model} \cdot MSE}{SS_{Total} + MSE}\)
  • Interpretation follows \(\eta^2\) scale but slightly smaller values.
F_table
           Df  Sum Sq Mean Sq F value    Pr(>F)
group       4  89.368 22.3420  4.4674 0.0020173
Residuals 135 675.149  5.0011      NA        NA
1attach(F_table)
2(Omega_2 <- (`Sum Sq`[1] - Df[1] * MSE) / (sum(`Sum Sq`) + MSE))
detach(F_table)
1
Attach data set so that you can directly call the columns without “$”
2
The formula of Omega square 
[1] 0.090139

Effect Size for Planned Contrasts

  • Correlation-based effect size: \(r = \sqrt{\frac{t^2}{t^2 + df}} = \sqrt{\frac{F}{F + df}}\)
  • Example: For \(t = 2.49, df = 135\): \(r = \sqrt{\frac{2.49^2}{2.49^2 + 135}} = 0.21\)
    • Small to moderate effect.
(coef_tbl <- as.data.frame(summary(fit)$coefficients))
            Estimate Std. Error t value   Pr(>|t|)
(Intercept)  4.25000    0.42262 10.0562 4.2275e-18
groupg2     -1.49107    0.59768 -2.4948 1.3810e-02
groupg3     -0.70536    0.59768 -1.1802 2.4001e-01
groupg4     -0.39321    0.59768 -0.6579 5.1172e-01
groupg5     -2.22571    0.59768 -3.7239 2.8718e-04
attach(coef_tbl)
round(sqrt(`t value`^2 / (`t value`^2 + 135)), 3)
detach(coef_tbl)
[1] 0.654 0.210 0.101 0.057 0.305
  • Shows a small to moderate positive relationship between g1 with g5.

Cohen’s d and Hedges’ g

  • Used for simple mean comparisons.
  • Cohen’s d formula: \(d = \frac{M_1 - M_2}{SD_{pooled}}\)
  • Hedges’ g corrects for small sample bias.
  • Guidelines:
    • Small: 0.2, Medium: 0.5, Large: 0.8

Guideline of Effect Size

  • For our example: there is a significant effect of academic program on growth mindset scores (F(4,135)=4.47).
  • Academic program explains 11.69% of variance in growth mindset scores. This is a large medium to large effect (\(\eta^1\) = 0.1169).

Summary

  • Planned contrasts allow hypothesis-driven mean comparisons.
  • Orthogonal contrasts maintain Type I error control.
  • Effect sizes help interpret the importance of results.
  • Combining planned contrasts with effect size measures enhances statistical analysis.