Lecture 02: Hypothesis Testing

Experimental Design in Education

Jihong Zhang*, Ph.D

Educational Statistics and Research Methods (ESRM) Program*

University of Arkansas

2025-08-18

Presentation Outline

  • Types of Statistics
    • Descriptive: Summarize data (central tendency, variability, shape)
    • Inferential: Make population inferences from samples
    • Predictive: Make predictions for new data
  • Hypothesis Testing Steps
    1. State H0 and HA
    2. Set α level
    3. Compute test statistics
    4. Conduct test and make decision
  • ANOVA Fundamentals
    • One dependent variable (DV), one independent variable (IV) with multiple levels
    • Between-subjects and within-subjects designs
    • Interaction effects
  • Test Components
    • Error types (Type I and Type II)
    • Variance decomposition (SSTotal, SSBetween, SSWithin)
    • F-statistics and critical values
  • Examples
    • Political attitudes study
    • Sleep and academic performance study
  • Decision Making
    • Compare Fobserved vs Fcritical
    • Compare p-value vs α
    • Interpret at (1−α) confidence level

Types of Statistics

Statistics can be classified by purpose:

  1. Descriptive Statistics
  2. Inferential Statistics
  3. Predictive Statistics

1. Descriptive Statistics

  • Definition: Describes and summarizes the collected data using numbers/values
    • Central tendency: mean, median, mode
    • Variability: range, interquartile range (IQR), variance, standard deviation
    • Shape of distribution: skewness, kurtosis
# install.packages("moments")
moments::skewness(c(1:10, 100))
[1] 2.793716
moments::skewness(rnorm(100, 0, 1)) # should be close to 0
[1] 0.3594185
  • Examples of skewness with two graphs:

Examples of positive and negative skewness in distributions

set.seed(1234)
# Simulate a skewed normal distribution using beta distribution
neg_skewed_data <- rbeta(10000,5,2)
hist(neg_skewed_data, main = "Negative Skewed Distribution")

pos_skewed_data <- rbeta(10000,2,5)
hist(pos_skewed_data, main = "Positive Skewed Distribution")

library(ggplot2)
library(tidyr)
data.frame(
  neg = neg_skewed_data,
  pos = pos_skewed_data
  ) |>
  pivot_longer(c(neg, pos), names_to = "Type") |> 
  ggplot(aes(y = value, fill = Type)) +
  geom_boxplot() +
  scale_fill_discrete(labels = c("Negative Skewed", "Positive Skewed"), name = "")

2. Inferential Statistics

  • Definition: Uses probability theory to infer/estimate population characteristics from a sample using hypothesis testing
  • Visual representation shows:
    • Population → Sampling → Sample
    • Sample → Inference → Population
      • Sample is analyzed using descriptive statistics
      • Inferential statistics used to make conclusions about population

The relationship between population, sample, and inference

3. Predictive Statistics

  • Definition: Use observed data to produce the most accurate prediction possible for new data. Here, the primary goal is that the predicted values have the highest possible fidelity to the true value of the new data.

  • Example: A simple example would be for a book buyer to predict how many copies of a particular book should be shipped to their store for the next month.

Example of predictive statistics workflow

Which type of statistics to use

  1. How many houses burned in California wildfire in the first week?

    • Descriptive

  2. Which factor is most important causing the fires?

    • Inference

  3. How likely the California wildfire will not happen again in next 5 years?

    • Predictive

  4. How likely human will live on Mars?

    • Not statistics. Sci-Fi

  5. Which type of statistics is used by ChatGPT?

    ChatGPT uses predictive statistics for text generation

Statistical Hypothesis Testing Steps

Steps for Inferential Statistical Testing:

  1. State null hypothesis (H0) and alternative hypothesis (HA)
    • Null hypothesis must be some statement that is statistically testable.
  2. Set alpha α (type I error rate) to determine significance levels
    • rejection region vs. p-value
  3. Compute test statistics (i.e., F-statistics)
  4. Conduct hypothesis testing:
    • Compare test statistics: critical value vs. observed value
    • Compare alpha and p-value

ANOVA Introduction

  • ANOVA is one of the most frequently used statistical tool for inferential statistics in experimental design.
  • Settings for Analysis of Variance (ANOVA):
    • One dependent variable (DV), “Outcome”
    • One independent variable (IV) with multiple levels, “Group”
  • Example question: “Are there mean differences in SAT math scores (outcome) for different high school program types (group)?”
  • Course covers advanced ANOVA topics:
    • Group comparisons (Group A vs. B vs. C)
    • Model comparisons
    • Between/within-subject design
    • Interaction effects

Types of ANOVA: Key Differences

  • One-Way ANOVA

    • Purpose: Tests one factor with three or more levels on a continuous outcome.

    • Use Case: Comparing means across multiple groups (e.g., diet types on weight loss).

  • Two-Way ANOVA

    • Purpose: Examines two factors and their interaction on a continuous outcome.

    • Use Case: Studying effects of diet and exercise on weight loss.

  • Repeated Measures ANOVA

    • Purpose: Tests the same subjects under different conditions or time points.

    • Use Case: Longitudinal studies measuring the same outcome over time (e.g., cognitive tests after varying sleep durations).

  • Mixed-Design ANOVA

    • Purpose: Combines between-subjects and within-subjects factors in one analysis.

    • Use Case: Evaluating treatment effects over time with control and experimental groups.

  • Multivariate Analysis of Variance (MANOVA)

    • Purpose: Assesses multiple continuous outcomes (dependent variables) influenced by independent variables.

    • Use Case: Impact of psychological interventions on anxiety, stress, and self-esteem.

1 Example 1: Political Study on Tax Reform Attitudes

Background

  • A political scientist studies tax reform attitudes across political groups:
    • Groups: Democrats (n=4), Republicans (n=5), Independents (n=8)
    • Outcome measure: Attitude scores (higher score = greater concern for tax reform)
    • Analysis: Conducted at α=.05
    • Variables:
      • party: Political affiliation
      • scores: Attitude scores for survey respondents
## Install the package ESRM64103 from GitHub
remotes::install_github("JihongZ/ESRM64103")
library(ESRM64103)
library(dplyr)
exp_political_attitude
         party scores
1     Democrat      4
2     Democrat      3
3     Democrat      5
4     Democrat      4
5     Democrat      4
6   Republican      6
7   Republican      5
8   Republican      3
9   Republican      7
10  Republican      4
11  Republican      5
12 Independent      8
13 Independent      9
14 Independent      8
15 Independent      7
16 Independent      8

Descriptive Statistics: Summary Statistics

  • Calculate mean, standard deviation, and variance for each political group

  • Grand mean across all groups: 5.625

# Grand mean
mean(exp_political_attitude$scores)
[1] 5.625
exp_political_attitude$party <- factor(exp_political_attitude$party, levels = c("Democrat", "Republican", "Independent"))
mean_byGroup <- exp_political_attitude |> 
  group_by(party) |> 
  summarise(Mean = mean(scores),
            SD = round(sd(scores), 2),
            Vars = round(var(scores), 2),
            N = n())
mean_byGroup
# A tibble: 3 × 5
  party        Mean    SD  Vars     N
  <fct>       <dbl> <dbl> <dbl> <int>
1 Democrat        4  0.71   0.5     5
2 Republican      5  1.41   2       6
3 Independent     8  0.71   0.5     5

Descriptive Statistics: Bar Plot

library(ggplot2)
ggplot(data = mean_byGroup) +
  geom_bar(mapping = aes(x = party, y = Mean, fill = party), stat = "identity", width = .5) +
  geom_label(aes(x = party, y = Mean, label = Mean), nudge_y = .3) +
  labs(title = "Attitudes Toward the Tax Return") +
  theme(text = element_text(size = 15))

Steps of ANOVA

  1. State the null hypothesis and alternative hypothesis:

    • H0: ˉXdem = ˉXrep = ˉXind
    • HA: At least two groups are significantly different
    • Question: Why not testing ¯SDdem = ¯SDrep = ¯SDind?
    • Answer: You definitely can in statistics. Variances homogeneity.

  2. Set the significant alpha = 0.05

  3. Quick Review of F-statistics:

    Fobs=SSb/dfbSSw/dfw

    • Degrees of freedom: dfb = 3 (groups) - 1 = 2, dfw = 16 (samples) - 3 (groups) = 13

    • Between-group sum of squares: SSb=g∑j=1nj(ˉYj−ˉY)2=43.75 where nj is group sample size, ˉYj is group mean, and ˉY is the grand mean.

    • Within-group sum of squares: SSw=3∑j=1nj∑i=1(Yij−ˉYj)2=14.00 where Yij is individual i’s score in group j.

    GrandMean <- mean(exp_political_attitude$scores)
## Between-group Sum of Squares
sum(mean_byGroup$N * (mean_byGroup$Mean - GrandMean)^2)
## Within-group Sum of Squares
SSw_dt <- exp_political_attitude |> 
  group_by(party) |> 
  mutate(GroupMean = mean(scores),
         Diff_sq = (scores - GroupMean)^2) 
sum(SSw_dt$Diff_sq)
    • Fcritical (df_num = 2, df_deno = 13) = 3.81
    • Fobserved = 20.31
    anova_model <- lm(scores ~ party, data = exp_political_attitude)
anova(anova_model)
    Analysis of Variance Table

Response: scores
          Df Sum Sq Mean Sq F value    Pr(>F)    
party      2  43.75 21.8750  20.312 9.994e-05 ***
Residuals 13  14.00  1.0769                      
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Results show rejection of H0 (Fobs > Fcritical)

Step 1: State the null hypothesis and alternative hypothesis

  1. Formulate the null hypothesis (H0) and the alternative hypothesis (HA)
    • Prior to any statistical tests, start with a working hypothesis based on an initial guess about the phenomenon.
    • Example: Investigating whether political groups affect their attitudes.
      • Research question: “Is there a variance in attitude score among different groups?”
      • Hypothesis: “Different political groups will show varied attitudes.”
    • Operational Definitions:
      • Null hypothesis (H0): No observed difference or effect (“Something is something”).
        • Group A’s mean - Group B’s mean = 0
      • Alternative hypothesis (HA): Noticeable difference or effect, contrary to H0 (“Something is not something”)
    • The adequacy of the data will dictate if H0 can be confidently rejected.

Step 2: Rejection region (alpha)

F-statistic has two degree of freedoms (df = 2). This is the density distribution of F-statistics for degree of freedoms as 2 and 13.

Code 
# Set degrees of freedom for the numerator and denominator
num_df <- 2  # Change this as per your specification
den_df <- 13  # Change this as per your specification

# Generate a sequence of F values
f_values <- seq(0, 8, length.out = 1000)

# Calculate the density of the F-distribution
f_density <- df(f_values, df1 = num_df, df2 = den_df)

# Create a data frame for plotting
data_to_plot <- data.frame(F_Values = f_values, Density = f_density)
data_to_plot$Reject05 <- data_to_plot$F_Values > 3.81
data_to_plot$Reject01 <- data_to_plot$F_Values > 6.70
# Plot the density using ggplot2
ggplot(data_to_plot) +
  geom_area(aes(x = F_Values, y = Density), fill = "grey", 
            data = filter(data_to_plot, !Reject05)) + # Draw the line
  geom_area(aes(x = F_Values, y = Density), fill = "yellow", 
            data = filter(data_to_plot, Reject05)) + # Draw the line
  geom_area(aes(x = F_Values, y = Density), fill = "tomato", 
            data = filter(data_to_plot, Reject01)) + # Draw the line
  geom_vline(xintercept = 3.81, linetype = "dashed", color = "red") +
  geom_label(label = "F_crit = 3.81 (alpha = .05)", x = 3.81, y = .5, color = "red") +
  geom_vline(xintercept = 6.70, linetype = "dashed", color = "royalblue") +
  geom_label(label = "F_crit = 6.70 (alpha = .01)", x = 6.70, y = .5, color = "royalblue") +
  ggtitle("Density of F-Distribution") +
  xlab("F values") +
  ylab("Density") +
  theme_classic()

  1. Set the alpha α (i.e., type I error rate)—rejection rate vs. p-value

    • Alpha determines several values for statistical hypothesis testing: the critical value of the test statistics, the rejection region, etc.

    • Large sample sizes typically use lower alpha levels: .01 or .001 (more restrictive rejection rate)

  2. When we conduct hypothesis testing, four possible outcomes can occur:

Type I & II Error
Reality
Decision H0 is true H0 is false
Fail to reject H0 Correct Decision

Error made.

Type II error (β).
Reject H0

Error made.

Type I error (α)

 Correct Decision (Power)

Step 3: Compute the test statistics

  • Investigate where the variability in the outcome comes from.

    • In this study: Do people’s attitude scores differ because of their political party affiliation?

    • When we have factors influencing the outcome, the total variability can be decomposed as follows:

Sources of variability: Total variance can be decomposed into between-group and within-group variance

F-statistics

  • Core idea: Comparing the variances between groups and within groups to ascertain if the means of different groups are significantly different from each other.

  • Logic: If the between-group variance (due to systematic differences caused by the independent variable) is significantly greater than the within-group variance (attributable to random error), the observed differences between group means are likely not due to chance.

  • F-statistics formula for one-way ANOVA:

    Fobs=SSbetween/dfbetweenSSwithin/dfwithin

    • Degrees of freedom: dfbetween = 3 (groups) - 1 = 2, dfwithin = 16 (samples) - 3 (groups) = 13
    • SSbetween = ∑nj(ˉYj−ˉY)2 = 43.75
      • Variability in the differences between groups (weighted by group sample size)
    • SSwithin = ∑3j=1∑nji=1(Yij−ˉYj)2 = 14.00; where Yij is individual i’s score in group j
      • Random error within groups—individuals differ in attitudes for unknown reasons

Step 4: Conduct a hypothesis testing

  • In addition to the comparison of the critical value and the observed value of the test statistics, we can also compare the alpha and the p-value:

F-distribution showing rejection regions for different alpha levels
  • We determine Fcrit by setting α value.
    • α = (acceptable) type I error rate = probability that we wrongly reject H0 when H0 is true
  • From the data, we obtain Fobs with p-value.
    • p-value = probability of datasets having F-statistics larger than Fobs
  • If the F statistic from the data (Fobs) is larger than Fcritical, then you are in the rejection region and can reject H0 and accept HA with (1−α) level of confidence.
  • If the p-value obtained from the ANOVA is less than α, then reject H0 and accept HA with (1−α) level of confidence.

Step 5: Results Report

A one-way ANOVA was conducted to compare the level of concern for tax reform among three political groups: Democrats, Republicans, and Independents. There was a significant effect of political affiliation on tax reform concern at the p<.001 level for the three conditions [F(2,13)=20.31, p<.001]. This result indicates significant differences in attitudes toward tax reform among the groups.

Note: Relationship Between P-values and Type I Error

  1. P-values: The probability of observing data as extreme as, or more extreme than, the data observed under the assumption that the null hypothesis is true.

    • The lower the p-value, the less likely we would see the observed data given the null hypothesis is true

    • Question: Given that we already have the observed data, does a lower p-value mean the null hypothesis is unlikely to be true?

    • Answer: No. P(observed data|H0=true)≠P(H0=true|observed data). P-values are often misconstrued as the probability that the null hypothesis is true given the observed data. However, this interpretation is incorrect.

  2. Type I error, also known as a “false positive,” occurs when the null hypothesis is incorrectly rejected when it is, in fact, true.

  3. The alpha level α set before conducting a test (commonly α=0.05) defines the cutoff point for the p-value below which the null hypothesis will be rejected.

    • A p-value less than the alpha level suggests a low probability that the observed data would occur if the null hypothesis were true. Consequently, rejecting the null hypothesis in this context implies there is a statistically significant difference likely not due to random chance.

Note: Limitations of p-values

Relying solely on p-values to reject the null hypothesis can be problematic for several reasons:

  • Binary Decision Making: The use of a threshold (e.g., α=0.05) to determine whether to reject the null hypothesis reduces the complexity of the data to a binary decision. This can oversimplify the interpretation and overlook nuances in the data.

    • Alternatives: Confidence intervals, Bayesian statistics (reporting posterior distributions)

  • Neglect of Effect Size: P-values do not convey the size or practical importance of an effect. A very small effect can produce a small p-value if the sample size is large enough, leading to rejection of the null hypothesis even when the effect may not be practically significant.

    • Solution: Report effect sizes that are independent of sample size

  • Probability of Extremes Under the Null: Since p-values quantify the extremeness of the observed data under the null hypothesis, they do not address whether similarly extreme data could also occur under alternative hypotheses. This can lead to an overemphasis on the null hypothesis and potentially disregard other plausible explanations for the data.

    • Solution: Explore theory, find alternative explanations, try varied models

2 Example 2: the Effect of Sleep on Academic Performance (Simulation)

Background

  • A study investigates the effect of different sleep durations on the academic performance of university students. Three groups are defined based on nightly sleep duration: Less than 6 hours, 6 to 8 hours, and more than 8 hours.

  • We can simulate the data

# Set seed for reproducibility
set.seed(42)

# Generate data for three sleep groups
less_than_6_hours <- rnorm(30, mean = 65, sd = 10)
six_to_eight_hours <- rnorm(50, mean = 75, sd = 8)
more_than_8_hours <- rnorm(20, mean = 78, sd = 7)

# Combine data into a single data frame
sleep_data <- data.frame(
  Sleep_Group = factor(c(rep("<6 hours", 30), rep("6-8 hours", 50), rep(">8 hours", 20))),
  Exam_Score = c(less_than_6_hours, six_to_eight_hours, more_than_8_hours)
)

# View the first few rows of the dataset
head(sleep_data)

Descriptive Statistics

  • Groups:

    • Less than 6 hours: 30 students

    • 6 to 8 hours: 50 students

    • More than 8 hours: 20 students

  • Performance Metric: Average exam scores out of 100.

    • Less than 6 hours: Mean = 65, SD = 10

    • 6 to 8 hours: Mean = 75, SD = 8

    • More than 8 hours: Mean = 78, SD = 7

Your Turn:

F-test

  • Analysis: One-way ANOVA was conducted to compare the average exam scores among the three groups.

  • Results: Fobserved = [Calculate from your analysis], p = [Report p-value]

Interpretation

  • Alpha Level: α=0.05

  • P-value Interpretation: Compare your p-value to alpha and interpret the result

  • Conclusion: Based on the results, what can you conclude about the effect of sleep duration on academic performance?

Homework 1

Due on next Tuesday 5PM.

ESRM 64503

1 / 33
Lecture 02: Hypothesis Testing Experimental Design in Education Jihong Zhang*, Ph.D Educational Statistics and Research Methods (ESRM) Program* University of Arkansas 2025-08-18

  1. Slides

  2. Tools

  3. Close
  • Lecture 02: Hypothesis Testing
  • Presentation Outline
  • Types of Statistics
  • 1. Descriptive Statistics
  • Examples of skewness...
  • set.seed(1234) #...
  • library(ggplot2)...
  • 2. Inferential Statistics
  • 3. Predictive Statistics
  • Which type of statistics to use
  • Statistical Hypothesis Testing Steps
  • ANOVA Introduction
  • Types of ANOVA: Key Differences
  • 1 Example 1: Political Study on Tax Reform Attitudes
  • Background
  • ## Install the package...
  • Descriptive Statistics: Summary Statistics
  • Descriptive Statistics: Bar Plot
  • Steps of ANOVA
  • Step 1: State the null hypothesis and alternative hypothesis
  • Step 2: Rejection region (alpha)
  • Set the alpha \(\alpha\)...
  • Step 3: Compute the test statistics
  • F-statistics
  • Step 4: Conduct a hypothesis testing
  • Step 5: Results Report
  • Note: Relationship Between P-values and Type I Error
  • Note: Limitations of p-values
  • 2 Example 2: the Effect of Sleep on Academic Performance (Simulation)
  • Background
  • Descriptive Statistics
  • Your Turn:
  • Homework 1
  • f Fullscreen
  • s Speaker View
  • o Slide Overview
  • e PDF Export Mode
  • r Scroll View Mode
  • b Toggle Chalkboard
  • c Toggle Notes Canvas
  • d Download Drawings
  • ? Keyboard Help