Educational Statistics and Research Methods (ESRM) Program*

University of Arkansas

Published

September 16, 2024

1 Today’s Class

Review last class - Random variables and their distribution

Review Homework System

Maximum Likelihood Estimation

The basics

ML-based Test (Likelihood ratio test, Wald test, Information criteria)

MLEs for GLMs

2 More Details about Homework Grade

You should be able to check your current grade for homework here.

You need your 4-digit unique code to check your homework.

Homework 0: 20 points; Homework 1: 21 points

I adjusted everyone’s scores based on how many errors you have in your homework that are “key parts”

If the position within vcov() is incorrect (i.e., answer is vcov(model)[1, 1], while the response is vcov(model)[1, 2]), you will lose 3 points

If you found your scores incorrect or have any suggestions, feel free to talk to me and I am willing to help at any time.

3 Today’s Example Data #1

To use the class example data, type in following R code: pak::pak("JihongZ/ESRM64503", upgrade = TRUE) or devtools::install_github("JihongZ/ESRM64503", upgrade = "always") to upgrade ESRM64503 package

3.1 Data Information

Purpose: imagine an employer is looking to hire employees for a job where IQ is important

We will use 5 observations so as to show the math behind the estimation calculations

The employer collects two variables:

Predictor: IQ scores (labelled as iq)

Outcome: Job performance (labelled as perf)

3.2 Descriptive Statistics

# pak::pak("JihongZ/ESRM64503", upgrade = TRUE)library(ESRM64503)library(kableExtra)library(tidyverse)#data for class exampledat <- dataIQ |>mutate(ID =1:5) |>rename(IQ = iq, Performance = perf) |>relocate(ID)show_table(dat)

dat |>select(IQ, Performance) |>cov() |>as.data.frame() |>rownames_to_column("Covariance Matrix") |>show_table()

Covariance Matrix

IQ

Performance

IQ

5.3

5.1

Performance

5.1

5.2

4 How Estimation Works (More or Less)

We need models to test the hypothesis, while models need estimation method to provide estimates of effect size.

Most estimation routines do one of three things:

Minimize Something: Typically found with names that have “least” in the title. Forms of least squares include “Generalized”, “Ordinary”, “Weighted”, “Diagonally Weighted”, “WLSMV”, and “Iteratively Reweighted.” Typically the estimator of last resort…

Maximize Something: Typically found with names that have “maximum” in the title. Forms include “Maximum likelihood”, “ML”, “Residual Maximum Likelihood” (REML), “Robust ML”. Typically the gold standard of estimators

Use Simulation to Sample from Something: more recent advances in simulation use resampling techniques. Names include “Bayesian Markov Chain Monte Carlo”, “Gibbs Sampling”, “Metropolis Hastings”, “Metropolis Algorithm”, and “Monte Carlo”. Used for complex models where ML is not available or for methods where prior values are needed.

Note

Loss function in Machine Learning belong to 1st type

lm use QR algorithm, which is one of methods for Ordinary Least Square (OLS) Estimation

Question: Maximum Likelihood Estimation belongs to 2nd type

Note: the estimates of regression coefficients provided by lm are using the OLS estimation method.

summary(lm(perf ~ iq, data = dataIQ))

Call:lm(formula = perf ~ iq, data = dataIQ)Residuals: 1 2 3 4 5 -0.4906 0.5472 0.6226 -0.2642 -0.4151 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -97.2830 15.5176 -6.269 0.00819 **iq 0.9623 0.1356 7.095 0.00576 **---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Residual standard error: 0.6244 on 3 degrees of freedomMultiple R-squared: 0.9438, Adjusted R-squared: 0.925 F-statistic: 50.34 on 1 and 3 DF, p-value: 0.005759

5 An Brief Introduction to Maximum Likelihood Estimation (MLE)

MLE has several good statistical properties that make it the mostly commonly used in multivariate statistics:

Asymptotic Consistency: as the sample size increases, the estimator converges in probability to parameters’ true values

Asymptotic Normality: as the sample size increases, the distribution of the estimator is normal (note: variance is given by “information” matrix)

Efficiency: No other estimator will have a smaller standard error

Terminology

Estimator: the algorithm that can get the estimates of parameters

Asymptotic: the properties that will occur when the sample size is sufficiently large

ML-based estimators: other variants of maximum likelihood like robust maximum likelihood, full information maximum likelihood

6 Maximum Likelihood: Estimates Based on Statistical Distributions

Maximum likelihood estimates come from statistical distributions - assumed distributions of data

We will begin today with the univariate normal distribution but quickly move to other distributions

For a single random variable x, the univariate normal distribution is:

Provides the height of the curve for a value of x, \mu_x, and \sigma^2

Last week we pretended we knew \mu_x and \sigma_x^2

Today we will only know values of x and “guess” the parameters \mu_x and \sigma^2_x

7 Key Idea: Constructing a Likelihood Function

A likelihood function provides a value of a likelihood for a set of statistical parameters given observed data (denoted as L(\mu_x, \sigma^2_x|x))

If we observe the data (known) but do not know the mean and/or variance (unkown), then we call this the sample likelihood function

The process of finding the values that maximizing the sample likelihood function is named Maximum Likelihood function

Likelihood functions is a probability density function (PDFs) of random variables that input observed data values

Density functions are provided for each observation individually

Each observation has its likelihood function.

Mark this: We can get the “whole” likelihood of observed data by multiple all case-level likelihood.

Rather than providing the height of the curve of any value of x, it provides the likelihood for any possible values of parameters (i.e., \mu_x and/or \sigma^2_x)

Goal of maximizing the sample likelihood function is to find optimal values of parameters

Assumption:

All observed random variables follow certain probability density functions (i.e., x follows normal dis. and y follows Bernoulli dist.).

The sample likelihood can be thought of as a joint distribution of all the observations, simultaneously

In univariate statistics, observations are considered independent, so the joint likelihood for the sample is constructed through a product

To demonstrate, let’s consider the likelihood function for one observation that is used to estimate the mean and the variance of x

8 Example: One-Observation Likelihood function

Let’s assume:

We just observed the first value of IQ (x = 112)

Our initial guess is “the IQ comes from a normal distribution x \sim N(100, 5)”

Our goal is to test whether this guess needed to be adjusted

Step 1: For this one observation, the likelihood function takes its assumed distribution and uses its PDF:

If you want to know the value of L(x_1), you can easily use R function dnorm(112, 100, sqrt(5)) , which gives your 9.94\times 10^{-8}.

Because it is too small, typical we use log transformation of likelihood, which is called log likelihood. The value is -16.12

9 More observations: Multiplication of Likelihood

L(x_1 = 112|\mu_x = 100, \sigma^2_x = 5) can be interpreted as the Likelihood of first observation value being 112 given the underlying distribution is a normal distribution with mean as 100 and variance as 5.

Step 2: Then, for following observations, we can calculate their likelihood just similar to the first observation

Now, we have L(x_2), L(x_3), …, L(x_N)

Step 3: Now, we multiple them together L(X|\mu_x=100, \sigma^2_x=5) =L(x_1)L(x_2)L(x_3)\cdots L(x_N), which represents the Likelihood of observed data existing given the underlying distribution is a normal distribution with mean as 100 and variance as 5.

Step 4: Again, we log transformed the likelihood to make the scale more easy to read: LL(X|\mu_x=100, \sigma^2_x=5)

Step 5: Finally, we change the value of parameters \mu_x and \sigma^2_x and calculate their LL (i.e., LL(X|\mu_x=101, \sigma^2_x=6))

Eventually, we can get a 3-D density plot with x-axis and y-axis are varied values of \mu_x and \sigma^2_x; z-axis is their corresponding log-likelihood

The set of \mu_x and \sigma^2_x which has the highest log-likelihood are the estimates of MLE

10 Visualization of MLE

The values of \mu_x and \sigma_x^2 that maximize L(\mu_x, \sigma^2_x|x) is \hat\mu_x = 114.4 and \hat{\sigma^2_x} = 4.2. We also know the mean of IQ is 114.2 and the variance of IQ is 5.3. Why the mean is same to estimate mu but variance is different? Answer: MLE is a biased estimator

⌘+C

library(plotly) cal_LL <-function(x, mu, variance) {sum(log(sapply(x, \(x) dnorm(x, mean = mu, sd =sqrt(variance))))) }x <-seq(80, 150, .1)y <-seq(3, 8, .1)x_y <-expand.grid(x, y)colnames(x_y) =c("mean", "variance")dat2 <- x_ydat2 <- dat2 |>rowwise() |>mutate(LL =cal_LL(x = dat$IQ, mu = mean, variance = variance)) |>ungroup() |>mutate(highlight =ifelse(LL ==max(LL), TRUE, FALSE))plot_ly(data = dat2, x =~mean, y =~variance, z =~LL, colors =c("royalblue", "tomato"), color =~highlight, stroke ="white", alpha_stroke = .5,alpha = .9, type ="scatter3d",width =1600, height =800)