Lecture 01 (Continued): Statistics in the AI Era

What Changes, What Endures

Jihong Zhang, Ph.D.

Educational Statistics and Research Methods (ESRM) Program

University of Arkansas

2026-08-24

Presentation Outline

This is a working draft. The topics, examples, and relative emphasis will be revised later.

  1. What can AI do for you?
  2. Do you still need to learn programming?
  3. Do you still need to learn statistics?
  4. Why statistical thinking matters more than ever
  5. Responsible AI-assisted statistical practice
  6. Activity: Trust, but verify

Learning Objectives

By the end of this lecture, you will be able to:

  1. Identify tasks for which AI can support statistical learning and practice.
  2. Explain why programming literacy still matters when AI can generate code.
  3. Explain why statistical knowledge is necessary for evaluating AI-generated analyses.
  4. Use statistical thinking to question data, models, evidence, and conclusions.
  5. Evaluate AI-generated statistical claims and analyses critically.

Unit 1: What Can AI Do for You?

Opening Question

When AI can generate code, analyses, visualizations, and written interpretations, what does a statistician contribute?

  • Think independently for two minutes.
  • Discuss your answer with a partner.
  • Share one idea with the class.

Statistics, Machine Learning, and AI

  • Statistics focuses on learning from data while accounting for variability and uncertainty.
  • Machine learning emphasizes algorithms that learn patterns and make predictions from data.
  • Artificial intelligence is a broader collection of systems designed to perform tasks associated with human intelligence.

Discussion

Where do these areas overlap, and where do their goals differ?

What AI Can Help You Do

  • Explain statistical concepts in different ways
  • Generate, revise, and explain R code
  • Help clean and reshape data
  • Suggest statistical models and visualizations
  • Interpret error messages
  • Draft summaries, documentation, and reports
  • Serve as an always-available learning assistant

AI as an Assistant, Not an Oracle

  • AI can accelerate routine work and help us explore possible solutions.
  • AI can also produce incorrect code, fabricated information, and unsupported interpretations.
  • A confident answer is not necessarily a correct answer.
  • The analyst remains responsible for checking the work and defending the conclusion.

Important

AI lowers the cost of producing an analysis, but increases the value of judging an analysis.

Unit 2: Do You Still Need to Learn Programming?

The Short Answer: Yes, but the Goal Is Changing

  • You may spend less time memorizing syntax.

  • You still need enough programming knowledge to:

    • Read and understand generated code
    • Modify code for your own data and research question
    • Detect fabricated functions or packages
    • Debug errors and inspect intermediate results
    • Protect private or sensitive data
    • Make an analysis reproducible

Note

AI may help you write code, but you remain responsible for what the code does.

From Producing Syntax to Supervising a Workflow

  • Before AI: Write, run, debug, and interpret the code.
  • With AI: Specify, generate, inspect, test, challenge, revise, and interpret the code.
  • Programming knowledge helps you distinguish a useful draft from a convincing mistake.

Warning

Faster code does not guarantee a better research question, better data, or a valid conclusion.

Unit 3: Do You Still Need to Learn Statistics?

The Short Answer: Yes—Perhaps More Than Before

AI can calculate statistics and generate plausible explanations, but it should not independently decide:

  • Whether the research question is meaningful
  • Whether variables measure the intended constructs
  • Whether the data support the conclusion
  • Whether model assumptions are reasonable
  • Whether association is being mistaken for causation
  • Whether uncertainty is adequately represented
  • Whether the analysis is fair, ethical, and generalizable

Statistical Judgment Is Still Central

  • What is the research question?
  • How were the constructs measured?
  • How were the data generated or collected?
  • Which assumptions are reasonable?
  • What evidence would change our conclusion?
  • To which population or setting can we generalize?

Prediction Is Not Explanation

  • A model may predict accurately without explaining why an outcome occurs.
  • Association does not establish causation.
  • A plausible explanation produced by AI is not empirical evidence.
  • The purpose of the analysis should guide model choice and interpretation.

Uncertainty Does Not Disappear

  • Sampling variability
  • Measurement error
  • Missing data
  • Model uncertainty
  • Distribution shift
  • Human and algorithmic bias

Guiding Question

How should uncertainty be communicated when an AI system presents a single confident answer?

Unit 4: Why Statistical Thinking Matters More Than Ever

Six Questions for Statistical Thinking

  1. Question: What exactly are we trying to learn?
  2. Measurement: What do the variables actually represent?
  3. Data generation: Where did the data come from?
  4. Variation: How much uncertainty is present?
  5. Evidence: What conclusions do the data support?
  6. Validation: How do we know the result is trustworthy?

Task Execution and Statistical Judgment

Stage AI can help with Human judgment remains essential for
Research question Brainstorming and refinement Meaning, purpose, and context
Data preparation Generating and explaining code Data quality, measurement, and provenance
Modeling Suggesting models and producing syntax Appropriateness, assumptions, and design
Interpretation Drafting explanations Evidence, uncertainty, and causal limits
Communication Editing and formatting Accuracy, transparency, and accountability

AI Problems Are Often Multivariate

AI systems commonly work with many variables, features, outcomes, or representations simultaneously.

Multivariate concept Question it helps answer Connection to AI
Covariance and correlation Which variables move together? Redundancy and dependence among features
Dimension reduction Can many variables be summarized? Compact representations and embeddings
Latent variables What unobserved structure explains responses? Hidden representations and measurement models
Path models How might variables relate in a system? Structured relationships and causal hypotheses
Network models How are many elements conditionally connected? Complex systems and graphical representations

A Bridge to This Course

Throughout the semester, we will ask:

  1. What multivariate structure is present in the data?
  2. How can that structure be represented by a statistical model?
  3. What assumptions make the representation meaningful?
  4. How should we validate and interpret the result?

Unit 5: Responsible AI-Assisted Practice

From Tool User to Critical Supervisor

The statistician’s role is changing—not disappearing. Working responsibly with AI requires us to direct the task, evaluate the output, and remain accountable for the result.

Common Failure Modes

  • Fabricated references, data, or results
  • Incorrect but convincing code
  • Data leakage and overfitting
  • Unexamined bias in data or models
  • Disclosure of private or sensitive data
  • Irreproducible analyses
  • Confident interpretations that exceed the evidence

A Verification Loop

  1. Specify the research question, data, and desired output.
  2. Generate a draft with AI assistance when appropriate.
  3. Inspect the code, assumptions, and intermediate results.
  4. Verify important claims using documentation, diagnostics, and independent reasoning.
  5. Revise the analysis and document human decisions.
  6. Report AI assistance transparently when required.

Unit 6: Discussion and Activity

Activity: Trust, but Verify

Working in small groups, evaluate an AI-generated statistical response.

  1. Identify the research question and proposed analysis.
  2. List the assumptions the response makes.
  3. Find claims that require verification.
  4. Decide what information or diagnostics are missing.
  5. Revise the response into a defensible analysis plan.

Tip

An example AI-generated analysis will be added after we select the activity dataset and statistical problem.

Closing Discussion

  • Which statistical skills become more important in the AI era?
  • Which routine tasks are likely to change?
  • What should never be delegated without verification?
  • How can AI support learning without replacing thinking?

Takeaways

  1. AI can accelerate statistical learning, programming, analysis, and communication.
  2. Programming is shifting from syntax memorization toward code literacy, inspection, and supervision.
  3. Statistical knowledge is necessary for judging whether AI-generated work is meaningful and defensible.
  4. Research design, measurement, uncertainty, validation, and interpretation remain central.
  5. Productive AI use requires a deliberate cycle of generation, inspection, and verification.

Next Steps

  • Refine the lecture subsections and their relative emphasis.
  • Select a short case study or dataset for the audit activity.
  • Decide whether to include a live AI demonstration.