---
title: "Lecture 14: Regression Analysis of Experiments"
author: "MS&E 226"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, warning = FALSE, message = FALSE)
library(tidyverse)
library(sandwich)
library(lmtest)
set.seed(226)
```

# Introduction

This notebook demonstrates the key concepts from Lecture 14 on regression analysis of randomized experiments, with a focus on:

1. Simple regression for ATE estimation
2. Regression with covariate controls
3. **Interacted effects (IE) regression** - the recommended approach
4. Heterogeneous treatment effects
5. Robustness to model misspecification

All regression tables below use HC3 robust standard errors.

# Example 1: Simple Regression

First, let's replicate the simple example from the lecture where we estimate ATE using regression.

```{r simple_regression}
# Generate data: Y = 10 + 1*W + epsilon
n <- 2000
n1 <- n0 <- n/2

# True ATE = 1
true_ate <- 1

df_simple <- tibble(
  W = c(rep(0, n0), rep(1, n1)),
  epsilon = rnorm(n, 0, 1),
  Y = 10 + true_ate * W + epsilon
)

# Estimate ATE via regression
model_simple <- lm(Y ~ 1 + W, data = df_simple)
coeftest(model_simple, vcov = vcovHC(model_simple, type = "HC3"))

# The coefficient on W is our ATE estimate
cat("\nEstimated ATE:", coef(model_simple)["W"], "\n")
cat("True ATE:", true_ate, "\n")
```

# Example 2: Adding Covariate Controls

Now let's add a pre-treatment covariate X to improve precision.

```{r covariate_controls}
# Generate data with covariate: Y = 10 + 1*W + 2*X + epsilon
n <- 2000
n1 <- n0 <- n/2

df_controls <- tibble(
  X = rnorm(n, 0, 1),
  W = c(rep(0, n0), rep(1, n1)),
  epsilon = rnorm(n, 0, 1),
  Y = 10 + 1.0 * W + 2 * X + epsilon
)

# Regression without controls
model_no_control <- lm(Y ~ W, data = df_controls)

# Regression with controls
model_with_control <- lm(Y ~ W + X, data = df_controls)

# Compare standard errors
se_no_control <- sqrt(vcovHC(model_no_control, type = "HC3")["W", "W"])
se_with_control <- sqrt(vcovHC(model_with_control, type = "HC3")["W", "W"])

cat("Model without controls:\n")
coeftest(model_no_control, vcov = vcovHC(model_no_control, type = "HC3"))

cat("\n\nModel with controls:\n")
coeftest(model_with_control, vcov = vcovHC(model_with_control, type = "HC3"))

cat("\n\nComparison:\n")
cat("SE without controls:", se_no_control, "\n")
cat("SE with controls:", se_with_control, "\n")
cat("Reduction in SE:", round((1 - se_with_control/se_no_control)*100, 1), "%\n")
```

Adding the covariate X reduces the standard error substantially.

# Example 3: Interacted Effects (IE) Regression

Now let's demonstrate that IE regression can be more efficient than simple regression with controls.

We use HC3 robust standard errors for **conditional inference given the sample**: inference about the average treatment effect among sampled individuals. These SEs can be conservative.

## IE Regression with Homogeneous Treatment Effects

Even when treatment effects are homogeneous (constant across all individuals), IE regression is asymptotically at least as efficient as simple regression (similar SE).

```{r ie_regression_homogeneous}
# Same data as before (no heterogeneity)
# Y = 10 + 1*W + 2*X + epsilon

# Simple regression with controls
model_simple_controls <- lm(Y ~ W + X, data = df_controls)

# IE regression with interaction
df_controls <- df_controls %>%
  mutate(X_centered = X - mean(X))

model_ie <- lm(Y ~ W + X + W:X_centered, data = df_controls)

# Get robust standard errors for IE regression
ie_robust <- coeftest(model_ie, vcov = vcovHC(model_ie, type = "HC3"))

# Compare standard errors on W
se_simple <- sqrt(vcovHC(model_simple_controls, type = "HC3")["W", "W"])
se_ie <- ie_robust["W", "Std. Error"]

cat("Simple regression with controls:\n")
coeftest(model_simple_controls, vcov = vcovHC(model_simple_controls, type = "HC3"))

cat("\n\nIE Regression (with robust SEs):\n")
ie_robust

cat("\n\nComparison:\n")
cat("SE (simple regression):", se_simple, "\n")
cat("SE (IE regression with robust SEs):", se_ie, "\n")

# Note: interaction coefficient should be close to 0 (no true heterogeneity)
```

## IE Regression with Heterogeneous Treatment Effects

When treatment effects ARE heterogeneous, IE regression can be asymptotically strictly more efficient when the treatment/control allocation is not 50/50. With a 50/50 allocation, as here, the two estimators have the same asymptotic variance.

```{r ie_regression_heterogeneous}
# Generate data with HTE: Y = 10 + (1 + 2*X)*W + 2*X + epsilon
# Treatment effect = 1 + 2*X (varies with X)
# ATE = 1 (since E[X] = 0)

df_hte <- tibble(
  X = rnorm(n, 0, 1),
  W = c(rep(0, n0), rep(1, n1)),
  epsilon = rnorm(n, 0, 1),
  Y = 10 + (1.0 + 2.0*X) * W + 2 * X + epsilon,
  X_centered = X - mean(X)
)

# Simple regression with controls
model_simple_hte <- lm(Y ~ W + X, data = df_hte)

# IE regression
model_ie_hte <- lm(Y ~ W + X + W:X_centered, data = df_hte)

# Get robust standard errors for IE regression
ie_hte_robust <- coeftest(model_ie_hte, vcov = vcovHC(model_ie_hte, type = "HC3"))

# Compare standard errors on W (the ATE estimate)
se_simple_hte <- sqrt(vcovHC(model_simple_hte, type = "HC3")["W", "W"])
se_ie_hte <- ie_hte_robust["W", "Std. Error"]

cat("Simple regression with controls:\n")
coeftest(model_simple_hte, vcov = vcovHC(model_simple_hte, type = "HC3"))

cat("\n\nIE Regression (with robust SEs):\n")
ie_hte_robust

cat("\n\nComparison:\n")
cat("SE (simple regression):", se_simple_hte, "\n")
cat("SE (IE regression with robust SEs):", se_ie_hte, "\n")
```

The interaction coefficient is significant and close to 2, indicating heterogeneous treatment effects. The different reported SEs do not establish an efficiency gain; we compare sampling variances directly in Example 5.

# Example 4: Consistency Under Model Misspecification

A key property of IE regression: it gives a **consistent estimate of ATE even when the true model is NOT linear in X**.  For this example we use a larger value of $n$ to illustrate.

```{r misspecification}
# Generate data where true model is NONLINEAR in X
# Y = 10 + 1*W + X^2 + epsilon
# True ATE = 1 (does not depend on X)

n <- 5000
n1 <- n0 <- n/2

df_nonlinear <- tibble(
  X = rnorm(n, 0, 1),
  W = c(rep(0, n0), rep(1, n1)),
  epsilon = rnorm(n, 0, 0.5),
  Y = 10 + 1.0 * W + X^2 + epsilon,  # Nonlinear in X!
  X_centered = X - mean(X)
)

# Even though the model is misspecified (we're fitting linear in X),
# IE regression still gives consistent ATE estimate

model_ie_nonlinear <- lm(Y ~ W + X + W:X_centered, data = df_nonlinear)

# Get robust standard errors
ie_nonlinear_robust <- coeftest(model_ie_nonlinear, vcov = vcovHC(model_ie_nonlinear, type = "HC3"))

cat("True ATE:", 1.0, "\n")
cat("Estimated ATE (IE regression):", coef(model_ie_nonlinear)["W"], "\n")
cat("\nFull regression results (with robust SEs):\n")
ie_nonlinear_robust

# Plot: visualize the nonlinearity
ggplot(df_nonlinear, aes(x = X, y = Y, color = factor(W))) +
  geom_point(alpha = 0.5) +
  geom_smooth(method = "lm", se = FALSE, linetype = "dashed") +
  stat_function(fun = function(x) 10 + x^2,
                aes(color = "True E[Y|X,W=0]"),
                linetype = "solid", linewidth = 1) +
  stat_function(fun = function(x) 11 + x^2,
                aes(color = "True E[Y|X,W=1]"),
                linetype = "solid", linewidth = 1) +
  labs(title = "IE Regression with Misspecified Model",
       subtitle = "True model: Y = 10 + 1*W + X² + ε (quadratic in X)",
       color = "Group",
       y = "Outcome Y") +
  theme_minimal()
```

Even though we fit a linear model and the true relationship is quadratic, IE regression still estimates the ATE correctly (asymptotically as the sample size grows).

# Example 5: Simulation of sampling distribution

Let's run a simulation to verify the theoretical properties:
1. Both estimators are consistent for ATE.
2. IE regression has lower variance than simple regression in this example, where treatment effects are heterogeneous and the allocation is unbalanced (30/70).

This simulation holds the covariate matrix X fixed and only resamples treatment assignment and errors; the target is the average effect for these fixed covariates.

```{r monte_carlo, cache=TRUE}
# Simulation parameters
n_sims <- 1000
n <- 2000
n1 <- 600   # 30% treatment (UNBALANCED to show efficiency gain)
n0 <- 1400  # 70% control

# Generate FIXED covariate matrix (same X for all simulations)
set.seed(226)
X <- rnorm(n, 0, 1)
X_centered <- X - mean(X)
true_ate <- mean(1 + 2*X)

# Storage for estimates
results <- tibble(
  sim = 1:n_sims,
  ate_simple = NA_real_,
  ate_ie = NA_real_,
  se_simple = NA_real_,
  se_ie = NA_real_
)

for (i in 1:n_sims) {
  # Generate data with HETEROGENEOUS treatment effects
  # Control (W=0): Y = 10 + 2*X + epsilon
  # Treatment (W=1): Y = 10 + (1 + 2*X) + 2*X + epsilon = 11 + 4*X + epsilon
  # => Treatment effect = (11 + 4*X) - (10 + 2*X) = 1 + 2*X

  # Only resample W and epsilon (X is FIXED across simulations)
  W <- sample(c(rep(0, n0), rep(1, n1)))  # Random permutation
  epsilon <- rnorm(n, 0, 1)
  Y <- 10 + (1.0 + 2.0 * X) * W + 2 * X + epsilon

  df_sim <- tibble(X = X, W = W, Y = Y, X_centered = X_centered)

  # Simple regression
  m_simple <- lm(Y ~ W + X, data = df_sim)
  results$ate_simple[i] <- coef(m_simple)["W"]
  results$se_simple[i] <- sqrt(vcovHC(m_simple, type = "HC3")["W", "W"])

  # IE regression with centered X
  m_ie <- lm(Y ~ W + X + W:X_centered, data = df_sim)
  # Use robust standard errors
  ie_robust <- coeftest(m_ie, vcov = vcovHC(m_ie, type = "HC3"))
  results$ate_ie[i] <- coef(m_ie)["W"]
  results$se_ie[i] <- ie_robust["W", "Std. Error"]
}

# Analyze results
cat("=== Monte Carlo Results (", n_sims, "simulations) ===\n\n")

cat("SIMPLE REGRESSION:\n")
cat("  Mean estimate:", mean(results$ate_simple), "\n")
cat("  Bias:", mean(results$ate_simple) - true_ate, "\n")
cat("  SD of estimates:", sd(results$ate_simple), "\n")
cat("  Mean SE:", mean(results$se_simple), "\n\n")

cat("IE REGRESSION:\n")
cat("  Mean estimate:", mean(results$ate_ie), "\n")
cat("  Bias:", mean(results$ate_ie) - true_ate, "\n")
cat("  SD of estimates:", sd(results$ate_ie), "\n")
cat("  Mean SE:", mean(results$se_ie), "\n\n")

cat("EFFICIENCY GAIN:\n")
cat("  Variance reduction:",
    round((1 - var(results$ate_ie)/var(results$ate_simple))*100, 1), "%\n")

# Visualize sampling distributions
results %>%
  pivot_longer(cols = c(ate_simple, ate_ie),
               names_to = "method",
               values_to = "estimate") %>%
  mutate(method = recode(method,
                         ate_simple = "Simple Regression",
                         ate_ie = "IE Regression")) %>%
  ggplot(aes(x = estimate, fill = method)) +
  geom_histogram(alpha = 0.6, bins = 50, position = "identity") +
  geom_vline(xintercept = true_ate, linetype = "dashed", linewidth = 1) +
  labs(title = "Sampling Distribution of ATE Estimates",
       subtitle = paste0(n_sims, " simulations with heterogeneous treatment effects"),
       x = "Estimated ATE",
       y = "Frequency",
       fill = "Method") +
  theme_minimal()
```

**Key findings:**
- Both methods are approximately unbiased for the average effect for these fixed covariates
- IE regression has **lower variance** (narrower sampling distribution) due to unbalanced allocation (30/70) and heterogeneous treatment effects

# Example 6: Testing for Heterogeneous Treatment Effects

```{r test_heterogeneity}
# Use the HTE data from earlier
model_ie_hte <- lm(Y ~ W + X + W:X_centered, data = df_hte)

# Get robust standard errors
ie_hte_robust <- coeftest(model_ie_hte, vcov = vcovHC(model_ie_hte, type = "HC3"))

# Test H0: beta_WX = 0 (no heterogeneity under the linear CATE model)
# This is just a t-test on the interaction coefficient
cat("IE Regression with robust SEs:\n")
ie_hte_robust

cat("\nInterpretation:\n")
cat("The interaction coefficient is", coef(model_ie_hte)["W:X_centered"], "\n")
cat("This is highly significant (p < 0.001), indicating strong evidence of HTE.\n")
cat("The treatment effect increases by", coef(model_ie_hte)["W:X_centered"],
    "for each unit increase in X.\n")
```

# Summary and Recommendations

## Key Takeaways:

1. **Always use IE regression** when analyzing randomized experiments:
   - `Y ~ W + X + W:I(X - mean(X))`
   - NOT just `Y ~ W + X`

2. **Properties of IE regression:**
   - ✓ Consistent for ATE even if model is misspecified
   - ✓ Consistent for ATE even with heteroskedastic errors
   - ✓ Asymptotically at least as efficient as simple regression
   - ✓ Can be asymptotically strictly more efficient when treatment effects are heterogeneous, and treatment allocation is unbalanced
   - ✓ Allows you to test for and estimate heterogeneous treatment effects under a linear CATE model

3. **Practical implementation:**
   - Center covariates: use `W:I(X - mean(X))` in the interaction
   - **MUST use robust standard errors** (HC3)
   - Standard `lm()` SEs can be incorrect due to heteroskedasticity
   - Test for heterogeneity using t-test on interaction coefficient

4. **Citation:**
   - Lin, W. (2013). "Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique." *Annals of Applied Statistics*, 7(1), 295-318.

```{r session_info}
sessionInfo()
```
