## Inference on Regression


## Plato's Allegory of the Cave

<div class="centered">
![plato](http://4.bp.blogspot.com/-rV1c4Xh4SSE/UZshhTTdFsI/AAAAAAAACQA/1VkmOaF7WFU/s1600/plato-cave.jpg)
</div>


## Statistical Inference {.build}

**Goal**: use *statistics* calculated from data to makes inferences about the 
nature of *parameters*.

In regression,

- statistics: $\hat{\beta}_0$, $\hat{\beta}_1$
- parameters: $\beta_0$, $\beta_1$

Classical tools of inference:

- Confidence Intervals
- Hypothesis Tests


## Quick Review (start the timer)


## Confidence Intervals {.build}

A confidence interval expresses the amount of uncertainly we have in our estimate
of a particular parameter.  A general 1 - $\alpha$ confidence interval takes the form

\[ \hat{\theta} \pm t^{*} * SE(\hat{\theta}) \]

- $\alpha$: is the confidence level, often .05
- $\hat{\theta}$: a statistic (point estimate)
- $t^{*}$ is the $100(1 - \alpha / 2)$ quantile of the sampling distribution of $\hat{\theta}$
- $SE$ is the standard error of $\hat{\theta}$, i.e. the standard deviation of its sampling
distribution.


## Regression Assumptions {.build}

1. $Y$ is related to $x$ by a simple linear regression model.
\[ E(Y|X) = \beta_0 + \beta_1 * x \]

2. The errors $e_1, e_2, \ldots, e_n$ are independent of one another.

3. The errors have a common variance $\sigma^2$.

4. The errors are normally distributed: $e \sim N(0, \sigma^2)$


## The Sampling Distribution of $\hat{\beta}_1$

Let's assume the following model as true:

\[ E(Y|X) = 12 + .7 * x; e \sim N(0, 4) \]

```{r, echo=FALSE,eval=TRUE}
n <- 60
beta_0 <- 12
beta_1 <- .7
sigma <- 2
x <- rnorm(n, mean = 20, sd = 3)

plot(20, 25, xlim = c(12, 28), ylim = c(17, 35), ylab = "y", xlab = "x", type = "n") # set up an empty plot
abline(a = beta_0, b = beta_1, col = "orange", lwd = 2) # add mean function
```

```{r eval = FALSE, echo = FALSE}
# generate data
f_mean <- beta_0 + beta_1 * x # mean function
f_data <- f_mean + rnorm(n, mean = 0, sd = sigma) # data generating function

points(x, f_data, pch = 16, col = "steelblue") # add generated data

# try to recover the true mean function
m1 <- lm(f_data ~ x)
m1
abline(m1)
```


## The Sampling Distribution of $\hat{\beta}_1$

```{r, echo=FALSE,eval=TRUE, cache=TRUE}
it <- 5000
coef_mat <- matrix(rep(NA, it * 2), ncol = 2)
for(i in 1:it) {
  x <- rnorm(n, mean = 20, sd = 3)
  f_mean <- beta_0 + beta_1 * x
  f_data <- f_mean + rnorm(n, mean = 0, sd = sigma)
  coef_mat[i, ] <- lm(f_data ~ x)$coef
}
```

```{r, eval=TRUE, echo=FALSE}
plot(20, 25, xlim = c(12, 28), ylim = c(17, 35), ylab = "y", xlab = "x", type = "n") # set up an empty plot

points(x, f_data, pch = 16, col = "steelblue") # add generated data
for(i in 1:it) {
  abline(coef_mat[i, 1], coef_mat[i, 2], col = rgb(0, 0, 0, 0.02))
}
abline(a = beta_0, b = beta_1, col = "orange", lwd = 2) # add mean function
```


## The Sampling Distribution of $\hat{\beta}_1$

```{r, echo=FALSE}
beta_1s <- coef_mat[, 2]
hist(beta_1s, breaks = 20, 
     main = expression(paste("Sampling distribution of ", hat(beta)[1])),
     xlab = expression(hat(beta)[1]), prob = TRUE)
# mean(beta_1s)
# sd(beta_1s)
```


## The Sampling Distribution of $\hat{\beta}_1$

```{r, echo=FALSE}
beta_1s <- coef_mat[, 2]
hist(beta_1s, breaks = 20, 
     main = expression(paste("Sampling distribution of ", hat(beta)[1])),
     xlab = expression(hat(beta)[1]), prob = TRUE)
abline(v = beta_1, col = "orange", lwd = 2)
text(beta_1+.02, .5, expression(beta[1]))
```


## The Sampling Distribution of $\hat{\beta}_1$

```{r, echo=FALSE}
beta_1s <- coef_mat[, 2]
hist(beta_1s, breaks = 20, 
     main = expression(paste("Sampling distribution of ", hat(beta)[1])),
     xlab = expression(hat(beta)[1]), prob = TRUE)
abline(v = beta_1, col = "orange", lwd = 2)
text(beta_1+.02, .5, expression(beta[1]))
xl <- seq(min(beta_1s), max(beta_1s), length.out = 100)
lines(xl, dnorm(xl, mean = beta_1, sd = sqrt(sigma^2/sum((x-mean(x))^2))), 
      lwd = 2, col = "thistle3")
```


## The Sampling Distribution of $\hat{\beta}_1$

Characteristics:

1. Centered at $\beta_1$, i.e. $E(\hat{\beta}_1) = \beta$.
2. $Var(\hat{\beta}_1) = \frac{\sigma^2}{SXX}$.
3. $\hat{\beta}_1 | X \sim N (\beta_1, \frac{\sigma^2}{SXX})$.


## Approximating the Sampling Dist. of $\hat{\beta}_1$ {.build}

Our best guess of $\beta_1$ is $\hat{\beta}_1$. And since we have to estimate
$\sigma$ with $\hat{\sigma}^2 = RSS/n-2$, the distribution isn't normal, but...

T with n - 2 degrees of freedom.

And we summarize that approximate sampling distribution using a CI:

\[ \hat{\beta}_1 \pm t_{\alpha/2, n-2} * SE(\hat{\beta}_1) \]

where

\[ SE(\hat{\beta}_1) = s/\sqrt(SXX) \]

## Constructing a CI for $\hat{\beta}_1$

\[ \hat{\beta}_1 \pm t_{\alpha/2, n-2} * SE(\hat{\beta}_1) \]

```{r, eval=FALSE}
beta_1 <- m1$coef[2]
alpha <- .05
t_stat <- qt(1-alpha/2, n - 2)
SE <- summary(m1)$coef[[4]]
moe <- t_stat * SE
c(beta_1 - moe, beta_1 + moe)

confint(m1, "x") # to double check
```


## Interpreting a CI for $\hat{\beta}_1$

We are *95% confident* that the true slope between x and y lies between LB and UB.


## Hypothesis test for $\hat{\beta}_1$

Suppose we are interested in testing the claim that the slope is zero.

\[ H_0: \beta_1^0 = 0 \\
H_A: \beta_1^0 \ne 0 \]

We know that

\[ T = \frac{\hat{\beta}_1 - \beta_1^0}{SE(\hat{\beta}_1)} \]

T will be t distributed with $n-2$ degrees of freedom and with $SE(\hat{\beta}_1)$
calculated the same as in the CI.


## Inference for $\hat{\beta}_0$

Often less interesting (but not always!).  You use the t-distribution again but
with a different $SE$.


## Activity 3b


