ETC5521: Exploratory Data Analysis

.info-box.w-50.bg-white[
These slides are viewed best by Chrome or Firefox and occasionally need to be refreshed if elements did not load properly. See <a href=lecture-04A.pdf>here for the PDF </a>. 
]

---

# .monash-blue[ETC5521: Exploratory Data Analysis]

<h2 style="font-weight:900!important;">Using computational tools to determine whether what is seen in the data can be assumed to apply more broadly</h2>

.bottom_abs.width100[

Lecturer: *Di Cook*

ETC5521.Clayton-x@monash.edu

Week 4 - Session 1

]

---

# Revisiting .yellow[hypothesis testing]

---

# (Frequentist) hypothesis testing framework

* Suppose `$X$` is the number of heads out of `$n$` independent tosses.
* Let `$p$` be the probability of getting a <img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"> for this coin.

| | |
|-----|-----|
|**Hypotheses** | `$H_0: p = 0.5$` vs. `$H_a: p > 0.5$`. Note `$p_0=0.5$`. Alternative `$H_a$` is saying we believe that the coin is biased to heads. .monash-orange2[Alternative needs to be decided before seeing data.] |
--

|**Assumptions**  | Each toss is independent with equal chance of getting a head.   |
--

|**Test statistic** | `$X \sim B(n, p)$`. Recall `$E(X\mid H_0) = np_0$`. We observe `$n, x, \widehat{p}$`. Test statistic is `$\widehat{p} - p_0$`.|
--

|**P-value** .font_small[(or critical value or confidence interval)] | `$P(X ~ \geq ~ x\mid H_0)$` |
--

|** Conclusion** | Reject null hypothesis when the `$p$`-value is less than some significance level `$\alpha$`. Usually `$\alpha = 0.05$`.|

---

# Testing coin bias .font_small[Part 1/2]

* Suppose I have a coin that I'm going to flip <img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"> <img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"> 
--

* If the coin is unbiased, what is the probability it will show heads?
--

* Yup, the probability should be 0.5. 
--

* So how would I test if a coin is biased or unbiased?
--

* We'll collect some data. 
--

* **Experiment 1**: I flipped the coin 10 times and this is the result:

<center>
<img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;">
</center>
--

* The result is 7 head and 3 tails. So 70% are heads. 
--

* Do you believe the coin is biased based on this data?

---

# Testing coin bias .font_small[Part 2/2]

* **Experiment 2**: Suppose now I flip the coin 100 times and this is the outcome:

<img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_T.png" height = "50px" style="vertical-align:middle;"><img src="images/Australian_20c_H.png" height = "50px" style="vertical-align:middle;">
--

* We observe 70 heads and 30 tails. So again 70% are heads. 
--

* Based on this data, do you think the coin is biased?

---
# Calculate it

## Experiment 1 (n=10)

- We observed `$x=7$`, or `$\widehat{p} = 0.7$`.

- Assuming `$H_0$` is true, we expect `$np=10\times 0.5=5$`.

- Calculate the `$P(X \geq 7)$`

```r
sum(dbinom(7:10, 10, 0.5))
```

```
## [1] 0.171875
```

]

.item.w-50[

## Experiment 1 (n=100)

- We observed `$x=70$`, or `$\widehat{p} = 0.7$`.

- Assuming `$H_0$` is true, we expect `$np=100\times 0.5=50$`.

- Calculate the `$P(X \geq 70)$`

```r
sum(dbinom(70:100, 100, 0.5))
```

```
## [1] 3.92507e-05
```

]

]
---

# Judicial system

.item.bg-gray80.pl3[
 
{{content}}
]

]

<center>
<img src="images/omni/statistical-court.png">
</center>
<ul>
<li> Evidence by test statistic</li>
<li> Judgement by p-value, critical value or confidence interval</li>
</ul>
{{content}}

Does the test statistic have to be a numerical summary statistic?

---

# Visual inference

---

# Visual inference

* Hypothesis testing in visual inference framework is where:
 * the .monash-blue[_test statistic is a plot_] and 
 * judgement is by human visual perception.
{{content}}

]
.item.bg-gray80.pa3[

]
]
--

* You (and many other people) actually do visual inference many times but generally in an informal fashion.
{{content}}
--

* Here, we are making an inference on whether the residual plot has any patterns based on a single data plot.

---

Data plots tend to be over-interpreted

Reading data plots require calibration

]

---

# Visual inference more formally

.flex[
.w-60[
1. State your null and alternate hypotheses.
{{content}}
]]
--
2. Define a visual test statistic, `$V(.)$`, i.e. a function of a sample to a plot.
{{content}}
--
3. Define a method to generate null data, `$\boldsymbol{y}_0$`. 
{{content}}
--
4. `$V(\boldsymbol{y})$` maps the actual data, `$\boldsymbol{y}$`, to the plot. We call this the data plot.
{{content}}
--
5. `$V(\boldsymbol{y}_0)$` maps a null data to a plot of the same form. We call this the null plot. We repeat this `$m - 1$` times to generate `$m-1$` null plots. 
{{content}}
--
6. A lineup displays these `$m$` plots in a random order. 
{{content}}
--
7. Ask `$n$` human viewers to select a plot in the lineup that looks different to others without any context given.

---

# Visual inference more formally

.flex[
.w-60[
1. State your null and alternate hypotheses.
2. Define a **visual test statistic**, `$V(.)$`, i.e. a function of a sample to a plot.
3. Define a method to generate **null data**, `$\boldsymbol{y}_0$`. 
4. `$V(\boldsymbol{y})$` maps the actual data, `$\boldsymbol{y}$`, to the plot. We call this the .monash-blue[**data plot**].
5. `$V(\boldsymbol{y}_0)$` maps a null data to a plot of the same form. We call this the .monash-blue[**null plot**]. We repeat this `$m - 1$` times to generate `$m-1$` null plots. 
6. A .monash-blue[**lineup**] displays these `$m$` plots in a random order. 
7. Ask `$n$` human viewers to select a plot in the lineup that looks different to others without any context given.

]
.w-40.pl3[

.info-box[
Suppose `$x$` out of `$n$` people detected the data plot from a lineup, then
* the .monash-blue[**visual inference p-value**] is given as `$$P(X \geq x)$$` where `$X \sim B(n, 1/m)$`, and 
* the .monash-blue[**power of a lineup**] is estimated as `$x/n$`.
]

]]

---

# .monash-blue[Lineup] .circle.monash-bg-blue.white[1] In which plot has a pattern that is different from other plots?

.flex[
.item.w-70[
<img src="images/week4A/cars-lineup-1.png" width="80%" style="display: block; margin: auto;" />

]

.item.w-30.f5[
{{content}}
]

]

Recall the linear model for cars shown in week 3.

```r
lm(dist ~ speed, data = cars)
```
 
* This is a lineup of the residual plot
* <text style="color: #D93F00;"> Which plot (if any) looks different from the others? </text>
* <text style="color: #D93F00;"> Why do you think it looks different? </text>
* Note: there is no correct answer here. 
{{content}}

```
> decrypt("clZx bKhK oL 3OHohoOL 0B")
[1] "True data in position 11"
```
 
<text style="color: #D93F00;"> How do we calculate statistical significance from this?</text>

---

# Visual inference p-value (or "see"-value)

.flex[
.item.w-45[
* So `$x$` out of `$n$` people chose the data plot.
* So the visual inference `$p$`-value is `$P(X \geq x)$` where `$X \sim B(n, 1/10)$`.
* In R, this is 
```r
1 - pbinom(x - 1, n, 1/20) 
# OR 
nullabor::pvisual(x, n, 20)
```
* The calculation is made with the assumption that the chance of a single observer randomly chooses the true plot is 1/20. 
]

.item.w-10[
.white[space]
]

.item.w-45[
Suppose `$x=2$` out of `$n=16$` people chose plot 11 (previous slide).

The probability that this happens by random guessing (p-value) is

```r
1 - pbinom(2 - 1, 16, 1/20)
```

```
## [1] 0.1892403
```

```r
nullabor::pvisual(2, 16, 20)
```

```
##      x simulated     binom
## [1,] 2     0.204 0.1892403
```
]
]

---

# .monash-blue[Lineup] .circle.monash-bg-blue.white[2] In which plot has a pattern that is different from other plots?

.flex[
.item.w-70[
<img src="images/week4A/diamonds-lineup-1.png" width="80%" style="display: block; margin: auto;" />

]

.item.w-30.f5[
{{content}}
]

]

Recall the linear model for diamonds shown in week 3.

```r
d_fit <- lm(lprice ~ lcarat, data=diamonds)
```
 
* This is a lineup of the residual plot for the model where both carat and price are log-transformed
* <text style="color: #D93F00;"> Which plot (if any) looks different from the others? </text>
* <text style="color: #D93F00;"> Why do you think it looks different? </text>
* Note: there is no correct answer here. 
{{content}}

```
> decrypt("clZx bKhK oL 3OHohoOL 0Q")
[1] "True data in position  15"
```

---

# Visual inference p-value (or "see"-value)

The probability that this happens by random guessing (p-value) is

```r
1 - pbinom(8 - 1, 12, 1/20)
```

```
## [1] 1.612352e-08
```

```r
nullabor::pvisual(8, 12, 20)
```

```
##      x simulated        binom
## [1,] 8         0 1.612352e-08
```
]

.item.w-10[
.white[space]
]

.item.w-45[

Next, how the residuals are different from "good" residuals has to be determined by the follow-up question: how did you decide your chosen plot was different?

Plot 15 has a different variance pattern, it's not the regular up-down pattern seen in the other plots. This suggests that there is some .monash-orange2[heteroskedasticity] in the data that is not captured by the error distribution in the model.

]
]

---
class: transition

# Why?

---
# Residual plot (1/3)

.item.w-70[
<img src="images/week4A/unnamed-chunk-10-1.png" width="60%" style="display: block; margin: auto;" />
]
]

---
# Residual plot (2/3)

.item.w-70[
<img src="images/week4A/unnamed-chunk-11-1.png" width="60%" style="display: block; margin: auto;" />
]
]

---
# Residual plot (3/3)

.item.w-70[
<img src="images/week4A/unnamed-chunk-12-1.png" width="60%" style="display: block; margin: auto;" />
]
]

---
class: transition

# Residual plots need context

It's really hard to decide that there is NO PATTERN!

<center>
Residual plots are better when viewed in the context of good residual plots, where we know the assumptions of the model are satisfied.
</center>

---

.item.w-70[
<img src="images/week4A/unnamed-chunk-13-1.png" width="80%" style="display: block; margin: auto;" />
]
]

---

.item.w-70[

<img src="images/week4A/unnamed-chunk-14-1.png" width="80%" style="display: block; margin: auto;" />
]
]

---

.item.w-70[

<img src="images/week4A/unnamed-chunk-15-1.png" width="80%" style="display: block; margin: auto;" />
]
]

---
class: transition

# It's not only for residual plots

---

.item.w-70[

]
]

---
.flex[
.item.w-30[
 
Which plot is most different?
]

.item.w-70[

]
]

---
.flex[
.item.w-30[
 
Which plot is most different?
]

.item.w-70[
<img src="images/week4A/unnamed-chunk-18-1.png" width="80%" style="display: block; margin: auto;" />

]
]

---
class: transition

# Reading any plot can benefit from the context of null plots

---

# Resources and Acknowledgement

.font18[
- Buja, Andreas, Dianne Cook, Heike Hofmann, Michael Lawrence, Eun-Kyung Lee, Deborah F. Swayne, and Hadley Wickham. 2009. “Statistical Inference for Exploratory Data Analysis and Model Diagnostics.” Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences 367 (1906): 4361–83.
- Wickham, Hadley, Dianne Cook, Heike Hofmann, and Andreas Buja. 2010. “Graphical Inference for Infovis.” IEEE Transactions on Visualization and Computer Graphics 16 (6): 973–79.
- Hofmann, H., L. Follett, M. Majumder, and D. Cook. 2012. “Graphical Tests for Power Comparison of Competing Designs.” IEEE Transactions on Visualization and Computer Graphics 18 (12): 2441–48.
- Majumder, M., Heiki Hofmann, and Dianne Cook. 2013. “Validation of Visual Statistical Inference, Applied to Linear Models.” Journal of the American Statistical Association 108 (503): 942–56.
- Data coding using [`tidyverse` suite of R packages](https://www.tidyverse.org) 
- Slides originally written by Emi Tanaka and constructed with [`xaringan`](https://github.com/yihui/xaringan), [remark.js](https://remarkjs.com), [`knitr`](http://yihui.name/knitr), and [R Markdown](https://rmarkdown.rstudio.com).

]

---

background-size: cover
class: title-slide
background-image: url("images/bg-01.png")

<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a> This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

.bottom_abs.width100[

Lecturer: *Di Cook*

ETC5521.Clayton-x@monash.edu

Week 4 - Session 1

]