Hypothesis Testing

class: center, middle, inverse, title-slide

# Hypothesis Testing
## NEUS 643
### Ted Laderas
### 2020-05-19

---

# Learning Objectives

- Understand the steps of hypothesis testing
- Understand that hypothesis testing specifies a null hypothesis
- Understand the importance of the *null hypothesis* in generating the null distribution
- Apply *shuffling* to permute the null distribution
- Interpret the p-value under permutations

---
<img src="image/week7/stats_help.jpg" width=800>

---
# How Special is our Data?

How special is our data compared to fake data generated under the *null hypothesis*?

---
![](08_hypothesis_testing_files/figure-html/unnamed-chunk-2-1.png)

---
# Why Hypothesis Test?

- Given our observed sample, assess whether we can make an inference about the real value given:
    - Limited sample size
    - Small Effect size
- What kind of inference? Are we comfortable with our results compared to randomly generated data?

---
# Hypothesis Test

- Test between a *null* and *alternative* hypothesis
- Alternative hypothesis (Ha) - What we're interested in
- Null Hypothesis (Ho) - what we expect from just chance and noise

---
# Steps in Hypothesis Testing

1. Define Test Statistic
2. Specify alternative hypothesis with test statistic
3. Use alternative hypothesis to specify null hypothesis
4. Decide on a significance level
5. Calculate observed test statstic
6. Permute Data and generate distribution
7. Calculate p-value
8. Interpret!
-

???

Adapted from https://www.nedarc.org/statisticalHelp/advancedStatisticalTopics/hypothesisTesting.html

---
# Our Test Statistic

- Difference between means of the two groups
  - `bc` and `nuc`

---

---
<img src="image/week7/why_null.jpg" width=800>
---
<img src="image/week7/theoretical_null.JPG" width=800>

---
# Empirical versus Theoretical

- Theoretical distributions rely on assumptions
  - Normality is a big one!
- Empirical distributions utilize data

---
<img src="image/week7/observed_calc.JPG" width=800>

---
<img src="image/week7/null_permute.JPG" width=800>
---
<img src="image/week7/null_permute2.JPG" width=800>
---
<img src="image/week7/p_permute.JPG" width=800>
---
<img src="image/week7/p_value_significance.JPG" width=800>
---
class: center, middle

# Real Life Example

---
# Our Data

.pull-left[
First few rows of table

#ynmbqrnhtr .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#ynmbqrnhtr .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#ynmbqrnhtr .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#ynmbqrnhtr .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#ynmbqrnhtr .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#ynmbqrnhtr .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#ynmbqrnhtr .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#ynmbqrnhtr .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#ynmbqrnhtr .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#ynmbqrnhtr .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#ynmbqrnhtr .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#ynmbqrnhtr .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#ynmbqrnhtr .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#ynmbqrnhtr .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#ynmbqrnhtr .gt_from_md > :first-child {
  margin-top: 0;
}

#ynmbqrnhtr .gt_from_md > :last-child {
  margin-bottom: 0;
}

#ynmbqrnhtr .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#ynmbqrnhtr .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#ynmbqrnhtr .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#ynmbqrnhtr .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#ynmbqrnhtr .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#ynmbqrnhtr .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#ynmbqrnhtr .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#ynmbqrnhtr .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#ynmbqrnhtr .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#ynmbqrnhtr .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#ynmbqrnhtr .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#ynmbqrnhtr .gt_left {
  text-align: left;
}

#ynmbqrnhtr .gt_center {
  text-align: center;
}

#ynmbqrnhtr .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#ynmbqrnhtr .gt_font_normal {
  font-weight: normal;
}

#ynmbqrnhtr .gt_font_bold {
  font-weight: bold;
}

#ynmbqrnhtr .gt_font_italic {
  font-style: italic;
}

#ynmbqrnhtr .gt_super {
  font-size: 65%;
}

#ynmbqrnhtr .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="ynmbqrnhtr" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">type</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1">x_0_s_area</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">770</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">747</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">939</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">1144</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">1210</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">1225</td>
    </tr>
  </tbody>
  
  
</table></div>
]

.pull-right[
Counts in each group

```
##     type    n   percent
##       bc 1153 0.5070361
##  nucleus 1121 0.4929639
```
]

---

# Difference between means in data

.pull-left[
We observe a difference in mean area of 19.713712 pixels between the `bc` and `nuc` groups
]

.pull-right[
![](08_hypothesis_testing_files/figure-html/unnamed-chunk-6-1.png)
]
---

---
# A couple permutations

.pull-left[
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#wssluuvbho .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#wssluuvbho .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#wssluuvbho .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#wssluuvbho .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#wssluuvbho .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#wssluuvbho .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#wssluuvbho .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#wssluuvbho .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#wssluuvbho .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#wssluuvbho .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#wssluuvbho .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#wssluuvbho .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#wssluuvbho .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#wssluuvbho .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#wssluuvbho .gt_from_md > :first-child {
  margin-top: 0;
}

#wssluuvbho .gt_from_md > :last-child {
  margin-bottom: 0;
}

#wssluuvbho .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#wssluuvbho .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#wssluuvbho .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#wssluuvbho .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#wssluuvbho .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#wssluuvbho .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#wssluuvbho .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#wssluuvbho .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#wssluuvbho .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#wssluuvbho .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#wssluuvbho .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#wssluuvbho .gt_left {
  text-align: left;
}

#wssluuvbho .gt_center {
  text-align: center;
}

#wssluuvbho .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#wssluuvbho .gt_font_normal {
  font-weight: normal;
}

#wssluuvbho .gt_font_bold {
  font-weight: bold;
}

#wssluuvbho .gt_font_italic {
  font-style: italic;
}

#wssluuvbho .gt_super {
  font-size: 65%;
}

#wssluuvbho .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="wssluuvbho" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">type</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1">x_0_s_area</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">run</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">770</td>
      <td class="gt_row gt_center">1</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">nucleus</td>
      <td class="gt_row gt_right">747</td>
      <td class="gt_row gt_center">1</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">nucleus</td>
      <td class="gt_row gt_right">939</td>
      <td class="gt_row gt_center">1</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">1144</td>
      <td class="gt_row gt_center">1</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">1210</td>
      <td class="gt_row gt_center">1</td>
    </tr>
  </tbody>
  
  
</table></div>
]

.pull-right[
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#azukgxhjmk .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#azukgxhjmk .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#azukgxhjmk .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#azukgxhjmk .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#azukgxhjmk .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#azukgxhjmk .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#azukgxhjmk .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#azukgxhjmk .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#azukgxhjmk .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#azukgxhjmk .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#azukgxhjmk .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#azukgxhjmk .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#azukgxhjmk .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#azukgxhjmk .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#azukgxhjmk .gt_from_md > :first-child {
  margin-top: 0;
}

#azukgxhjmk .gt_from_md > :last-child {
  margin-bottom: 0;
}

#azukgxhjmk .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#azukgxhjmk .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#azukgxhjmk .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#azukgxhjmk .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#azukgxhjmk .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#azukgxhjmk .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#azukgxhjmk .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#azukgxhjmk .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#azukgxhjmk .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#azukgxhjmk .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#azukgxhjmk .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#azukgxhjmk .gt_left {
  text-align: left;
}

#azukgxhjmk .gt_center {
  text-align: center;
}

#azukgxhjmk .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#azukgxhjmk .gt_font_normal {
  font-weight: normal;
}

#azukgxhjmk .gt_font_bold {
  font-weight: bold;
}

#azukgxhjmk .gt_font_italic {
  font-style: italic;
}

#azukgxhjmk .gt_super {
  font-size: 65%;
}

#azukgxhjmk .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="azukgxhjmk" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">type</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1">x_0_s_area</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">run</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr>
      <td class="gt_row gt_left">nucleus</td>
      <td class="gt_row gt_right">770</td>
      <td class="gt_row gt_center">4</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">747</td>
      <td class="gt_row gt_center">4</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">939</td>
      <td class="gt_row gt_center">4</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">1144</td>
      <td class="gt_row gt_center">4</td>
    </tr>
    <tr>
      <td class="gt_row gt_left">bc</td>
      <td class="gt_row gt_right">1210</td>
      <td class="gt_row gt_center">4</td>
    </tr>
  </tbody>
  
  
</table></div>
]
---
# Calculate and plot our Observed Statistic

![](08_hypothesis_testing_files/figure-html/unnamed-chunk-10-1.png)

---
# Generate statistics from permuted data

![](08_hypothesis_testing_files/figure-html/unnamed-chunk-11-1.png)

---
# 100 Permutations

![](08_hypothesis_testing_files/figure-html/unnamed-chunk-12-1.png)

---
# 1000 Permutations

![](08_hypothesis_testing_files/figure-html/unnamed-chunk-13-1.png)

---
# P-Value

.pull-left[
We observe 13 permuted data sets under the null distirbution have a mean difference of 19.713712 or higher.

Our p-value is 13 / 1000 = 0.013
]

.pull-right[

![](08_hypothesis_testing_files/figure-html/unnamed-chunk-14-1.png)
]
---
# A Smaller Data Set

```
##     type  n   percent
##       bc 13 0.4333333
##  nucleus 17 0.5666667
```

---
![](08_hypothesis_testing_files/figure-html/unnamed-chunk-16-1.png)

```
## [1] 404
```

---
## Bigger Difference, Smaller n

.pull-left[
We observe 404 permuted data sets under the null distribution have a mean difference of 19.713712 or higher.

Our p-value is 404 / 1000 = 0.404

]
.pull-right[
![](08_hypothesis_testing_files/figure-html/unnamed-chunk-18-1.png)
]
---
# There is One Test

---
# Next Time

- Lab: Colocalization analysis using correlation

---
# Reading

- [There's still only one test](http://allendowney.blogspot.com/2016/06/there-is-still-only-one-test.html)
- [The Permutation Test, by Jared Wilber](https://www.jwilber.me/permutationtest/)
- [A practical guide to evaluating colocalization in biological microscopy](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3074624/)