A quick tour of R

class: center, middle, inverse, title-slide

# A quick tour of R
### Abhijit Dasgupta, PhD

---

layout: true

<div class="my-header">
<span>BIOF 439: Data Visualization using R</span></div>

---

## The bare essentials

- R is a **language**
- Everything in R is an object with a name

- Things are nouns
    - Nouns in R are _data objects_, like scalars, matrices, data.frames/tibbles, strings, vectors
- Nouns are acted upon by verbs
    - Verbs in R are _functions_, like `mean(x)`, `nrow(d)`, `dim(d)`, `ggplot` and so on
- You can modify verbs with adverbs
    - Adverbs in R are _function options_, like `mean(x, na.rm=T)`, `geom_point(color='green')`

---

## The bare essentials

- If you want to save and re-use an object (noun or verb), you have to name it
    - This is done with the `<-` operator, e.g. 
        - `beaches <- read_csv('data/sydneybeaches3.csv')` (dataset)
        - `mn <- mean(x, na.rm=T)` (a number)
        - `my_theme <- function() {theme_bw() + theme(axis.title=element_text(size=14))}` stores the function, which you'll call as `my_theme()`

> You can see the objects you have created either by typing `ls()` in the console, or looking in the Environment pane

> Note, built-in objects don't show up in the Environment pane or using `ls()`

---

## The bare essentials

### Naming rules

1. Identifiers can be a combination of letters, digits, period (.) and underscore (_).
1. It must start with a letter or a period. If it starts with a period, it cannot be followed by a digit.
1. Reserved words in R cannot be used as identifiers.

### Naming conventions

1. Make names expressive
1. Write multi-word names in CamelCase or snake_case

---

## The bare essentials

### Allowing non-conventional names

If a name doesn't follow the rules *per se*, it can still be used by quoting them
using backticks.

So the following names are valid, as long as backticks are present.

```
`Pelvic girdle`
`1`
`Sample 03/23/2019`
```

This is rather difficult to type on a regular basis, and can lead to syntactical errors

----

The **janitor** package has a `clean_names` function that can fix the names of columns
when you import datasets, transforming them into snake case by default.

---

## The bare essentials (brackets)

`[]` are used for extracting elements from arrays, matrices, data frames.

- `x[3]` is the 3rd element of an array `x`
- `d[1,3]` is the element in the 1st row and 3rd column of a matrix/data frame `d`
- `d[2,]` is the entire first __row__ of a matrix/data frame `d`
- `d[,4]` is the entire 4th __column__ of a matrix/data frame `d`
- `d[,'gender']` is the column named _gender_ in a _data frame_ `d`

For a list object (we'll meet them in a few slides) you use a double bracket
to extract elements

- `lst[[3]]` is the third element of the list `lst`
- `lst[['apple']]` is the element named _apple_ in a named list `lst`
    
---

## The bare essentials (brackets)

There are three kinds of brackets in R

`()` are used for specifying arguments to functions

- `mean(x)` gives the mean of an array of numbers `x`
- `summary(d)` gives a summary representation of a data frame `d`
- There are several functions used to make the following visualization

```r
ggplot(beaches, aes(x = temperature, y = rainfall)) + 
  geom_point() + 
  geom_smooth() + 
  theme_classic()
```
---

## The bare essentials (brackets)

There are three kinds of brackets in R

`{}` are used to contain groups of commands/statements

.pull-left[

A conditional statement

```r
if (age < 18){
   person <- 'Minor'
} else if (age > 65) {
  person <- 'Senior'
} else {
  person <- 'Adult'
}
```
]
.pull-right[

A function definition

```r
my_mean <- function(x, na.rm = T){
  if(na.rm){
    x <- x[!is.na(x)]
  }
  s <- sum(x)
  n <- length(x)
  mn <- s / n # There is a built-in function mean, so I don't use that word
  return(mn)
}
```

]

See reading and supplementary materials for more details about these constructs

---

## Data structures

A __scalar__ (data taking a single value)

- `29`
- `"cherry"`
- `TRUE`

---

## Data structures

A __scalar__ (data taking a single value)

- `29` : _numeric_
- `"cherry"`: _character_
- `TRUE`: _logical_

----
These are different data *types*, which determine how they are stored in memory. You may also see _integer_ and 
_double_, which are both covered under _numeric_.

You can check an object's type using `class`: `class(29)` (resulting in `numeric`) <br/>
You can check if an object is of a particular type: `is.character(29)` (resulting in `FALSE`)<br/>
You can transform an object to a different type, if allowed: `as.numeric("29")` (resulting in `29`)<p>
The `is.___` and `as.___` functions are extremely useful for data cleaning and transformations<p>
You typically use the variable names rather than the actual values for these functions
---

## Data structures

### Vectors/Arrays

These are constructed using the `c()` function (for _concatenate_).

.pull-left[

```r
c(1,2,5,6,7,8)
```

```
#>  [1] 1 2 5 6 7 8
```

```r
c('apple','berry','melon','citrus')
```

```
#>  [1] "apple"  "berry"  "melon"  "citrus"
```
]
.pull-right[
Vectors must all contain objects of the same type. Can't mix and match<p> 
If you check the `class` of an array, it will give the common data type of the elements of the array
]

---

## Data structures

### Matrices (2-d arrays)

These are typically built from vectors

```r
x <- c(1,2,4,5,6,7)
y <- 10:16  # Shortcut for c(10,11,12,13,14,15,16)
```

.pull-left[

```r
cbind(x, y) # Vectors as columns
```

```
#>       x  y
#>  [1,] 1 10
#>  [2,] 2 11
#>  [3,] 4 12
#>  [4,] 5 13
#>  [5,] 6 14
#>  [6,] 7 15
#>  [7,] 1 16
```
]
.pull-right[

```r
rbind(x, y) # Vectors as rows
```

```
#>    [,1] [,2] [,3] [,4] [,5] [,6] [,7]
#>  x    1    2    4    5    6    7    1
#>  y   10   11   12   13   14   15   16
```
]

---

## Data structures

### Matrices (2-d arrays)

You can also build matrices directly using the `matrix` function.

```r
matrix(seq(1, 6), nrow = 3) # Automatically determines number of columns
```

```
#>       [,1] [,2]
#>  [1,]    1    4
#>  [2,]    2    5
#>  [3,]    3    6
```

----
Play around with different ways of creating vectors and matrices

---

## Data structures

### Lists

Lists are basically buckets or containers. Each element of a list can be anything, even other lists

.pull-left[

```r
my_list <- list('a', c(2,3,5,6), head(ggplot2::mpg))
my_list
```

This has a scalar, a vector and a data set<p>
Since lists can be composed of heterogeneous objects, the `class` of a list is `list`.
]
.pull-right[

```
#>  [[1]]
#>  [1] "a"
#>  
#>  [[2]]
#>  [1] 2 3 5 6
#>  
#>  [[3]]
#>  # A tibble: 6 x 11
#>    manufacturer model displ  year   cyl trans      drv     cty   hwy fl    class 
#>    <chr>        <chr> <dbl> <int> <int> <chr>      <chr> <int> <int> <chr> <chr> 
#>  1 audi         a4      1.8  1999     4 auto(l5)   f        18    29 p     compa…
#>  2 audi         a4      1.8  1999     4 manual(m5) f        21    29 p     compa…
#>  3 audi         a4      2    2008     4 manual(m6) f        20    31 p     compa…
#>  4 audi         a4      2    2008     4 auto(av)   f        21    30 p     compa…
#>  5 audi         a4      2.8  1999     6 auto(l5)   f        16    26 p     compa…
#>  6 audi         a4      2.8  1999     6 manual(m5) f        18    26 p     compa…
```
]
---

## Data structures

### Lists

You can create named lists, where every element has a name

```r
my_list2 <- list('scalar' = 'a', 'vector' = c(2,3,5,6), 'data' = head(ggplot2::mpg))
my_list2[['vector']]
```

```
#>  [1] 2 3 5 6
```

```r
my_list2[['data']]
```

```
#>  # A tibble: 6 x 11
#>    manufacturer model displ  year   cyl trans      drv     cty   hwy fl    class 
#>    <chr>        <chr> <dbl> <int> <int> <chr>      <chr> <int> <int> <chr> <chr> 
#>  1 audi         a4      1.8  1999     4 auto(l5)   f        18    29 p     compa…
#>  2 audi         a4      1.8  1999     4 manual(m5) f        21    29 p     compa…
#>  3 audi         a4      2    2008     4 manual(m6) f        20    31 p     compa…
#>  4 audi         a4      2    2008     4 auto(av)   f        21    30 p     compa…
#>  5 audi         a4      2.8  1999     6 auto(l5)   f        16    26 p     compa…
#>  6 audi         a4      2.8  1999     6 manual(m5) f        18    26 p     compa…
```

---
## Data structures

### data.frame/tibble

This is the typical container for tabular data

- must be rectangular
- each column can be of a different type
- elements within each column have to be of the same type

----
The data frame is probably the single most important structure in R for data analysis. It
allows you to keep different kinds of data for each observation together, regardless of type. 
Many other scripting systems couldn't keep heterogeneous data together, which made practical data analyses
very difficult. <p>
This kind of structure is now present in almost every serious data analytic platform, including Python, Julia,
SAS, SPSS. Matlab remains a significant exception.

---

## Data structures

### data.frame/tibble

```r
# use the import function from the package rio
beaches <- rio::import('data/sydneybeaches3.csv') 
class(beaches)
```

```
#>  [1] "data.frame"
```

```r
dim(beaches)
```

```
#>  [1] 344  12
```

```r
head(beaches)
```

```
#>          date year month day season rainfall temperature enterococci day_num
#>  1 2013-01-02 2013     1   2      1      0.0        23.4         6.7       2
#>  2 2013-01-06 2013     1   6      1      0.0        30.3         2.0       6
#>  3 2013-01-12 2013     1  12      1      0.0        31.4        69.1      12
#>  4 2013-01-18 2013     1  18      1      0.0        46.4         9.0      18
#>  5 2013-01-24 2013     1  24      1      0.0        27.5        33.9      24
#>  6 2013-01-30 2013     1  30      1      0.6        26.6        26.5      30
#>    month_num month_name season_name
#>  1         1    January      Summer
#>  2         1    January      Summer
#>  3         1    January      Summer
#>  4         1    January      Summer
#>  5         1    January      Summer
#>  6         1    January      Summer
```

---

## Data structures

### data.frame/tibble

```r
library(tidyverse) # Activate the tidyverse package
beaches_t <- as_tibble(beaches)
class(beaches_t)
```

```
#>  [1] "tbl_df"     "tbl"        "data.frame"
```

```r
beaches_t
```

```
#>  # A tibble: 344 x 12
#>     date        year month   day season rainfall temperature enterococci day_num
#>     <date>     <int> <int> <int>  <int>    <dbl>       <dbl>       <dbl>   <int>
#>   1 2013-01-02  2013     1     2      1      0          23.4         6.7       2
#>   2 2013-01-06  2013     1     6      1      0          30.3         2         6
#>   3 2013-01-12  2013     1    12      1      0          31.4        69.1      12
#>   4 2013-01-18  2013     1    18      1      0          46.4         9        18
#>   5 2013-01-24  2013     1    24      1      0          27.5        33.9      24
#>   6 2013-01-30  2013     1    30      1      0.6        26.6        26.5      30
#>   7 2013-02-05  2013     2     5      1      0.1        25.7        66.9      36
#>   8 2013-02-11  2013     2    11      1      8          22.2       118.       42
#>   9 2013-02-17  2013     2    17      1     13.6        26.3        75        48
#>  10 2013-02-23  2013     2    23      1      7.2        24.8       311.       54
#>  # … with 334 more rows, and 3 more variables: month_num <int>,
#>  #   month_name <chr>, season_name <chr>
```

---

## Data structures

### data.frame/tibble

We can take a quick look at the data types of each column

.pull-left[

```r
str(beaches)
```

```
#>  'data.frame':	344 obs. of  12 variables:
#>   $ date       : IDate, format: "2013-01-02" "2013-01-06" ...
#>   $ year       : int  2013 2013 2013 2013 2013 2013 2013 2013 2013 2013 ...
#>   $ month      : int  1 1 1 1 1 1 2 2 2 2 ...
#>   $ day        : int  2 6 12 18 24 30 5 11 17 23 ...
#>   $ season     : int  1 1 1 1 1 1 1 1 1 1 ...
#>   $ rainfall   : num  0 0 0 0 0 0.6 0.1 8 13.6 7.2 ...
#>   $ temperature: num  23.4 30.3 31.4 46.4 27.5 26.6 25.7 22.2 26.3 24.8 ...
#>   $ enterococci: num  6.7 2 69.1 9 33.9 ...
#>   $ day_num    : int  2 6 12 18 24 30 36 42 48 54 ...
#>   $ month_num  : int  1 1 1 1 1 1 2 2 2 2 ...
#>   $ month_name : chr  "January" "January" "January" "January" ...
#>   $ season_name: chr  "Summer" "Summer" "Summer" "Summer" ...
```
]
.pull-right[

```r
glimpse(beaches)
```

```
#>  Rows: 344
#>  Columns: 12
#>  $ date        <date> 2013-01-02, 2013-01-06, 2013-01-12, 2013-01-18, 2013-01-…
#>  $ year        <int> 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, 201…
#>  $ month       <int> 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, …
#>  $ day         <int> 2, 6, 12, 18, 24, 30, 5, 11, 17, 23, 1, 7, 13, 19, 25, 2,…
#>  $ season      <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, …
#>  $ rainfall    <dbl> 0.0, 0.0, 0.0, 0.0, 0.0, 0.6, 0.1, 8.0, 13.6, 7.2, 28.6, …
#>  $ temperature <dbl> 23.4, 30.3, 31.4, 46.4, 27.5, 26.6, 25.7, 22.2, 26.3, 24.…
#>  $ enterococci <dbl> 6.700000, 2.000000, 69.100000, 9.000000, 33.900000, 26.50…
#>  $ day_num     <int> 2, 6, 12, 18, 24, 30, 36, 42, 48, 54, 60, 66, 72, 78, 84,…
#>  $ month_num   <int> 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, …
#>  $ month_name  <chr> "January", "January", "January", "January", "January", "J…
#>  $ season_name <chr> "Summer", "Summer", "Summer", "Summer", "Summer", "Summer…
```

]
---

## Data structures

### data.frame/tibble

Extracting columns from a data frame:

1. .fatinline[`beaches$temperature`] (works during exploration or when only single column needed)
1. .heatinline[`beaches[,'temperature']`] (*This is my preferred method*)
1. `beaches[['temperature']]`
1. `beaches[,7]`
1. `beaches[[7]]`
---

## Packages in R

R is a modular environment with some base functionality that is augmented by __packages__ (think of them as modules)

- Packages can contain _functions_ and _data_
- There are over 15K packages on CRAN, the Comprehensive R Archive Network
- There are over 1600 packages on Bioconductor, the main repository for bioinformatics resources
    - Analytic, Annotation, Experimental data and Workflow packages

#### Finding packages

1) CRAN [Task views](https://cran.r-project.org/web/views/)  
2) Bioconductor [BiocViews](http://bioconductor.org/packages/release/BiocViews.html)  
3) GitHub (open source collaboration and version control environment)

---

## Installing packages

.pull-left[
### From CRAN

```r
install.packages("tidyverse")
```

### From Bioconductor

```r
install.packages("BiocManager") # do once 
BioCManager::install('limma')
```

### From GitHub

```r
install.packages('remotes') # do once
remotes::install_github("rstudio/rmarkdown") 
# usual format is username/packagename
```
]
.pull-right[
> GitHub often hosts development version of packages published on CRAN or Bioconductor

> Both CRAN and Bioconductor have stringent checks to make sure packages can run properly, with no obvious program flaws. There are typically no
  guarantees about analytic or theoretical correctness, but most packages have been crowd-validated and there are several reliable developer groups
  including RStudio
]

---

## Using packages

You have to first "activate" the package in your current working session using the `library` function.

.pull-left[

```r
ggplot(beaches, aes(temperature, rainfall)) +
  geom_point()
```

```
#>  Error in ggplot(beaches, aes(temperature, rainfall)): could not find function "ggplot"
```

The first thing you look for in this error message is 
which package isn't loaded. Then either load it, or
install it, as the case may be<p>

Recall, you have to  
install packages once .fatinline[_per computer_],
but  
load packages once .heatinline[_per session_]
]
--

.pull-right[

```r
library(ggplot2) # or library(tidyverse)
ggplot(beaches, aes(temperature, rainfall)) +
  geom_point()
```

![](03-Rintro_files/figure-html/Rintro-15-1.png)

]

---
class: center, middle

# Tidying data using the tidyverse

---

## What is the ["Tidyverse"](http://www.tidyverse.org)?

The tidyverse is an opinionated collection of R packages designed for data science. All packages share an underlying design philosophy, grammar, and data structures

</div>

---

## What is the "Tidyverse"?

A set of R packages that:

<ul>
  <li> help make data more computer-friendly
  <li>while making your code more human-friendly
</ul>

- Most of these packages are (co-)written by Dr. Hadley Wickham, who has rockstar status in the R world
- They are supported by the company RStudio

----
The [tidyverse.org](https://www.tidyverse.org) site and the [R4DS book](https://r4ds.had.co.nz) 
are the definitive sources for tidyverse information.<p>
The packages are united in a common philosophy of how data analysis should be done.

---
class:middle, center

# Tidying data

---

## Tidy data

Tidy datasets are all alike, <br/>
but every messy data is messy in its own way

</div>

---

## Tidy data

Tidy data is a **computer-friendly** format based on the following characteristics:

- Each row is one observation
- Each column is one variable
- Each set of observational unit forms a table

All other forms of data can be considered **messy data**.

---

## Let us count the ways

There are many ways data can be messy. An incomplete list....

+ Column headers are values, not variables
+ Multiple variables are stored in a single column
+ Variables are stored in both rows and columns
+ Multiple types of observational units are saved in the same table
+ A single observational unit is stored in multiple tables

---

## Ways to have messy (i.e. not tidy) data

1. Column headers contain values

Country   |   < $10K    | $10-20K    | $20-50K   | $50-100K    | > $100K
----------|-------------|------------|-----------|-------------|---------
India     |   40        |  25        |   25      |  9          |  1
USA       |   20        |  20        |  20       | 30          |  10

---

## Ways to have messy (i.e. not tidy) data

Column headers contain values

Country   |   Income  | Percentage
----------|-----------|------------
India     |  < $10K   |  40
USA       |  < $10K   | 20

This is a case of reshaping or melting

---

## Ways to have messy (i.e. not tidy) data

Multiple variables in one column

Country  | Year   | M_0-14  | F_0-14  | M_ 15-60  | F_15-60  | M_60+  | F_60+
---------|--------|---------|---------|-----------|----------|--------|-------
UK       |  2010  |         |         |           |          |        | 
UK       |  2011  |         |         |           |          |        |

<p>
Separating columns into different variables

Country  | Year   | Gender  | Age    | Count
---------|--------|---------|--------|-------

---

## Tidying data

The typical steps are

+ Transforming data from wide to tall (`pivot_longer`) and from tall to wide (`pivot_wider`)
+ Separating columns into different columns (`separate`)
+ Putting columns together into new variables (`unite`)

----
>The functions `pivot_longer` and `pivot_wider` supercede the older functions `gather` and `spread`, 
which I have used in previous iterations of this class. However, if you are familiar with `gather` and `spread`, they aren't gone and can still be used in the current **tidyr** package.

---
class: middle, center

# Cleaning data

---

## Some actions on data

+ Creating new variables (`mutate`)
+ Choose some columns (`select`)
+ Selecting rows based on some criteria (`filter`)
+ Sort data based on some variables (`arrange`)
+ Reduce multiple values to a single summary (`summarize`/`summarise`)
+ Change the order of rows (`arrange`)

---

## Example data

```r
head(mtcars, 3)
```

```
#>                 mpg cyl disp  hp drat    wt  qsec vs am gear carb
#>  Mazda RX4     21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
#>  Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
#>  Datsun 710    22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
```

- Car names are in an attribute of the data.frame called `rownames`. So it's not in a column
- We might want to convert fuel economy to metric 
- We might just want to look at the relationship between displacement and fuel economy based on number of cylinders

----
The function `tibble::rownames_to_column` makes short work of the first point. <p>
Row names are an older construct to give labels to each row. They are strongly discouraged outside
of indices since using them as data created problems.
---

## Example data ([link](https://dl.dropboxusercontent.com/s/pqavhcckshqxtjm/brca.csv))

```r
link <- 'https://dl.dropboxusercontent.com/s/pqavhcckshqxtjm/brca.csv'
brca_data <- rio::import(link)
```

![](img/brca_data.png)

---

## The `tidyverse` package

The `tidyverse` package is a meta-package that installs a set of packages that are useful for data 
cleaning, data tidying and data munging (manipulating data to get a computationally "attractive" 
dataset)

---

## The `tidyverse` package

```r
# install.packages('tidyverse')
library(tidyverse)
```

#### Core `tidyverse` packages

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Package </th>
   <th style="text-align:left;"> Description </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> ggplot2 </td>
   <td style="text-align:left;"> Data visualization </td>
  </tr>
  <tr>
   <td style="text-align:left;"> tibble </td>
   <td style="text-align:left;"> data.frame on steroids </td>
  </tr>
  <tr>
   <td style="text-align:left;"> tidyr </td>
   <td style="text-align:left;"> Data tidying (today) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> readr </td>
   <td style="text-align:left;"> Reading text files (CSV) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> purrr </td>
   <td style="text-align:left;"> Applying functions to data iteratively </td>
  </tr>
  <tr>
   <td style="text-align:left;"> dplyr </td>
   <td style="text-align:left;"> Data cleaning and munging (today) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> stringr </td>
   <td style="text-align:left;"> String (character) manipulation </td>
  </tr>
  <tr>
   <td style="text-align:left;"> forcats </td>
   <td style="text-align:left;"> Manipulating categorical variables </td>
  </tr>
</tbody>
</table>

---

## Additional `tidyverse` packages

### Additional useful packages

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Package </th>
   <th style="text-align:left;"> Description </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> broom </td>
   <td style="text-align:left;"> Turns the results of models or analysis into tidy datasets </td>
  </tr>
  <tr>
   <td style="text-align:left;"> fs </td>
   <td style="text-align:left;"> Allows directory and file manipulation in OS-agnostic manner </td>
  </tr>
  <tr>
   <td style="text-align:left;"> here </td>
   <td style="text-align:left;"> Allows robust specification of directory structure in a Project </td>
  </tr>
</tbody>
</table>

---
class: middle, center

# Pipes

---

## Pipes

Pipes (denoted `%>%`, spoken as "then") are to analytic pipelines as `+` is to `ggplot` layers

```r
mpg1 <- mpg %>% mutate(id=1:n()) %>% select(id, year, trans, cty, hwy)
mpg_metric <- mpg1 %>% 
  mutate(across(c(cty, hwy), function(x) {x * 1.6/3.8}))
```

.pull-left[

Original data

<table>
 <thead>
  <tr>
   <th style="text-align:right;"> id </th>
   <th style="text-align:right;"> year </th>
   <th style="text-align:left;"> trans </th>
   <th style="text-align:right;"> cty </th>
   <th style="text-align:right;"> hwy </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1999 </td>
   <td style="text-align:left;"> auto(l5) </td>
   <td style="text-align:right;"> 18 </td>
   <td style="text-align:right;"> 29 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1999 </td>
   <td style="text-align:left;"> manual(m5) </td>
   <td style="text-align:right;"> 21 </td>
   <td style="text-align:right;"> 29 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2008 </td>
   <td style="text-align:left;"> manual(m6) </td>
   <td style="text-align:right;"> 20 </td>
   <td style="text-align:right;"> 31 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2008 </td>
   <td style="text-align:left;"> auto(av) </td>
   <td style="text-align:right;"> 21 </td>
   <td style="text-align:right;"> 30 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 5 </td>
   <td style="text-align:right;"> 1999 </td>
   <td style="text-align:left;"> auto(l5) </td>
   <td style="text-align:right;"> 16 </td>
   <td style="text-align:right;"> 26 </td>
  </tr>
</tbody>
</table>

]
.pull-right[

Transformed data
<table>
 <thead>
  <tr>
   <th style="text-align:right;"> id </th>
   <th style="text-align:right;"> year </th>
   <th style="text-align:left;"> trans </th>
   <th style="text-align:right;"> cty </th>
   <th style="text-align:right;"> hwy </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1999 </td>
   <td style="text-align:left;"> auto(l5) </td>
   <td style="text-align:right;"> 7.578947 </td>
   <td style="text-align:right;"> 12.21053 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1999 </td>
   <td style="text-align:left;"> manual(m5) </td>
   <td style="text-align:right;"> 8.842105 </td>
   <td style="text-align:right;"> 12.21053 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2008 </td>
   <td style="text-align:left;"> manual(m6) </td>
   <td style="text-align:right;"> 8.421053 </td>
   <td style="text-align:right;"> 13.05263 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2008 </td>
   <td style="text-align:left;"> auto(av) </td>
   <td style="text-align:right;"> 8.842105 </td>
   <td style="text-align:right;"> 12.63158 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 5 </td>
   <td style="text-align:right;"> 1999 </td>
   <td style="text-align:left;"> auto(l5) </td>
   <td style="text-align:right;"> 6.736842 </td>
   <td style="text-align:right;"> 10.94737 </td>
  </tr>
</tbody>
</table>

]

Note I'm assigning a name to the transformed data. Otherwise it'll be lost

---

## Verbs to use in pipes

The verbs in `tidyverse` are specially useful in pipes

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Verb </th>
   <th style="text-align:left;"> Functionality </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> mutate </td>
   <td style="text-align:left;"> Transform a column with some function </td>
  </tr>
  <tr>
   <td style="text-align:left;"> select </td>
   <td style="text-align:left;"> Select some columns in the data </td>
  </tr>
  <tr>
   <td style="text-align:left;"> arrange </td>
   <td style="text-align:left;"> Order the data frame by values of a column(s) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> filter </td>
   <td style="text-align:left;"> Keep only rows that meet some data criterion </td>
  </tr>
  <tr>
   <td style="text-align:left;"> group_by </td>
   <td style="text-align:left;"> Group by levels of a variable </td>
  </tr>
  <tr>
   <td style="text-align:left;"> pivot_longer </td>
   <td style="text-align:left;"> Transform a wide dataset to a long dataset </td>
  </tr>
  <tr>
   <td style="text-align:left;"> pivot_wider </td>
   <td style="text-align:left;"> Transform a long dataset to a wide dataset </td>
  </tr>
  <tr>
   <td style="text-align:left;"> separate </td>
   <td style="text-align:left;"> Separate one column into several columns </td>
  </tr>
  <tr>
   <td style="text-align:left;"> unite </td>
   <td style="text-align:left;"> Concatenate several columns into 1 column </td>
  </tr>
</tbody>
</table>

Pipes almost always start with a data.frame/tibble object, and then "pipes" that data through different transformations (functions)

At each `%>%`, the results of the previous step are used as input for the next step.

---

## A new verb: `across`

**dplyr** version 1.0 introduced a new verb, `across`. It supercedes the older scoped verbs
`*_at`, `*_if` and `*_all`, where `*` might be `mutate` or `summarise`. Use whichever makes sense to you.

.pull-left[

```r
beaches %>% 
  summarise_if(is.numeric, mean)
```

```
#>        year    month      day  season rainfall temperature enterococci  day_num
#>  1 2015.494 6.377907 15.88953 2.52907       NA    23.57994          NA 178.7994
#>    month_num
#>  1  36.30814
```

```r
beaches %>% 
  group_by(season_name) %>% 
  summarise_at(vars(rainfall, enterococci),
               median)
```

```
#>  # A tibble: 4 x 3
#>    season_name rainfall enterococci
#>  * <chr>          <dbl>       <dbl>
#>  1 Autumn             0          NA
#>  2 Spring            NA          NA
#>  3 Summer             0          NA
#>  4 Winter             0          NA
```
]
.pull-right[

```r
beaches %>% 
  summarise(across(where(is.numeric), mean))
```

```r
beaches %>% 
  group_by(season_name) %>% 
  summarise(across(c(rainfall, enterococci),
                   median))
```

```
#>  # A tibble: 4 x 3
#>    season_name rainfall enterococci
#>  * <chr>          <dbl>       <dbl>
#>  1 Autumn             0          NA
#>  2 Spring            NA          NA
#>  3 Summer             0          NA
#>  4 Winter             0          NA
```

]

---

## `across` (Updated Spring 2021)

The `across` function is now the more flexible standard way to apply most dplyr 
verbs across multiple columns, which can be selected the same way you would in `select`. In 
addition to what's on the previous slide, here are some more examples.

.pull-left[

```r
summaries <- list(
  mean = ~mean(.x, na.rm=T), 
  median = ~median(.x, na.rm=T)
) # both the ~ and .x are required

beaches %>% 
  summarise(across(.cols = where(is.numeric), 
                   .fns = summaries))
```

```
#>    year_mean year_median month_mean month_median day_mean day_median season_mean
#>  1  2015.494        2016   6.377907            6 15.88953         16     2.52907
#>    season_median rainfall_mean rainfall_median temperature_mean
#>  1             3      4.191228               0         23.57994
#>    temperature_median enterococci_mean enterococci_median day_num_mean
#>  1               23.3         34.59322           9.372727     178.7994
#>    day_num_median month_num_mean month_num_median
#>  1            177       36.30814               37
```

]
.pull-right[

```r
beaches %>% 
  summarise(
    across(where(is.numeric), ~mean(.x, na.rm=T), .names='mean_{.col}'),
    across(where(is.numeric), ~median(.x, na.rm=T), .anems='median_{.col}'))
```

```
#>    mean_year mean_month mean_day mean_season mean_rainfall mean_temperature
#>  1  2015.494   6.377907 15.88953     2.52907      4.191228         23.57994
#>    mean_enterococci mean_day_num mean_month_num year month day season rainfall
#>  1         34.59322     178.7994       36.30814 2016     6  16      3        0
#>    temperature enterococci day_num month_num
#>  1        23.3    9.372727     177        37
```

This groups the columns by function rather than by variable

]
---

## `across` (Updated Spring 2021)

.pull-left[

```r
beaches %>% 
  filter(across(everything(), ~!is.na(.x))) %>% 
  head() # rows with no missing values
```

]

.pull-right[

#### For more examples see https://dplyr.tidyverse.org/articles/colwise.html
]
---
class: middle, center

# Modeling and the broom package

---

## Representing model relationships

In R, there is a particularly convenient way to express models, where you have

- one dependent variable
- one or more independent variables, with possible transformations and interactions

```
y ~ x1 + x2 + x1:x2 + I(x3^2) + x4*x5
```

---

## Representing model relationships

```
y ~ x1 + x2 + x1:x2 + I(x3^2) + x4*x5
```

`y` depends on ...

- `x1` and `x2` linearly
- the interaction of `x1` and `x2` (represented as `x1:x2`)
- the square of `x3` (the `I()` notation ensures that the `^` symbol is interpreted correctly)
- `x4`, `x5` and their interaction (same as `x4 + x5 + x4:x5`)

---

# Representing model relationships

```
y ~ x1 + x2 + x1:x2 + I(x3^2) + x4*x5
```

This interpretation holds for the vast majority of statistical models in R

- For decision trees and random forests and neural networks, don't add interactions or transformations, since the model will try to figure those out on their own

---

# Our first model

```r
library(survival)
myLinearModel <- lm(chol ~ bili, data = pbc)
```

Note that everything in R is an **object**, so you can store a model in a variable name.

This statement runs the model and stored the fitted model in `myLinearModel`

R does not interpret the model, evaluate the adequacy or appropriateness of the model, or comment on whether looking at the relationship between cholesterol and bilirubin makes any kind of sense.

---

# Our first model

```r
myLinearModel
```

```
#>  
#>  Call:
#>  lm(formula = chol ~ bili, data = pbc)
#>  
#>  Coefficients:
#>  (Intercept)         bili  
#>       303.20        20.24
```

> Not very informative, is it?

---

# Our first model

```r
summary(myLinearModel)
```

```
#>  
#>  Call:
#>  lm(formula = chol ~ bili, data = pbc)
#>  
#>  Residuals:
#>      Min      1Q  Median      3Q     Max 
#>  -565.39  -89.90  -35.36   44.92 1285.33 
#>  
#>  Coefficients:
#>              Estimate Std. Error t value Pr(>|t|)    
#>  (Intercept)  303.204     15.601  19.435  < 2e-16 ***
#>  bili          20.240      2.785   7.267 3.63e-12 ***
#>  ---
#>  Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>  
#>  Residual standard error: 213.2 on 282 degrees of freedom
#>    (134 observations deleted due to missingness)
#>  Multiple R-squared:  0.1577,	Adjusted R-squared:  0.1547 
#>  F-statistic:  52.8 on 1 and 282 DF,  p-value: 3.628e-12
```

> A little better

???

Talk about the different metrics in this slide

---

# Our first model

```r
broom::tidy(myLinearModel)
```

```
#>  # A tibble: 2 x 5
#>    term        estimate std.error statistic  p.value
#>    <chr>          <dbl>     <dbl>     <dbl>    <dbl>
#>  1 (Intercept)    303.      15.6      19.4  5.65e-54
#>  2 bili            20.2      2.79      7.27 3.63e-12
```

```r
broom::glance(myLinearModel)
```

```
#>  # A tibble: 1 x 12
#>    r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
#>        <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>
#>  1     0.158         0.155  213.      52.8 3.63e-12     1 -1925. 3856. 3867.
#>  # … with 3 more variables: deviance <dbl>, df.residual <int>, nobs <int>
```

---

## Displaying model results

Let's start with this model:

```r
myModel <- lm(log10(enterococci) ~ rainfall + temperature + season_name + factor(year), data = beaches)
broom::tidy(myModel)
```

```
#>  # A tibble: 11 x 5
#>     term              estimate std.error statistic  p.value
#>     <chr>                <dbl>     <dbl>     <dbl>    <dbl>
#>   1 (Intercept)        1.23      0.208       5.92  8.39e- 9
#>   2 rainfall           0.0284    0.00296     9.58  2.72e-19
#>   3 temperature       -0.00830   0.00769    -1.08  2.81e- 1
#>   4 season_nameSpring -0.325     0.0840     -3.87  1.31e- 4
#>   5 season_nameSummer  0.0862    0.0909      0.948 3.44e- 1
#>   6 season_nameWinter -0.332     0.0889     -3.74  2.22e- 4
#>   7 factor(year)2014   0.0498    0.102       0.486 6.27e- 1
#>   8 factor(year)2015   0.0807    0.100       0.804 4.22e- 1
#>   9 factor(year)2016   0.0815    0.0975      0.836 4.04e- 1
#>  10 factor(year)2017  -0.0676    0.0972     -0.696 4.87e- 1
#>  11 factor(year)2018   0.0440    0.107       0.411 6.81e- 1
```

---

# Displaying model results

Let's start with this model:

```r
plt_data <- broom::tidy(myModel)
plt_data <- plt_data %>% 
* filter(term != '(Intercept)') %>%
  mutate(term = str_replace(term, 
*                           'season_name',''))
plt_data
```

```
#>  # A tibble: 10 x 5
#>     term             estimate std.error statistic  p.value
#>     <chr>               <dbl>     <dbl>     <dbl>    <dbl>
#>   1 rainfall          0.0284    0.00296     9.58  2.72e-19
#>   2 temperature      -0.00830   0.00769    -1.08  2.81e- 1
#>   3 Spring           -0.325     0.0840     -3.87  1.31e- 4
#>   4 Summer            0.0862    0.0909      0.948 3.44e- 1
#>   5 Winter           -0.332     0.0889     -3.74  2.22e- 4
#>   6 factor(year)2014  0.0498    0.102       0.486 6.27e- 1
#>   7 factor(year)2015  0.0807    0.100       0.804 4.22e- 1
#>   8 factor(year)2016  0.0815    0.0975      0.836 4.04e- 1
#>   9 factor(year)2017 -0.0676    0.0972     -0.696 4.87e- 1
#>  10 factor(year)2018  0.0440    0.107       0.411 6.81e- 1
```

---

# Displaying model results

Let's start with this model:

```r
plt_data <- broom::tidy(myModel)
plt_data <- plt_data %>% 
  filter(term != '(Intercept)') %>% 
  mutate(term = str_replace(term, 
                            'season_name','')) %>% 
  mutate(term = str_replace(term, 
*                           'factor\$year\$','')) # Brackets are "escaped" using \\
plt_data
```

```
#>  # A tibble: 10 x 5
#>     term        estimate std.error statistic  p.value
#>     <chr>          <dbl>     <dbl>     <dbl>    <dbl>
#>   1 rainfall     0.0284    0.00296     9.58  2.72e-19
#>   2 temperature -0.00830   0.00769    -1.08  2.81e- 1
#>   3 Spring      -0.325     0.0840     -3.87  1.31e- 4
#>   4 Summer       0.0862    0.0909      0.948 3.44e- 1
#>   5 Winter      -0.332     0.0889     -3.74  2.22e- 4
#>   6 2014         0.0498    0.102       0.486 6.27e- 1
#>   7 2015         0.0807    0.100       0.804 4.22e- 1
#>   8 2016         0.0815    0.0975      0.836 4.04e- 1
#>   9 2017        -0.0676    0.0972     -0.696 4.87e- 1
#>  10 2018         0.0440    0.107       0.411 6.81e- 1
```

---

# Displaying model results

Let's start with this model:

.pull-left[

```r
plt_data <- broom::tidy(myModel)
plt_data %>% 
  filter(term != '(Intercept)') %>% 
  mutate(term = str_replace(term, 
                    'season_name','')) %>% 
  mutate(term = str_replace(term, 
                    'factor\$year\$','')) %>% # Brackets need to be "escaped" using \\
  ggplot(aes(x = term, y = estimate, 
             ymin = estimate - 2 * std.error, 
             ymax = estimate + 2 * std.error))+
    geom_pointrange()
```
]
.pull-right[
![](03-Rintro_files/figure-html/Rintro-44-1.png)

]

---

# Displaying model results

Let's start with this model:

.pull-left[

```r
plt_data <- broom::tidy(myModel)
plt_data %>% 
  filter(term != '(Intercept)') %>% 
  mutate(term = str_replace(term, 
                  'season_name','')) %>% 
  mutate(term = str_replace(term, 
                  'factor\$year\$','')) %>% # Brackets need to be "escaped" using \\
  ggplot(aes(x = term, y = estimate, 
             ymin = estimate - 2 * std.error, 
             ymax = estimate + 2 * std.error))+
    geom_pointrange() + 
    geom_hline(yintercept = 0, linetype=2) + 
    theme_bw() + 
    coord_flip()
```
]
.pull-right[
![](03-Rintro_files/figure-html/Rintro-45-1.png)

]
---

# Displaying model results

Let's start with this model:

.pull-left[

```r
plt_data <- broom::tidy(myModel)
plt_data %>% 
  filter(term != '(Intercept)') %>% 
  mutate(term = str_replace(term, 
                  'season_name','')) %>% 
  mutate(term = str_replace(term, 
                  'factor\$year\$','')) %>% # Brackets need to be "escaped" using \\
  ggplot(aes(x = term, y = estimate, 
             ymin = estimate - 2 * std.error, 
             ymax = estimate + 2 * std.error))+
    geom_pointrange() + 
    geom_hline(yintercept = 0, linetype=2) + 
    theme_bw() + 
    coord_flip()

ggsave('results.png') # ggsave knows format from file
```

You can also save the graph from the RStudio Plots pane, but coding it using `ggsave` is more reproducible

]
.pull-right[
![](03-Rintro_files/figure-html/Rintro-47-1.png)

If you need to get a high-definition TIFF on a Mac, your best bet is to save your graph as a PDF and then convert it using Acrobat or other scripts (ask me if interested). The TIFF printer in R only creates 72 DPI TIFF files
]

---