A set of R packages that:

• Most of these packages are (co-)written by Dr. Hadley Wickham, who has rockstar status in the R world
• They are supported by the company RStudio

Tidy data is a computer-friendly format based on the following characteristics:

• Each row is one observation
• Each column is one variable
• Each set of observational unit forms a table

All other forms of data can be considered messy data.

There are many ways data can be messy. An incomplete list….

• Column headers are values, not variables
• Multiple variables are stored in a single column
• Variables are stored in both rows and columns
• Multiple types of observational units are saved in the same table
• A single observational unit is stored in multiple tables

1. Column headers contain values

1. Column headers contain values

This is a case of reshaping or melting

1. Multiple variables in one column

Separating columns into different variables

The typical steps are

• Transforming data from wide to tall (gather) and from tall to wide (spread)
• Separating columns into different columns
• Putting columns together into new variables

• Creating new variables (mutate)
• Choose some columns (select)
• Selecting rows based on some criteria (filter)
• Sort data based on some variables (arrange)

data(mtcars)
knitr::kable(head(mtcars, 3))

• Car names are in an attribute of the data.frame called rownames. So it’s not in a column
• We might want to convert fuel economy to metric
• We might just want to look at the relationship between displacement and fuel economy based on number of cylinders

link <- 'https://dl.dropboxusercontent.com/s/pqavhcckshqxtjm/brca.csv'
download.file(link, 'brca.csv')
brca_data <- read.csv('brca.csv', stringsAsFactors=FALSE)

The tidyverse package is a meta-package that installs a set of packages that are useful for data cleaning, data tidying and data munging (manipulating data to get a computationally “attractive” dataset)

# install.packages('tidyverse')
library(tidyverse)

## ── Attaching packages ────────────────────────── tidyverse 1.2.1 ──

## ✔ ggplot2 3.0.0     ✔ purrr   0.2.5
## ✔ tibble  1.4.2     ✔ dplyr   0.7.6
## ✔ tidyr   0.8.1     ✔ stringr 1.3.1
## ✔ readr   1.1.1     ✔ forcats 0.3.0

## ── Conflicts ───────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()

library(tidyverse)
updated_cars <- 
  mtcars %>% 
  rownames_to_column(var = 'Model') %>% 
  mutate(kmpg = mpg * 1.6) %>% 
  select(Model, kmpg, cyl, disp) %>% 
  filter(cyl == 6)

library(tidyverse)
updated_cars <- 
  mtcars %>% # Take the data set
  rownames_to_column(var = 'Model') %>% # Make rownames a column
  select(Model, mpg, cyl, disp) %>% # Keep only certain columns
  mutate(kmpg = mpg * 1.6) %>% # Create a new variable
  filter(cyl == 6) # Keep only certain rows

The idea is to use verbs to express operations on a dataset, so it is easier to express what you want to do in code.

The pipe operator %>% (technically from the package magrittr) takes a data.frame or tibble object on the left, then “pipes” it to a function that takes the data.frame object as its first argument.

mtcars %>% mutate(kmpg = mpg * 1.6)

would be the same as

mutate(mtcars, kmpg = mpg * 1.6)

With pipes

mtcars %>% 
  rownames_to_column(var = "Model") %>% 
  select(Model:disp) %>% 
  mutate(kmpg = mpg * 1.6) %>% 
  filter(cyl == 6)

Without pipes

tmp <- rownames_to_column(mtcars, var="Model")
tmp3 <- select(tmp2, Model:disp)
tmp2 <- mutate(tmp, kmpg = mpg * 1.6)
tmp4 <- filter(tmp3, cyl == 6)

Both are fine, but I find pipes help translating my thoughts into code better

With pipes + tidyverse

updated_cars <- mtcars %>% 
  rownames_to_column(var = "Model") %>% 
  select(Model:disp) %>% 
  mutate(kmpg = mpg * 1.6) %>% 
  filter(cyl == 6)

Without tidyverse

mtcars[,'Model'] <- rownames(mtcars)
tmp <- mtcars[,c('Model','mpg','cyl','disp')] # Can't use the : operator for names
tmp[,'kmpg'] <- tmp[,'mpg'] * 1.6 # Note we need to quote the names
updated_cars <- tmp[tmp[,'cyl'] == 6,] 

Idea was to make it easier to write expressive code without getting too hung up with syntax.

mtcars

mtcars %>% 
  rownames_to_column(var = "Model")

mtcars %>% 
  rownames_to_column(var = "Model") %>% 
  select(Model:disp)

mtcars %>% 
  rownames_to_column(var = "Model") %>% 
  select(Model:disp) %>% 
  mutate(kmpg = mpg * 1.6)

mtcars %>% 
  rownames_to_column(var = "Model") %>% 
  select(Model:disp) %>% 
  mutate(kmpg = mpg * 1.6) %>% 
  filter(cyl == 6)

link <- 'https://dl.dropboxusercontent.com/s/pqavhcckshqxtjm/brca.csv'
download.file(link, 'brca.csv')
brca_data <- read.csv('brca.csv', stringsAsFactors=FALSE)

library(tidyverse)
names(brca_data)

##  [1] "id"                      "diagnosis"              
##  [3] "radius_mean"             "texture_mean"           
##  [5] "perimeter_mean"          "area_mean"              
##  [7] "smoothness_mean"         "compactness_mean"       
##  [9] "concavity_mean"          "concave.points_mean"    
## [11] "symmetry_mean"           "fractal_dimension_mean" 
## [13] "radius_se"               "texture_se"             
## [15] "perimeter_se"            "area_se"                
## [17] "smoothness_se"           "compactness_se"         
## [19] "concavity_se"            "concave.points_se"      
## [21] "symmetry_se"             "fractal_dimension_se"   
## [23] "radius_worst"            "texture_worst"          
## [25] "perimeter_worst"         "area_worst"             
## [27] "smoothness_worst"        "compactness_worst"      
## [29] "concavity_worst"         "concave.points_worst"   
## [31] "symmetry_worst"          "fractal_dimension_worst"
## [33] "X"

1. For the same variable (e.g., radius) there are 3 columns giving the mean, se and worst value. So the names of the metric are stored in the column names
2. There are really 3 kinds of summaries for each metric – mean, se and worst.

library(tidyverse)
brca_data

library(tidyverse)
brca_data %>% 
  select(id, diagnosis, ends_with('mean'), ends_with('se'), ends_with('worst'),
         -starts_with('fractal')) 
                             # removes columns starting with "fractal"

library(tidyverse)
brca_data %>% 
  select(id, diagnosis, ends_with('mean'),
         ends_with('se'), ends_with('worst'),
         -starts_with('fractal')) %>% 
  gather(variable, value, -id, -diagnosis) # operate on everything but

Column names become variable, everything stays aligned with id and diagnosis

library(tidyverse)
brca_data %>% 
  select(id, diagnosis, ends_with('mean'),
         ends_with('se'), ends_with('worst'),
         -starts_with('fractal')) %>% 
  gather(variable, value, -id, -diagnosis) %>% 
  separate(variable, c("Variable","stat"), sep="_", remove = T) 

Split variable into 2 cols, Variable and stat

library(tidyverse)
brca_data %>% 
  select(id, diagnosis, ends_with('mean'),
         ends_with('se'), ends_with('worst'),
         -starts_with('fractal')) %>% 
  gather(variable, value, -id, -diagnosis) %>% 
  separate(variable, c("Variable","stat"), sep="_", remove = T) %>% 
  spread(stat, value)

1. Introduction to dplyr
2. Tidy data
3. The paper on tidy data