23-hierarchical_data.Rmd

# Hierarchical data

**Learning objectives:**

1. Introduce a new data structure called lists
2. Learn how to unnest nested lists
3. Get acquainted with JSON

## Packages

```{r message=FALSE}
library(dplyr)
library(tidyr)
library(purrr)
library(repurrrsive)
library(jsonlite)
```

## Introduction to lists {-}

**Vectors**:

- all elements are of the same data type
- flat structure
- support naming

```{r}
c("string 1", "string 2")
70:79
c(mango = 100, banana = 200)
```

## Introduction to lists {-}

**Lists**:

- elements can be of different data types
- elements can be _any R object_ (model object, plot, ...)
- support nesting
- support naming

## Introduction to lists {-}

```{r}
mylist <- list("a", TRUE, lubridate::today())
mylist
str(mylist)
mylist[2:3] # list subsetting (shortens the list)
mylist[[3]] # element extraction (one level deeper)
```

## Introduction to lists {-}

A named, nested list.

```{r}
otherlist <- list(
  mango = 100, 
  nice = list(sweet = TRUE, dirty = FALSE)
)
otherlist
str(otherlist)
# element extraction:
otherlist$nice # one level deeper
otherlist[[2]] # one level deeper
otherlist$nice$sweet # two levels deeper
otherlist[[2]][[1]] # two levels deeper
```

## Introduction to lists {-}

Often unnamed.

```{r}
anotherlist <- list("a", list(TRUE, FALSE))
anotherlist
str(anotherlist)
anotherlist[[2]]
anotherlist[[2]][[1]]
```

## Introduction to lists {-}

What is a data frame?

```{r}
mydf <- data.frame(a = c("x", "y", "z"), b = 1:3)
mydf
is.list(mydf)
```

## Introduction to lists {-}

What is the code to regenerate dataframe `mydf` without using `data.frame()`?

```{r}
dput(mydf)
```

- A data frame is just a list of vectors (the 'columns') which has been attributed the 'data.frame' class!
- It has methods defined in base R, e.g. defining how to print a data.frame using `print()`; see `methods(class = "data.frame")`.

## Introduction to lists {-}

Extracting elements using `{purrr}`.

```{r}
str(otherlist)
purrr::pluck(otherlist, "nice", "dirty")
str(anotherlist)
purrr::pluck(anotherlist, 2, 1)
```

## Introduction to lists {-}

If _multiple_ elements are provided to `c()`, it tries to flatten them to one level:

- usually creates a vector
- creates a flat list if one element is a list (can be hierarchical) and another isn't

```{r}
c(mango = 100, nice = list(sweet = TRUE, dirty = FALSE))
c("a", list(TRUE, FALSE))
```


## Tibbles can have a list column {-}

Tibbles: just like a data frame, but the columns can also be lists instead of vectors.

This means that tibbles can have columns with different data types, nested lists, ...

## Tibbles can have a list column {-}

Example:

```{r}
df <- tibble(
  x = 1:2, 
  y = c("a", "b"),
  z = list(list(1, 2), list(3, 4, 5))
)
df
df |> 
  filter(x == 1)
```

## Tibbles can have a list column {-}

Extracting parts of a tibble with a list column:

```{r}
# whole list column:
df |> pull(z) |> str()
df |> pluck("z") |> str()
# elements inside the list column:
df |> pluck("z", 2, 3)
```

- `dplyr::pull()` just takes one variable as argument
- `purrr::pluck()` is designed to also take subsequent nested elements.

## Rectangling data by unnesting list columns {-}

Two cases:

- `tidyr::unnest_longer()`: typically for **unnamed lists**
- `tidyr::unnest_wider()`: typically for **named lists**

## Rectangling data by unnesting list columns {-}

```{r}
df_unnamed <- tribble(
  ~x, ~y,
  1, list(11, 12, 13),
  2, list(21),
  3, list(31, 32),
)
df_unnamed
```

Unnesting an unnamed list column? Make it longer!

## Rectangling data by unnesting list columns {-}

```{r}
df_unnamed |> unnest_longer(y)
```

## Rectangling data by unnesting list columns {-}

```{r}
df_named <- tribble(
  ~x, ~y,
  1, list(a = 11, b = 12),
  2, list(a = 21, b = 22),
  3, list(a = 31, b = 32),
)
df_named
```

Unnesting a named list column?

- Usually you want to make it wider.
- The names become the column names.

## Rectangling data by unnesting list columns {-}

```{r}
df_named |> unnest_wider(y)
```

## Rectangling data: special cases {-}

```{r}
df_unnamed_paired <- tribble(
  ~x, ~y, ~z,
  "a", list("a1", "a2"), list("A1", "A2"),
  "b", list("b1", "b2", "b3"), list("B1", "B2", "B3")
)
df_unnamed_paired
```

## Rectangling data: special cases {-}

```{r}
df_unnamed_paired |> unnest_longer(c(y, z))
```

## Rectangling data: special cases {-}

```{r}
df_unnamed_heterogeneous <- tribble(
  ~x, ~y,
  "a", list(1),
  "b", list("a", TRUE, 5)
)
df_unnamed_heterogeneous
```

## Rectangling data: special cases {-}

```{r}
df_unnamed_heterogeneous |> unnest_longer(y)
```

## Rectangling data: special cases {-}

Want to recycle the original variable name with `dplyr::unnest_wider()`?

```{r}
df_named
str(df_named)
```

## Rectangling data: special cases {-}

```{r}
df_named |> unnest_wider(y, names_sep = "_")
```

## Rectangling data: applications {-}

Considering some real-life situations: data from the `{repurrrsive}` package.

- `repos`: deeply nested list
- `got_chars`: relational data
- `gmaps_cities`: deeply nested tibble

## Rectangling data: applications {-}

```{r}
class(gh_repos)
str(gh_repos, 4, list.len = 2)
```

## Rectangling data: applications {-}

```{r}
gh_repos |> 
  pluck(1, 1) |> 
  names()
```

## Rectangling data: applications {-}

```{r}
gh_repos |> 
  pluck(1, 1, "owner") |> 
  str()
```

## Rectangling data: applications {-}

```{r}
repos <- tibble(x = gh_repos)
repos |> 
  unnest_longer(x) |> 
  unnest_wider(x) |> 
  select(id, full_name, owner, description) |> 
  unnest_wider(owner, names_sep = "_") |> 
  glimpse()
```


## Rectangling data: applications {-}

```{r}
str(got_chars, 2, list.len = 3)
```

## Rectangling data: applications {-}

```{r}
chars <- tibble(json = got_chars)
chars
```
 
## Rectangling data: applications {-}

```{r}
chars |>
  unnest_wider(json)
```

## Rectangling data: applications {-}

```{r}
chars |>
  unnest_wider(json) |>
  select(id, where(is.list))
```

## Rectangling data: applications {-}

```{r}
chars |> 
  unnest_wider(json) |> 
  select(id, titles) |> 
  unnest_longer(titles)
```

## Rectangling data: applications {-}

```{r echo=FALSE}
locations <- gmaps_cities |> 
  unnest_wider(json) |> 
  select(-status) |> 
  unnest_longer(results) |> 
  unnest_wider(results)
```

`locations`: result from several unnesting operations on `gmaps_cities`.

```{r}
locations |> 
  select(city, formatted_address, geometry)
```

## Rectangling data: applications {-}

```{r}
locations$geometry[[1]] |> str()
```

## Rectangling data: applications {-}

What we want wrt the geometry column:

- for each element, only unnest all contents of `bounds` list
- using custom column names
- do it all in one step!!

So we need something similar to `purrr::pluck()`, but for unnesting.

```{r}
locations$geometry[[1]] |> str()
```

## Rectangling data: applications {-}

Rectangling specific elements of a nested list: use `tidyr::hoist()`!

```{r}
locations |> 
  select(city, formatted_address, geometry) |> 
  hoist(
    geometry,
    ne_lat = c("bounds", "northeast", "lat"),
    sw_lat = c("bounds", "southwest", "lat"),
    ne_lng = c("bounds", "northeast", "lng"),
    sw_lng = c("bounds", "southwest", "lng"),
  ) |> 
  select(!geometry)
```

## JSON {-}

- JSON is a string format to store hierarchical data
- **j**ava**s**cript **o**bject **n**otation
- It is the format used by most web APIs to return data


## JSON {-}

Limited set of data types!

- **`null`**: same as `NA`
- data types that can only represent a single value:
  - **string**: double qoutes!
  - **number**:
    - can be decimal, integer or scientific
    - doesn’t support `Inf`, `-Inf`, or `NaN`
  - **boolean**: lowercase `true` or `false`

## JSON {-}

- data types to represent multiple values:
  - **arrays**:
    - like an unnamed list in R
    - written with `[]`
    - e.g. `[null, 1, "string", false]`
  - **objects**:
    - like a named list in R
    - written with `{}`
    - the names (keys in JSON terminology) are strings, so must be surrounded by quotes
    - e.g. `{"x": 1, "y": 2}`

## Get JSON into R {-}

Reading a JSON string or file as a list in R:

- `jsonlite::read_json(<filepath>)`
- `jsonlite::parse_json(<string>)`

## Get JSON into R {-}

Example.

```{r}
json <- '[
  {"name": "John", "age": 34},
  {"name": "Susan", "age": 27}
]'
df <- tibble(json = parse_json(json))
df
df |> 
  unnest_wider(json)
```

## JSON and data frames {-}

There are two ways to encode a data frame as JSON:

- an _object_ of (named) columns

```json
{
  "x": ["a", "x", "z"],
  "y": [10, null, 3]
}
```

- an _array_ of (unnamed) rows

```json
[
  {"x": "a", "y": 10},
  {"x": "x", "y": null},
  {"x": "z", "y": 3}
]
```

## JSON and data frames {-}

An _object_ of (named) columns

```{r}
json_col <- parse_json('
  {
    "x": ["a", "x", "z"],
    "y": [10, null, 3]
  }
')
str(json_col)
```

## JSON and data frames {-}

The elements are not rows, so putting them in a list column before doing `unnest_wider()` must happen **in a single-row tibble**, by using `list()`:

```{r}
df_col <- tibble(x = list(json_col))
df_col
```

## JSON and data frames {-}

Step 1: create the columns

```{r}
df_col |> 
  unnest_wider(x)
```

## JSON and data frames {-}

Step 2: unnest the rows

```{r}
df_col |> 
  unnest_wider(x) |> 
  unnest_longer(c(x, y))
```

## JSON and data frames {-}

An _array_ of (unnamed) rows

```{r}
json_row <- parse_json('
  [
    {"x": "a", "y": 10},
    {"x": "x", "y": null},
    {"x": "z", "y": 3}
  ]
')
str(json_row)
```

## JSON and data frames {-}

The elements are rows, so putting them in a list column before doing `unnest_wider()` can happen **directly**:

```{r}
df_row <- tibble(x = json_row)
df_row
```

## JSON and data frames {-}

We only need to create the columns, since the rows were already there:

```{r}
df_row |> 
  unnest_wider(x)
```




## Meeting Videos

### Cohort 7

`r knitr::include_url("https://www.youtube.com/embed/5t0xD2XsKYM")`

<details>
  <summary> Meeting chat log </summary>
```
00:04:06	Oluwafemi Oyedele:	Hi Tim, Good Evening!!!
00:04:46	Tim Newby:	Good evening - I’m struggling with a new webcam so not sure if sound/video will be working for me tonight!
00:04:49	Oluwafemi Oyedele:	Let us wait a few minute for others to join!!!
00:05:08	Oluwafemi Oyedele:	No problem!!!
00:05:22	Oluwafemi Oyedele:	You can use the chat!!!
00:11:49	Oluwafemi Oyedele:	start
00:21:08	Oluwafemi Oyedele:	https://tidyr.tidyverse.org/reference/unnest_wider.html
00:21:21	Oluwafemi Oyedele:	https://tidyr.tidyverse.org/reference/unnest_longer.html
00:52:48	Oluwafemi Oyedele:	stop
```
</details>

### Cohort 8

`r knitr::include_url("https://www.youtube.com/embed/jY5Mb82v77c")`

### Cohort 9

`r knitr::include_url("https://www.youtube.com/embed/F53PlR1rEqs")`