How to run LASD historical scrape

[erika-tyagi]

Opening this up as a draft PR since it should NOT be merged to the main branch!! – I just needed a place to add comments/instructions. Normal command line syntax to run (below):

To run redo-scrape:

Rscript production/redo_scrape/main_redo.R --scraper lasd --start 2020-11-01 --end 2020-11-04

To run WBM scrape:

./production/wayback_scrape/main_wayback.R -sc lasd -st 2020-11-01 -en 2021-11-02

---

Steps to take after scraping but before handing off to volunteers to minimize manual cleaning/entry:

• Combine extracted files (something like this should work):

df <- list.files("results/extracted_data/", full.names = T) %>%
    lapply(function(x){
        df_ <- fread(x)
        if(nrow(df_) == 0){
            df_ <- data.table()
        }
        if("Date" %in% names(df_)){
            df_[,Date := lubridate::as_date(Date)]
        }
        df_
    }) %>%
    rbindlist(fill=TRUE, use.names = TRUE) 

• If the lag and lead of an NA value are the same, replace the NA with that value
• Plot each metric over time to visually inspect anomalies

[erika-tyagi]

Commenting out the validate here to avoid dropping columns we want to keep!

[erika-tyagi]

Removing this to pull ALL days (regardless of day-of-the-week).

[erika-tyagi]

Same thing as above – don't want to validate to avoid dropping columns.

[erika-tyagi]

Note that we use ExtractTable here (which we don't do in the main scraper). I couldn't find a better way to pull the data from the tables with any reliability unfortunately. Obviously ExtractTable is cheap but not free (and reimbursement is still unclear).