Import BBS data by route

[diazrenata]

Functions to

• subset the BBS data to sufficiently long timeseries
• extract each route and clean species labels, etc
• add the community data for every route to datasets plans

The initial processing step (prepare_bbs_ts_data) does the cleaning and subsetting necessary to get the list of routes and regions for the plans, and saves the processed abundance table and the tables for making the plans in the directory with the data path. The remaining functions load those data tables as needed (trying to minimize the number of times we interact with the main dataframe - it's slow to load).

get_bbs_route_region_data pulls a specified route + region and does the final cleaning on it. I've broken the cleaning into two steps because this last step takes forever and crashed R for me a few times when I tried to do it on the whole dataframe at once.

build_bbs_datasets_plan makes a drake plan to get data for all the routes. (2500 of them)

build_datasets_plan now includes the option to include BBS, in which case it will use build_bbs_datasets_plan.

handles #125

[ha0ye]

Some general notes:

• build_datasets_plan(), build_bbs_datasets_plan(), prepare_bbs_ts_data(), and get_bbs_route_region_data() all take a path argument, but the path argument isn't consistently passed between them, and so the default option is relied on.
• some of the file paths are created using paste0; I think the recommend approach is to use file.path() so that we don't run into issues with forward-slash, back-slash confusion (across different operating systems)
• one line 130, the call to get_bbs_route_region_data() doesn't use the path argument, and it probably needs !!, as on lines 93 and 94.
• should the default end_yr in prepare_bbs_ts_data() be 2017? Maybe we can add the option to have the argument be NA, and have it be ignored?

[ha0ye]

More generally, it seems like get_bbs_route_region_data() is pretty slow, and I think this is because it's trying to load the entire bbs dataset. Since it seems like the new workflow is for prepare_bbs_ts_data() to be called first, maybe it should split the data into separate files for each route x region, which then speeds up get_bbs_route_region_data()?

[codecov-io]

Codecov Report

Merging #127 into master will decrease coverage by 19.09%.
The diff coverage is 0.83%.

@@            Coverage Diff             @@
##           master     #127      +/-   ##
==========================================
- Coverage   89.18%   70.08%   -19.1%     
==========================================
  Files          13       14       +1     
  Lines         490      575      +85     
==========================================
- Hits          437      403      -34     
- Misses         53      172     +119

Impacted Files	Coverage Δ
R/get_retriever_data.R	88.88% <0%> (-11.12%)	⬇️
R/bbs_cleaning_functions.R	0% <0%> (ø)
R/build_plans.R	65.95% <5.88%> (-34.05%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab5cc74...e02c118. Read the comment docs.

[codecov-io]

Codecov Report

Merging #127 into master will decrease coverage by 19.46%.
The diff coverage is 0.81%.

@@             Coverage Diff             @@
##           master     #127       +/-   ##
===========================================
- Coverage   89.18%   69.72%   -19.47%     
===========================================
  Files          13       14        +1     
  Lines         490      578       +88     
===========================================
- Hits          437      403       -34     
- Misses         53      175      +122

Impacted Files	Coverage Δ
R/get_retriever_data.R	88.88% <0%> (-11.12%)	⬇️
R/bbs_cleaning_functions.R	0% <0%> (ø)
R/build_plans.R	63.26% <5.26%> (-36.74%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab5cc74...3c1be4a. Read the comment docs.

[diazrenata]

@ha0ye, could you maybe take a look at this version of things? Hopefully it's a little more streamlined!

[ha0ye]

@diazrenata I just noticed that you have files in vignettes/.drake as part of your commit - did you mean to include those?

[diazrenata]

👀 Nope -apologies! I can get rid of them and fix the gitignore in a few...

…

On Tue, May 28, 2019 at 10:25 Hao Ye ***@***.***> wrote: @diazrenata <https://github.com/diazrenata> I just noticed that you have files in vignettes/.drake as part of your commit - did you mean to include those? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#127?email_source=notifications&email_token=AEH6DN5ZJIPWH6EGCBH7AALPXU6HPA5CNFSM4HMRIZ4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWMJR4A#issuecomment-496539888>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEH6DNZC7YVLSAT7B6KENK3PXU6HPANCNFSM4HMRIZ4A> .