Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compatibility with arrow to deal with larger-than-memory data #48

Open
rafapereirabr opened this issue Dec 14, 2023 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@rafapereirabr
Copy link
Member

rafapereirabr commented Dec 14, 2023

Problem:
Oftentimes, travel cost matrices can get really big in large study areas. This creates a challenge of dealing with larger-than-memory data. The current version of {accessibility} cannot cope with this problem. Although the package is super efficient because it is built on top od {data.table}, the package still requires users to load the travel_matrix and land_use_data input to RAM memory, which can be a constraint for these larger-than-memory cases.

Proposed solution:
One solution to this would be to make the {accessibility} compatible with {arrow} (link). There are different ways to do this. The alternative I would suggest would use the following workfkow:

  • If the user passes travel_matrix and land_use_data inputs as data.tables or data.frames, then the package follows the current natural flow.
  • If the user passes a string path pointing to the location of .parquet or .csv files that contain the travel_matrix and land_use_data inputs, then the package follows a different flow to calculate the indicador using a {dplyr} + {arrow} syntax

Implications:

  • For users, this only means more flexibility to work with larger-than-memory data and greater computation speed.
  • For developers, this means we would need to make a few changes to check inputs, but most of the work would be translating the accessibility functions into a {dplyr} syntax, which is generally super simple.
@rafapereirabr rafapereirabr added the enhancement New feature or request label Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant