Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decreasing memory footprint of Tech Mapping #504

Draft
wants to merge 9 commits into
base: mg/agg_zones
Choose a base branch
from

Conversation

mjgleason
Copy link
Collaborator

@mjgleason mjgleason commented Jan 23, 2025

❗ DO NOT MERGE ❗

This PR decreases the memory footprint of Tech Mapping to enable creation of tech maps for exclusions at 30m resolution. It does so primarily by writing the results in slices to the h5 dataset, rather than building it up as an array in memory and then writing the full array to disk. There are a few other memory-saving changes as well.

The PR also introduces a new CLI command tech-mapping which allows running the creation of tech maps in isolation from running supply curve aggregation.

This PR is technically breaking since the TechMapping.run() method used to return the tech map array. This functionality did not appear to be used anywhere (in the codebase or the tests), so I removed it since it more or less defeats the purpose of trying to optimize memory.

Note that this is built on #503 so I have (temporarily) pointed it to the mg/agg_zones branch for comparison.

TODOS:

  • expose batch_size as parameter up the call chain

@mjgleason mjgleason requested a review from ppinchuk January 23, 2025 00:26
@codecov-commenter
Copy link

codecov-commenter commented Jan 23, 2025

Codecov Report

Attention: Patch coverage is 91.20000% with 11 lines in your changes missing coverage. Please review.

Project coverage is 87.71%. Comparing base (381a7b9) to head (ed50e6c).

Files with missing lines Patch % Lines
reV/supply_curve/cli_tech_mapping.py 74.07% 7 Missing ⚠️
tests/test_supply_curve_tech_mapping.py 89.74% 4 Missing ⚠️
Additional details and impacted files
@@               Coverage Diff                @@
##           mg/agg_zones     #504      +/-   ##
================================================
- Coverage         87.72%   87.71%   -0.02%     
================================================
  Files               122      123       +1     
  Lines             18379    18445      +66     
================================================
+ Hits              16123    16179      +56     
- Misses             2256     2266      +10     
Flag Coverage Δ
unittests 87.71% <91.20%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants