Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(model): Add optional memoization to datasets during model training. #209

Merged
merged 9 commits into from
Apr 12, 2024

Conversation

ErikBavenstrand
Copy link
Collaborator

@ErikBavenstrand ErikBavenstrand commented Apr 12, 2024

By specifying the optional parameter memoized_dataset_cache_size > 0, the corresponding number of datasets will be kept in memory to avoid repeated conversion from vaex to pandas in settings where we perform repeated fitting using the same datasets e.g. hyperparamter tuning. Use with caution and always call clear_load_dataset_cache once completed to clear the cache.

Erik Båvenstrand added 9 commits April 12, 2024 10:23
By specifying the optional parameter `memoized_dataset_cache_size > 0`, the corresponding number of datasets will be kept in memory to avoid repeated conversion from `vaex` to `pandas` in settings where we perform repeated fitting using the same datasets e.g. hyperparamter tuning. Use with caution and always call `clear_load_dataset_cache` once completed to clear the cache.
@ErikBavenstrand ErikBavenstrand merged commit 2ca4465 into main Apr 12, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant