Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Create a DataBase class that examines files within a source #4

Open
Zeitsperre opened this issue Sep 19, 2019 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@Zeitsperre
Copy link
Collaborator

A DataBase class is an object with the following characteristics:

  • Initialized with a local address. Address is globbed to find data files (should data file types, ie file suffixes, be initialized as well?).
  • Creates a dictionary of data files; If grouping is set via either a list of indicators or a set of kwargs (e.g. dict(date=True, experiment=True, etc.)), dictionary keys are sorted by groups.
  • Contain methods for the following operations: archive or tar, transfer or copy, report_stats, and others
  • transfer or copy method should depend on local or connected server statistics. If insufficient space, raise a DiskSpaceError.
@Zeitsperre Zeitsperre added the enhancement New feature or request label Sep 19, 2019
@Zeitsperre Zeitsperre self-assigned this Sep 19, 2019
@Zeitsperre
Copy link
Collaborator Author

This has been partially achieved:

  • a Decoder class now exists that will decode files as needed according to discovered metadata.
  • a restructure_datasets function now is able to take data facets and create a filetree structure, with option of either copying or moving files.

There are currently no implementations of these in remote or archive operations. Grouping data for archival purposes would be much easier using the current system. This might be easier if we port the grouping code to class methods of an extended decoder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant