Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-45431: Update parquet formatter to use can_accept() #1059

Merged
merged 13 commits into from
Aug 19, 2024
Merged

DM-45431: Update parquet formatter to use can_accept() #1059

merged 13 commits into from
Aug 19, 2024
+435 −714

Commits on Aug 17, 2024

  1. Add can_accept directive to parquet formatter.

    @erykoff
    erykoff committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    dc88e65 View commit details
    Browse the repository at this point in the history

  2. Add can_accept() to StorageClassDelegate and InMemoryDatastore.

    @erykoff
    erykoff committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    35c6a1e View commit details
    Browse the repository at this point in the history

  3. Update ArrowTableDelegate to support all the arrow/parquet options.

    @erykoff
    erykoff committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    a9a61dd View commit details
    Browse the repository at this point in the history

  4. Remove now unused arrow/parquet delegates.

    @erykoff
    erykoff committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    0710b57 View commit details
    Browse the repository at this point in the history

  5. Update storageClasses.yaml using new arrow/parquet delegates.

    @erykoff
    erykoff committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    be1497d View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2024

  1. Factor out _checkAstropyTableEquality into helper function.

    @erykoff
    erykoff committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    c42bc8b View commit details
    Browse the repository at this point in the history

  2. Factor out _checkNumpyTableEquality and _checkNumpyDictEquality.

    @erykoff
    erykoff committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    efb3275 View commit details
    Browse the repository at this point in the history

  3. Add new tests for lossless storage into different storage classes. 

    The primary use case is storing astropy tables with units into
existing storage classes that are declared as DataFrames. With
this update astropy tables can be fully preserved and the
tests focus on this.
    @erykoff
    erykoff committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    4fa22ef View commit details
    Browse the repository at this point in the history

  4. Replace call of validate_put_parameters with isAcceptable for InMemor… 

    …yDatastore.
    @erykoff
    erykoff committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    3e14034 View commit details
    Browse the repository at this point in the history

  5. Update parquet tests after removal of extra delegates.

    @erykoff
    erykoff committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    f0b266d View commit details
    Browse the repository at this point in the history

  6. Remove no longer used _validate_put_parameters from GenericBaseDatast… 

    …ore.
    @erykoff
    erykoff committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    2328789 View commit details
    Browse the repository at this point in the history

  7. Add changelog entry.

    @erykoff
    erykoff committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    d6d8dec View commit details
    Browse the repository at this point in the history

  8. Refactor ParquetFormatter to reuse dataset checking code.

    @erykoff
    erykoff committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    54a7d55 View commit details
    Browse the repository at this point in the history