Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add schema hinting to new sheets functionality (C4-1088) #280

Merged
merged 68 commits into from
Oct 31, 2023
Merged

Add schema hinting to new sheets functionality (C4-1088) #280

merged 68 commits into from
Oct 31, 2023

Commits on Aug 14, 2023

  1. First cut at tools for parsing workbooks.

    @netsettler
    netsettler committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    4c84f0b View commit details
    Browse the repository at this point in the history

  2. Refactor to separate some functionality into a separate sevice class.

    @netsettler
    netsettler committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    7b73a67 View commit details
    Browse the repository at this point in the history

  3. Add a csv file for testing.

    @netsettler
    netsettler committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    3d4573f View commit details
    Browse the repository at this point in the history

  4. Add some negative testing.

    @netsettler
    netsettler committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    f4e5cfa View commit details
    Browse the repository at this point in the history

  5. Update lock file.

    @netsettler
    netsettler committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    e9d2465 View commit details
    Browse the repository at this point in the history

  6. Document new sheets_utils module.

    @netsettler
    netsettler committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    6e9060f View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2023

  1. Issue a beta for this functionality.

    @netsettler
    netsettler committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    df12c91 View commit details
    Browse the repository at this point in the history

  2. Fix documentation for sheet_utils.

    @netsettler
    netsettler committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    6a39c8a View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2023

  1. Add some declarations. Small refactors to improve modularity.

    @netsettler
    netsettler committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    eedb5c6 View commit details
    Browse the repository at this point in the history

  2. Rearrange some methods for presentational reasons.

    @netsettler
    netsettler committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    a6b68fe View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2023

  1. First cut at useful functionality.

    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    3ff63a9 View commit details
    Browse the repository at this point in the history

  2. Some name changes to make things more abstract. workbook becomes read… 

    …er_agent, for example
    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    39bd2e0 View commit details
    Browse the repository at this point in the history

  3. Rename sheetname to tabname throughout, to be more clear that this is… 

    … not the workbook level artifact. Better handling of init args.
    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    77b72f6 View commit details
    Browse the repository at this point in the history

  4. Add some doc strings. Rename load_table_set to just load. Arrange for… 

    … ItemManager.load to take a tab_name argument so that CSV files can perhaps infer a type name.
    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    ba8c55c View commit details
    Browse the repository at this point in the history

  5. Add load_items function. Fix some test names. Update changelog.

    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    50488cb View commit details
    Browse the repository at this point in the history

  6. Experimental bug fix from Will to hopefully make get_schema_names work.

    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    807e525 View commit details
    Browse the repository at this point in the history

  7. update changelog

    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    2a8e81a View commit details
    Browse the repository at this point in the history

  8. Update dcicutils/sheet_utils.py 

    Co-authored-by: drio18 <[email protected]>
    @netsettler @drio18
    netsettler and drio18 authored Aug 17, 2023
    Configuration menu
    Copy the full SHA
    718054a View commit details
    Browse the repository at this point in the history

  9. Merge branch 'master' into kmp_sheet_utils

    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    682c95a View commit details
    Browse the repository at this point in the history

  10. Merge branch 'kmp_sheet_utils' into kmp_sheet_utils_refactor_for_csv

    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    582f002 View commit details
    Browse the repository at this point in the history

  11. Add some comments in response to Doug's code review.

    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    56d1459 View commit details
    Browse the repository at this point in the history

  12. Support TSV files.

    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    2facf9e View commit details
    Browse the repository at this point in the history

  13. Add changelog info about tsv files.

    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    bcc4e63 View commit details
    Browse the repository at this point in the history

  14. Add a missing data file.

    @netsettler
    netsettler committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    9de282e View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2023

  1. First stable cut at schema hinting. Doesn't find schemas automaticall… 

    …y yet, though.
    @netsettler
    netsettler committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    8d6495f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into kmp_sheet_utils

    @netsettler
    netsettler committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    3a103ee View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2023

  1. Mark chardet as an acceptable license for use.

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    56f702a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'kmp_sheet_utils' into kmp_sheet_utils_refactor_for_csv

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    08d428e View commit details
    Browse the repository at this point in the history

  3. Merge branch 'kmp_sheet_utils_refactor_for_csv' into kmp_sheet_utils_… 

    …schema_hinting
    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    42ad579 View commit details
    Browse the repository at this point in the history

  4. Backport some small fixes and cosmetics from the schemas branch.

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    60ada3f View commit details
    Browse the repository at this point in the history

  5. Cosmetic fix.

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    690a833 View commit details
    Browse the repository at this point in the history

  6. Add some missing newlines in data files.

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    946b998 View commit details
    Browse the repository at this point in the history

  7. Support for coping with .tsv files where trailing whitespace is 'help… 

    …fully' removed by an editor that doesn't understand such whitespace might be significant in TSVs.
    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    36e7de0 View commit details
    Browse the repository at this point in the history

  8. PEP8

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    a51fb27 View commit details
    Browse the repository at this point in the history

  9. Document our choice of why is_uuid is defined here as it is.

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    6f097a6 View commit details
    Browse the repository at this point in the history

  10. PEP8

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    477c7a2 View commit details
    Browse the repository at this point in the history

  11. Fix error handling to be clearer.

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    09b4c43 View commit details
    Browse the repository at this point in the history

  12. Fix CHANGELOG to reflect recent renamings.

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    f3bd815 View commit details
    Browse the repository at this point in the history

  13. Fix a type hint and some PEP8.

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    7627f6f View commit details
    Browse the repository at this point in the history

  14. Implement a cut at escaping for tsv files.

    @netsettler
    netsettler committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    98cd37c View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2023

  1. Add a test case for all of the pieces of parsing and schema hinting p… 

    …ut together.
    @netsettler
    netsettler committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    3852e56 View commit details
    Browse the repository at this point in the history

  2. Small cosmetic changes and some additional support for upcoming work.

    @netsettler
    netsettler committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    660df9c View commit details
    Browse the repository at this point in the history

  3. Fix a unit test to conform to new google account name.

    @netsettler
    netsettler committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    34d528b View commit details
    Browse the repository at this point in the history

  4. Fix typo in comment (dcicutils/misc_utils.py) 

    Co-authored-by: drio18 <[email protected]>
    @netsettler @drio18
    netsettler and drio18 authored Aug 25, 2023
    Configuration menu
    Copy the full SHA
    1c34ad0 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2023

  1. Add some doc strings and comments.

    @netsettler
    netsettler committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    41fad79 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2023

  1. Rename tabname to tab_name throughout the sheet_utils interfaces.

    @netsettler
    netsettler committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    6e8ce2c View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2023

  1. Add support for reading inserts dirs, .json, .jsonl (two formats), an… 

    …d .tabs.json
    @netsettler
    netsettler committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    04eb58c View commit details
    Browse the repository at this point in the history

  2. Bump beta version.

    @netsettler
    netsettler committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    ce9f9bc View commit details
    Browse the repository at this point in the history

  3. Add yaml formats.

    @netsettler
    netsettler committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    0ea5b62 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2023

  1. Add class AbstractItemManager. Rename InsertsItemManager to InsertsDi… 

    …rectoryItemManager. Rename _JsonInsertsDataItemManager to InsertsItemManager.
    @netsettler
    netsettler committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    bcc1128 View commit details
    Browse the repository at this point in the history

  2. Rename ._parser() to ._parse_json_data(). Factor type checks out of .… 

    …_load_json_data() into ._check_json_data().
    @netsettler
    netsettler committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    7de093a View commit details
    Browse the repository at this point in the history

  3. Rename _parse_json_data, _load_json_data, and _check_json_data, respe… 

    …ctively, to _parse_inserts_data, _load_inserts_data, and _check_inserts_data.
    @netsettler
    netsettler committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    b01e34b View commit details
    Browse the repository at this point in the history

  4. WIP. Testing good.

    @netsettler
    netsettler committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    0ae48ee View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2023

  1. WIP. Tests passing.

    @netsettler
    netsettler committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    1e2c5a9 View commit details
    Browse the repository at this point in the history

  2. Rearrange the way escaping= works so both csv an tsv files can using … 

    …that argument.
    @netsettler
    netsettler committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    b8a4c39 View commit details
    Browse the repository at this point in the history

  3. Separate registration of regular table set managers from registration… 

    … of item managers.
    @netsettler
    netsettler committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    a2fe079 View commit details
    Browse the repository at this point in the history

  4. Stub in checking of required headers.

    @netsettler
    netsettler committed Sep 2, 2023
    Configuration menu
    Copy the full SHA
    91ddce0 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2023

  1. Bump beta version.

    @netsettler
    netsettler committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    142a20b View commit details
    Browse the repository at this point in the history

  2. PEP8

    @netsettler
    netsettler committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    e09af07 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. Merge branch 'kmp_schemas_from_vapp' into kmp_sheet_utils_with_vapp

    @netsettler
    netsettler committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    70762c6 View commit details
    Browse the repository at this point in the history

  2. Fix a bug in newly proposed ff_utils.get_schemas with vapp.

    @netsettler
    netsettler committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    7d2ecaa View commit details
    Browse the repository at this point in the history

  3. Extend VirtualApp to amke it easier to test by adding an AbstractVirt… 

    …ualApp.
    @netsettler
    netsettler committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    5e46273 View commit details
    Browse the repository at this point in the history

  4. Implement portal_vapp= in sheet_utils.

    @netsettler
    netsettler committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    53de60a View commit details
    Browse the repository at this point in the history

  5. Simplifications per Will's code review.

    @netsettler
    netsettler committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    630720f View commit details
    Browse the repository at this point in the history

  6. Merge utils 7.10.0 from master.

    @netsettler
    netsettler committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    5a07b69 View commit details
    Browse the repository at this point in the history

  7. Merge pull request #282 from 4dn-dcic/kmp_sheet_utils_with_vapp 

    Add portal_vapp= functionality to sheet_utils
    @netsettler
    netsettler authored Sep 7, 2023
    Configuration menu
    Copy the full SHA
    295adfe View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2023

  1. Merge branch 'master' into kmp_sheet_utils_schema_hinting

    @netsettler
    netsettler committed Sep 9, 2023
    Configuration menu
    Copy the full SHA
    486adce View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2023

  1. Add support for zipped files.

    @netsettler
    netsettler committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    54c51aa View commit details
    Browse the repository at this point in the history