Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting a key error on Princeton collections #637

Closed
jacobthill opened this issue Jan 23, 2025 · 0 comments · Fixed by #642
Closed

Getting a key error on Princeton collections #637

jacobthill opened this issue Jan 23, 2025 · 0 comments · Fixed by #642
Assignees
Labels
bug Something isn't working

Comments

@jacobthill
Copy link
Contributor

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 465, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 432, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 401, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 235, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 252, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/dlme_airflow/harvester/source_harvester.py", line 24, in data_source_harvester
    df_and_csv = dataframe_to_file(collection, last_harvest_start_date)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/dlme_airflow/utils/dataframe.py", line 40, in dataframe_to_file
    source_df = collection.catalog.read()
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/dlme_airflow/drivers/iiif_json.py", line 161, in read
    df = pd.concat([self.read_partition(i) for i in range(self.npartitions)])
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/intake/source/base.py", line 187, in read_partition
    return self._get_partition(i)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/dlme_airflow/drivers/iiif_json.py", line 126, in _get_partition
    result = self._open_manifest(self._manifest_urls[i])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/dlme_airflow/drivers/iiif_json.py", line 66, in _open_manifest
    self._extract_manifest_metadata(manifest_result.get("metadata", []))
  File "/opt/airflow/dlme_airflow/drivers/iiif_json.py", line 104, in _extract_manifest_metadata
    for data in self._metadata_from_row(row):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/dlme_airflow/drivers/iiif_json.py", line 119, in _metadata_from_row
    return _flatten_list(_tag_label(_format_label(label), value))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/dlme_airflow/drivers/iiif_json.py", line 192, in _tag_label
    return [_tag_label(label, val) for val in value]
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/dlme_airflow/drivers/iiif_json.py", line 194, in _tag_label
    return [{"label": f"{label}_{value['@language']}", "value": _listify_if_string_or_dict(value['@value'])}]
                                 ~~~~~^^^^^^^^^^^^^
KeyError: '@language'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants