Mismatch between datetime objects #730

nishsilva · 2025-01-08T21:04:30Z

Bug Severity

1 = Minor problem that does not affect total framework functionality (e.g., computation error in a POD, problem with logging output, or an issue on a single system
2 = Major problem that affects overall functionality, but that does not occur for all users (e.g., problems installing the framework with a specific Conda version, a framework option that causes one or more PODs to fail, or missing/incompatible Python modules).
3 = Catastrophic problem that occurs frequently for multiple users and/or on multiple systems (e.g.,framework consistently fails to install on multiple systems, or one or more PODs continuously fails after running successfully)

Describe the bug
I am trying to read in a model data set from ECMWF-LR output in the MDTF. For this I use a data catalog. When I run the MDTF I get an error (attached) saying that there is a mismatch between the cftime datetime objects in the dataset because one using the Gregorian calendar and the other using the noleap calendar.

Steps To Reproduce

Created a data catalog for a HighResMIP Model data files (variable zos) - CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.hist-1950.r1i1p1f1.Omon.zos.gn
Created a trial POD names sl_reginoal https://github.com/nishsilva/MDTF-diagnostics/tree/nish-pod/diagnostics/sl_regional
The runtime configuration file is https://github.com/nishsilva/MDTF-diagnostics/blob/nish-pod/diagnostics/sl_regional/runtime_config_NE_ECMWF_HR.yml

Environment
Describe the system environment:

OS: [LINUX]
Conda version: [conda version : 24.9.2]
branch name and link: [e.g., nish-pod, https://github.com/nishsilva/MDTF-diagnostics/tree/nish-pod]

Log information and/or terminal output

netige@crhtc59:/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics> ./mdtf -f ./diagnostics/sl_regional/runtime_config_NE_ECMWF_HR.yml
Preprocessing data for sl_regional
Querying /glade/work/netige/mdtf_Nov24/data_catalogs/s_ECMWF-IFS-HR_e_hist-1950_vl_r1i1p1f1_f_mon_v_zos.json for variable zos for case s_ECMWF-IFS-HR_e_hist-1950_vl_r1i1p1f1_f_mon_v_zos.
WARNING: /glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/intake_esm/_search.py:50: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.
  mask = df[column].str.contains(value, regex=True, case=True, flags=0)

Converting zos catalog subset to dataset dictionary
CRITICAL: **********************************************************************
Uncaught exception:
Traceback (most recent call last):
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: cftime.datetime(2011, 1, 16, 12, 0, 0, 0, calendar='noleap', has_year_zero=True)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6798, in get_slice_bound
    slc = self.get_loc(label)
          ^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/xarray/coding/cftimeindex.py", line 471, in get_loc
    return super().get_loc(key)
           ^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: cftime.datetime(2011, 1, 16, 12, 0, 0, 0, calendar='noleap', has_year_zero=True)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/cftime/_cftime.pyx", line 1425, in cftime._cftime.datetime.__richcmp__
  File "src/cftime/_cftime.pyx", line 1523, in cftime._cftime.datetime.change_calendar
ValueError: change_calendar only works for real-world calendars

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/mdtf_framework.py", line 243, in <module>
    exit_code = main(prog_name='MDTF-diagnostics')
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/mdtf_framework.py", line 199, in main
    cat_subset = data_pp.process(cases, ctx.config, model_paths.MODEL_WORK_DIR)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/preprocessor.py", line 1605, in process
    cat_subset = self.query_catalog(case_list, config.DATA_CATALOG)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/preprocessor.py", line 1132, in query_catalog
    cat_subset_dict[k] = self.crop_date_range(date_range,
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/preprocessor.py", line 923, in crop_date_range
    new_xr_ds = xr_ds.sel({time_coord.name: slice(ds_start, ds_end)})
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/xarray/core/dataset.py", line 3098, in sel
    query_results = map_index_queries(
                    ^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/xarray/core/indexing.py", line 193, in map_index_queries
    results.append(index.sel(labels, **options))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/xarray/core/indexes.py", line 741, in sel
    indexer = _query_slice(self.index, label, coord_name, method, tolerance)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/xarray/core/indexes.py", line 497, in _query_slice
    indexer = index.slice_indexer(
              ^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6662, in slice_indexer
    start_slice, end_slice = self.slice_locs(start, end, step=step)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6879, in slice_locs
    start_slice = self.get_slice_bound(start, "left")
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6801, in get_slice_bound
    return self._searchsorted_monotonic(label, side)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6733, in _searchsorted_monotonic
    return self.searchsorted(label, side=side)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/pandas/core/base.py", line 1352, in searchsorted
    return algorithms.searchsorted(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/pandas/core/algorithms.py", line 1329, in searchsorted
    return arr.searchsorted(value, side=side, sorter=sorter)  # type: ignore[arg-type]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/cftime/_cftime.pyx", line 1427, in cftime._cftime.datetime.__richcmp__
TypeError: cannot compare cftime.DatetimeGregorian(2011, 7, 16, 12, 0, 0, 0, has_year_zero=False) and cftime.datetime(2011, 1, 16, 12, 0, 0, 0, calendar='noleap', has_year_zero=True)
netige@crhtc59:/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics>

The text was updated successfully, but these errors were encountered:

jtmims · 2025-01-08T21:32:31Z

@nishsilva thank you for opening this issue! I will update you on what I find.

jtmims · 2025-01-16T15:52:57Z

Hey @nishsilva, there were a few issues found. They have been fixed in the latest PR #732. If you could try this out, let me know if this solves what you are seeing!

nishsilva · 2025-01-16T19:33:58Z

Thank you for the fix @jtmims the datetime issue has been resolved. However, I am encountering another error when I set preprocessing to true. Let me know if I need to open this as a new issue. Below is the error message I am receiving.

Preprocessing data for MDSL
Querying /glade/work/netige/mdtf_Nov24/data_catalogs/s_ECMWF-IFS-HR_e_hist-1950_vl_r1i1p1f1_f_mon_v_zos.json for variable zos for case s_ECMWF-IFS-HR_e_hist-1950_vl_r1i1p1f1_f_mon_v_zos.
WARNING: /glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/intake_esm/_search.py:50: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.
  mask = df[column].str.contains(value, regex=True, case=True, flags=0)

Converting zos catalog subset to dataset dictionary
Calling check_time_bounds for zos
Calling parse_ds for zos
Units for 'time' on var 'zos' found in dataset; setting to 'days since 1950-01-01 00:00:00'.
No calendar for 'time' found in dataset; setting to 'sentinel.NotSet'.
WARNING: /glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/cf_xarray/accessor.py:1772: UserWarning: Variables {'time_bnds'} not found in object but are referred to in the CF attributes.
  warnings.warn(

CRITICAL: **********************************************************************
Uncaught exception:
Traceback (most recent call last):
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/xarray/core/dataset.py", line 1407, in _copy_listed
    variables[name] = self._variables[name]
                      ~~~~~~~~~~~~~~~^^^^^^
KeyError: 'time_bnds'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/cf_xarray/accessor.py", line 863, in _getitem
    ds = ds.reset_coords()[varnames + coords]
         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/xarray/core/dataset.py", line 1553, in __getitem__
    return self._copy_listed(key)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/xarray/core/dataset.py", line 1409, in _copy_listed
    ref_name, var_name, var = _get_virtual_variable(
                              ^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/xarray/core/dataset.py", line 211, in _get_virtual_variable
    raise KeyError(key)
KeyError: 'time_bnds'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/preprocessor.py", line 1546, in parse_ds
    ds = self.parser.parse(var, ds)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/xr_parser.py", line 1394, in parse
    self.reconcile_variable(var, ds)
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/xr_parser.py", line 1222, in reconcile_variable
    self.reconcile_dimension_coords(tv, ds)
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/xr_parser.py", line 1143, in reconcile_dimension_coords
    self.reconcile_coord_bounds(coord, ds, ds_coord_name)
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/xr_parser.py", line 1076, in reconcile_coord_bounds
    bounds = ds.cf.get_bounds(ds_coord_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/cf_xarray/accessor.py", line 2285, in get_bounds
    results = self[[key]].cf.bounds.get(key, [])
              ~~~~^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/cf_xarray/accessor.py", line 2173, in __getitem__
    return _getitem(self, key)
           ^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/cf_xarray/accessor.py", line 881, in _getitem
    raise KeyError(
KeyError: "Dataset.cf does not understand the key 'time'. Use 'repr(Dataset.cf)' (or 'Dataset.cf' in a Jupyter environment) to see a list of key names that can be interpreted."

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/mdtf_framework.py", line 243, in <module>
    exit_code = main(prog_name='MDTF-diagnostics')
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/mdtf_framework.py", line 199, in main
    cat_subset = data_pp.process(cases, ctx.config, model_paths.MODEL_WORK_DIR)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/preprocessor.py", line 1585, in process
    var_xr_dataset = self.parse_ds(v, case_xr_dataset)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/preprocessor.py", line 1548, in parse_ds
    raise util.chain_exc(exc, f"parsing dataset metadata", util.DataPreprocessEvent)
  File "/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics/src/util/exceptions.py", line 58, in chain_exc
    raise new_exc_class(new_msg) from exc
src.util.exceptions.DataPreprocessEvent: Caught exception while parsing dataset metadata: KeyError("Dataset.cf does not understand the key 'time'. Use 'repr(Dataset.cf)' (or 'Dataset.cf' in a Jupyter environment) to see a list of key names that can be interpreted.").
(miniconda3) netige@crhtc38:/glade/work/netige/mdtf_Nov24/mdtf/MDTF-diagnostics> ```

jtmims · 2025-01-17T19:36:17Z

@nishsilva thanks for sending in these issues! I was able to get the preprocessor to work on this data and launch your POD using PR #733. Let me know if this works for you too!

nishsilva · 2025-01-18T00:02:58Z

@jtmims preprocessor is now working fine at my end too. Thank you so much for your support! I am closing the issue.

jtmims added bug Something isn't working framework Issue pertains to the framework code labels Jan 8, 2025

jtmims self-assigned this Jan 8, 2025

jtmims mentioned this issue Jan 13, 2025

add function to normalize all time coord units #732

Merged

11 tasks

jtmims mentioned this issue Jan 17, 2025

del bounds attr from coord in drop_attributes() #733

Merged

11 tasks

nishsilva closed this as completed Jan 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mismatch between datetime objects #730

Mismatch between datetime objects #730

nishsilva commented Jan 8, 2025

jtmims commented Jan 8, 2025

jtmims commented Jan 16, 2025

nishsilva commented Jan 16, 2025

jtmims commented Jan 17, 2025

nishsilva commented Jan 18, 2025

Mismatch between datetime objects #730

Mismatch between datetime objects #730

Comments

nishsilva commented Jan 8, 2025

jtmims commented Jan 8, 2025

jtmims commented Jan 16, 2025

nishsilva commented Jan 16, 2025

jtmims commented Jan 17, 2025

nishsilva commented Jan 18, 2025