Unable to load CMOR table for "tasmax" with mip "days"

[samlost9-9]

Hi all,

This is my first time posting here, I apologize in advance for anything that I may have done wrong.

For a little background of what I am trying to do: I simply want to use the extract_shape and regrid preprocessors to go through multiple models with tasmax. When I ran ESMValTool, it was able to find all the .nc files but failed to run successfully due to the CMOR table.
The exact message for the error is: Unable to load CMOR table (project) 'sd_pcic' for variable 'tasmax' with mip 'day'

I am not sure what is causing this error, I checked the CMIP6_days.json file and made sure tasmax is in it. I also tried to treat tasmax as a custom variable and create my own version of cmor table but the same error occured. Therefore, I am posting to see if anyone has a solution to my problem, or points me to some documentation and references.

I really appreciate any help you can provide.

Here is the Dropbox link to one of the .nc files, if anyone would like to take a closer look. https://www.dropbox.com/scl/fi/6dyrpipkc217zkcywlrl4/tasmax_day_MBCn-PCIC-Blend_CanESM5_historical-ssp370_r1i1p2f1_gn_20410101-20451231.nc?rlkey=fol3e17w4j5riy0657ep0jcd1&st=vlf8gaw1&dl=0

Here is my recipe:

datasets:
  - &dataset_para {dataset: CanESM5, project: sd_pcic, ensemble: r1i1p2f1, grid: gn} 

preprocessors:
  extract_on:
    extract_shape: 
      shapefile: ~/shapefile/prov_terr_bounary_census/ON_boundary.shp
      method: contains

diagnostics:
  testing:
    description: | 
      extract ontario region
    variables:
      tasmax:
        shortname: tasmax
        mip: day
        exp: ssp370
        grid: gn
        start_year: 2000
        end_year: 2010
        preprocessor: extract_on
    scripts:
      simple_qc:
        script: ~/esmvaltool_test/diag_scripts/general_qc.py

[valeriupredoi]

Hi @samlost9-9 and thanks for dropping by and using ESMValTool! It would help if you could post here (no Dropbox links etc) the recipe you are using, and if you are not using a recipe per se, post the script that you are running to see the way data is specified. The one clue I can already fish out is that you may have not put the custom table in the correct path where esmvalcore was built at (especially since I believe you are using esmvalcore from a stock conda environment built via ESMValTool). Also, I am not familiar with the sd_pcic project - that sounds to me like an experiment (pcic surely is an experiment) so that may be another issue. But if you post the run files here then it'll be easier to understand the chain of events. Cheers 🍺

[samlost9-9]

Hello @valeriupredoi, thank you for the response. You are absolutely correct, the sd_pcic project is an experimental project, and the variable is for testing.

For some reason, I can't recreate the error anymore, so I am not sure whether the issue has been fixed or not. However, I will include more details of what I did for clarification.
 
I created the cmor table below and placed it in the default cmor table directory esmvalcore/cmor/tables/custom, (hopefully, this is the right path)

SOURCE: CMIP5 (adapted from tas)
!============
variable_entry:    tasmax
!============
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     
units:             degC
cell_methods:      time: maximum
cell_measures:     area: areacella
long_name:         Daily Maximum Near-Surface Air Temperature
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time
out_name:          tasmax
type:              short
!----------------------------------
!

For reference is what my config-developer.yml looks like:

sd_pcic: 
  cmor_strict: false
  input_dir:
    default: '{dataset}_{short_name}_{ensemble}_{exp}_MBCn_5_year_files/'
  input_file: '{shortname}_{mip}_MBCn+PCIC-Blend_{dataset}_historical+{exp}_{ensemble}_{grid}_*.nc'
  output_file: '{short_name}_{mip}_{dataset}_{exp}_{ensemble}'

[valeriupredoi]

@samlost9-9 cheers, this looks good! And the location for the custom CMOR table (esmvalcore/cmor/tables/custom) is correct too, provided you are using the editable, development installation for the esmvalcore package (you can alywas check that with a simple conda list esmvalcore - if it says it's from pypi you can then see the location with pip show esmvalcore; if it says conda-forge then you are using the tagged version from conda that doesn't have your changes).

The problem here is that tasmax is a standard CMIP variable, so the code will not look for custom variables right away - it will look for a CMOR path of the project, which it can not find: you can add that path by creating it like in the case for CORDEX, in esmvaclore/cmor/tables/sd_pcic then you'd have to append the type of cmor and path to your config-developer.yml entry:

  cmor_type: 'CMIP5'
  cmor_path: 'sd_pcic'

Another option would be to use a custom variable table but have the variable renamed to eg tasmax2 like @malininae or @schlunma have done a few times AFAIK, but I think that means renaming some of your data files, or, at least, symlinking them

[samlost9-9]

@valeriupredoi Thank you for the confirmation, glad to know that I am putting the custom cmor table in the right directory. However, a new error occurred (shown below) after I appended the cmor_type and cmor_path entries to the config_developer.yml. I don't think is the recipe caused this error because the same error would occur even if I ran a different recipe.

Exception raised when loading ~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/cmor/tables/sd_pcic/Tables/CMOR_idETCCDI.dat
ERROR:esmvalcore._main:Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/_main.py", line 521, in run
    fire.Fire(ESMValTool())
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/_main.py", line 310, in __init__
    self.__setattr__(entry_point.name, entry_point.load()())
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2517, in load
    return self.resolve()
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2523, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvaltool/cmorizers/data/cmorizer.py", line 17, in <module>
    from esmvalcore._task import write_ncl_settings
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/_task.py", line 23, in <module>
    from ._citation import _write_citation_files
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/_citation.py", line 10, in <module>
    from .config._diagnostics import DIAGNOSTICS
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/config/__init__.py", line 11, in <module>
    from ._config_object import CFG, Config, Session
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/config/_config_object.py", line 297, in <module>
    CFG = Config._load_user_config(USER_CONFIG, raise_exception=False)
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/config/_config_object.py", line 72, in _load_user_config
    new.update(mapping)
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/_collections_abc.py", line 999, in update
    self[key] = other[key]
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/config/_validated_config.py", line 65, in __setitem__
    cval = self._validate[key](val)
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/config/_config_validators.py", line 234, in validate_config_developer
    load_config_developer(path)
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/config/_config.py", line 112, in load_config_developer
    read_cmor_tables(cfg_file)
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/cmor/table.py", line 122, in read_cmor_tables
    cmor_tables = _read_cmor_tables(cfg_file, mtime)
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/cmor/table.py", line 166, in _read_cmor_tables
    cmor_tables[table] = _read_table(cfg_developer, table, install_dir,
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/cmor/table.py", line 189, in _read_table
    return CMIP5Info(table_path,
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/cmor/table.py", line 756, in __init__
    self._load_table(table_file)
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/cmor/table.py", line 782, in _load_table
    self._read_table_file(table_file, table)
  File "~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore/cmor/table.py", line 807, in _read_table_file
    table[value] = self._read_variable(value, table.frequency)
AttributeError: 'NoneType' object has no attribute 'frequency'

[malininae]

@samlost9-9 can you please share the recipe of what you are trying to do?

[samlost9-9]

Sorry about that, I thought I already included it. Now it should be up, thank you!

[malininae]

Thanks, from what I see, I would do following adjustment to your config_developer.yml

sd_pcic: 
  cmor_strict: false
  input_dir:
    default: '{dataset}_{short_name}_{ensemble}_{exp}_MBCn_5_year_files/'
  input_file:
    default: '{short_name}_{mip}_MBCn+PCIC-Blend_{dataset}_{exp}_{ensemble}_{grid}_*.nc'
  output_file: '{short_name}_{mip}_{dataset}_{exp}_{ensemble}'
  cmor_type: 'CMIP6'
  cmor_default_table_prefix: 'CMIP6_'

and put exp: historical-ssp370 in the recipe. I think it should work then. Also, are you at PCIC?

[valeriupredoi]

@malininae brilliant that you are chipping in (and hopefully, fixing the problem, by the looks of it!), I am rather rusty with actually running the Tool 😁 @samlost9-9 what I was suggesting was to actually create the project's dir in esmvalcore/cmor and plopping the cmor file there - also, be sure that your esmvalcore is not a development one, but rather, stock from the conda environment in ~/tools/envs/esmvaltool2-8/lib/python3.10/site-packages/esmvalcore - just FYI, if you are making changes in a different esmvalcore - those won't get picked up at runtime 👍

[samlost9-9]

@malininae and @valeriupredoi, sorry for the late update. After a couple of trial and error. I changed the config_developer.yml like @malininae suggested, and the program can load the dataset successfully this time.

Correct me if I am wrong, I believe the cmor_type and cmor_default_table_prefix lines forced the program to use the cmor table from, becuase when I changed the cmor_type to CMIP5, the program can load the dataset and run all the preprocessors too.

As for the AttributeError: 'NoneType' object has no attribute 'frequency' that I mentioned earlier, it occurs only when I use a custom cmor_path. Is this the direct cause of the error or not, I cannot say for sure.

Nevertheless, my major issue has been resolved, so thank you very much to @valeriupredoi and @malininae for helping me!!!!😁

[valeriupredoi]

Correct me if I am wrong, I believe the cmor_type and cmor_default_table_prefix lines forced the program to use the cmor table from, becuase when I changed the cmor_type to CMIP5, the program can load the dataset and run all the preprocessors too.

That is indeed the case! Great to hear it works for you, and apols it a wee while for a reply, we were traveling to and from an ESMValTool workshop. It'd be great if you could mark Liza's reply as the Answer please, so we can keep Discussions items updated. Cheers muchly 🍺

[samlost9-9]

@valeriupredoi @malininae Thank you very much for the support!! =)