Investigate scanning CORDEX data with catalog-maker #10

agstephens · 2021-11-15T09:17:17Z

Find a small amount of CORDEX (C3S) data, and see if the catalog-maker works with it.

ellesmith88 · 2021-11-15T13:23:05Z

This seems to be working fine, however it is necessary to add is_default_for_path = True to [project:c3s-cordex] in roocs.ini to get it to work. I think we will need this when processing the c3s-cordex data as well.

Example entry:

{"ds_id": "c3s-cordex.output.EUR-11.MOHC.MOHC-HadGEM2-ES.rcp85.r1i1p1.MOHC-HadREM3-GA7-05.v1.mon.tas.v20200330", 
"path": "output/EUR-11/MOHC/MOHC-HadGEM2-ES/rcp85/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_MOHC-HadGEM2-ES_rcp85_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_200512-201012.nc", 
"size": 27812779, 
"project": "c3s-cordex", 
"product": "output", 
"domain": "EUR-11", 
"institute": "MOHC", 
"driving_model": "MOHC-HadGEM2-ES", 
"experiment": "rcp85", 
"ensemble": "r1i1p1", 
"rcm_name": "MOHC-HadREM3-GA7-05", 
"rcm_version": "v1", 
"time_frequency": "mon", 
"variable": "tas", 
"version": "v20200330", 
"start_time": "2005-12-16T00:00:00", 
"end_time": "2010-12-16T00:00:00", 
"bbox": "-44.59, 21.99, 64.96, 72.59", 
"level": "1.50"}

is output correct for product?

agstephens · 2021-11-15T15:36:49Z

Hi @ellesmith88, that's great news. Yes, output is correct as the value for product.

@cehbrecht: do you have a list of valid datasets for C3S-Cordex? When you do, please provide to @ellesmith88 who can create a new intake catalog from them.

cehbrecht · 2021-11-16T12:47:15Z

@ellesmith88 @agstephens I have no feedback by @glevava yet. But I have opened a pull request for cordex in our manifest repo:
cp4cds/c3s_34g_manifests#21

It contains two files with the complete file list of CORDEX datasets replicated to all 3 sites:

cordex/C3S-CORDEX_ipsl_sizes.txt.gz size for each file
cordex/C3S-CORDEX_ipsl_checksums.txt.gz checksum for each file

We used these files to check if our repos are in sync.

Note: These are not the official files given to the CDS people. But they should have the same datasets ... just in a different format.

glevava · 2021-11-16T12:53:57Z

Sorry to forgot answering this. I sent the CDS manifest to @cehbrecht through Slack.

cehbrecht · 2021-11-16T13:22:52Z

Sorry to forgot answering this. I sent the CDS manifest to @cehbrecht through Slack.

I have added it to the cordex pull request in the manifest repo :)

agstephens · 2021-11-16T13:25:45Z

Thanks @glevava @cehbrecht

agstephens · 2021-11-16T13:28:00Z

@ellesmith88 can you try running the catalog-maker on the datasets listed at:
https://github.com/cp4cds/c3s_34g_manifests/blob/master/cordex/manifest_C3S_34B_Lot1_CORDEX_20210708_fixed.tgz

Thanks

ellesmith88 · 2021-11-22T13:06:07Z

These jobs are running on LOTUS, look to be going well so far

agstephens · 2021-12-01T09:22:28Z

@ellesmith88: please can you give an update on the status of the CORDEX Intake catalog.

ellesmith88 · 2021-12-02T08:57:37Z

@agstephens 194/23196 left to scan, so I'm running it again. 104 with errors so far, some of which I might be able to fix so having a look today

ellesmith88 · 2021-12-06T13:17:23Z

Update on this:

out of 23196 total datasets:

23074 scanned successfully
For 36 there were no files found:

c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.psl.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.sfcWind.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.tasmax.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.tasmin.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.tas.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.ua850.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.va850.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.day.evspsbl.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.day.pr.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.day.psl.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.day.sfcWind.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.day.tasmax.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.day.tasmin.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.day.tas.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.day.ua850.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.day.va850.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.mon.evspsbl.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.mon.pr.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.mon.psl.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.mon.sfcWind.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.mon.tasmax.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.mon.tasmin.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.mon.tas.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.mon.ua850.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.rcp85.r1i1p1.ICTP-RegCM4-3.v1.mon.va850.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.pr.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.evspsbl.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.day.va850.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.day.ua850.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.day.tas.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.day.tasmin.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.day.tasmax.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.day.sfcWind.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.day.psl.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.day.pr.v20210111
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.day.evspsbl.v20210111

For the remaining 86 they failed because of one of these errors:

Latitude is not within expected bounds. The minimum and maximum are -3.6893488147419103e+19, 3.6893488147419103e+19
Exception: Time is not in expected range. Time range is 3991-12-02T12:00:00 to 4015-11-29T12:00:00
Datasets failing on get_bbox #11

These datasets were partially scanned (some files failed and some did not):

c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.6hr.vas.v20190926
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.3hr.tas.v20190926
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.3hr.pr.v20190926
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.3hr.psl.v20190926
c3s-cordex.output.NAM-44.DMI.ECMWF-ERAINT.evaluation.r1i1p1.DMI-HIRHAM5.v1.day.clt.v20200831
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.3hr.rsds.v20190926
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.3hr.rlds.v20190926
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.3hr.sfcWind.v20190926
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.3hr.ps.v20190926
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.6hr.uas.v20190926
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.3hr.hurs.v20190926
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.3hr.rsus.v20190926
c3s-cordex.output.EUR-11.DMI.MOHC-HadGEM2-ES.rcp85.r1i1p1.DMI-HIRHAM5.v2.3hr.huss.v20190926

So any successful runs were removed.

cehbrecht · 2021-12-08T15:56:06Z

I have compared the missing CORDEX files with our replicated data archive. We have those files about with a newer version.

Instead of:

c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.psl.v20210111

We have:

c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.psl.v20210505

@glevava Do you have an updated version of the manifest?
https://github.com/cp4cds/c3s_34g_manifests/blob/master/cordex/manifest_C3S_34B_Lot1_CORDEX_20210708_fixed.tgz

ellesmith88 · 2021-12-09T11:18:35Z

I've just looked again and so to give a bit more information:

Of the partially scanned datasets - some files of these datasets fail because of errors like:
Exception: Time is not in expected range. Time range is 3991-12-02T12:00:00 to 4015-11-29T12:00:00
Except for c3s-cordex.output.NAM-44.DMI.ECMWF-ERAINT.evaluation.r1i1p1.DMI-HIRHAM5.v1.day.clt.v20200831
which fails because of Exception: Latitude is not within expected bounds. The minimum and maximum are 12.538726806640625, 9.969209968386869e+36 on 1 file out of 6.

The file that fails is /gws/nopw/j04/cp4cds1_vol1/data/c3s-cordex/output/NAM-44/DMI/ECMWF-ERAINT/evaluation/r1i1p1/DMI-HIRHAM5/v1/day/clt/v20200831/clt_NAM-44_ECMWF-ERAINT_evaluation_r1i1p1_DMI-HIRHAM5_v1_day_20010101-20051231.nc
When looking at the latitude values in this file, the value 9.969209968386869e+36 is only in the data a few times.
This is the default fill value used by netCDF4
The other files from this dataset don't appear to have a fill value in xarray either

For the error Latitude is not within expected bounds. I have looked at a few example files (All seem to be ICHEC-EC-EARTH):

/gws/nopw/j04/cp4cds1_vol1/data/c3s-cordex/output/EUR-11/DMI/ICHEC-EC-EARTH/historical/r12i1p1/DMI-HIRHAM5/v1/3hr/sfcWind/v20190926/sfcWind_EUR-11_ICHEC-EC-EARTH_historical_r12i1p1_DMI-HIRHAM5_v1_3hr_195901010000-195912312100.nc:
The minimum and maximum are -3.6893488147419103e+19, 3.6893488147419103e+19

Dimensions:       (rlat: 412, rlon: 424, time: 2920)
Coordinates:
    lat           (rlat, rlon) float64 ...
    lon           (rlat, rlon) float64 ...
  * rlat          (rlat) float64 -23.38 -23.26 -23.16 ... 21.61 21.73 21.83
  * rlon          (rlon) float64 -28.38 -28.26 -28.16 ... 17.93 18.05 18.16
  * time          (time) datetime64[ns] 1959-01-01 ... 1959-12-31T21:00:00
    height        float64

Looking at the dataset shows the same is true for lon.
When I open it with xarray I can't see a fill value for lat or lon or the dataset as a whole.
When I look at it as a netCDF dataset, it still has the same min/max and says a default fill value of 9.969209968386869e+36 is being used.

Looking at the data, most latitude values are something like 2.96337414e+00, here's a subset of the output of ds.lat.values[0]:

array([ 3.68934881e+19,  2.84355974e+00,  3.68934881e+19,  2.84418488e+00,
       -3.68934881e+19,  2.84480786e+00, -1.08420217e-19,  2.84542894e+00,
       -3.68934881e+19,  2.84604788e+00,  3.68934881e+19,  2.84666491e+00,
        1.08420217e-19,  2.84727979e+00,  3.68934881e+19,  2.84789252e+00,
       -2.00000000e+00,  2.84850311e+00, -0.00000000e+00,  2.84911180e+00,
       -0.00000000e+00,  2.84971833e+00, -2.00000000e+00,  2.85032272e+00,
        2.00000000e+00,  2.85092521e+00,  0.00000000e+00,  2.85152555e+00,
       -3.68934881e+19,  2.85212350e+00,  0.00000000e+00,  2.85271978e+00,
        2.00000000e+00,  2.85331368e+00, -2.00000000e+00,  2.85390544e+00,
        3.68934881e+19,  2.85449529e+00,  0.00000000e+00,  2.85508299e+00,
       -3.68934881e+19,  2.85566831e+00, -2.00000000e+00,  2.85625172e+00,
       -2.00000000e+00,  2.85683298e+00, -3.68934881e+19,  2.85741210e+00,
 ...

So it looks as though they are maybe meant to be fill values, but it's strange because there are some that are -3.68934881e+19 or 3.68934881e+19 and some that are 1.08420217e-19 or -1.08420217e-19.

The other files all show the same thing.

agstephens · 2021-12-09T11:29:58Z

Thanks @ellesmith88, could you ncdump the sections in the example files to check whether the coordinate variables include a _FillValue or missing_value attribute?

ellesmith88 · 2021-12-09T11:57:42Z

@agstephens They don't include a _FillValue or missing_value

For the main variable in the file I looked at above there is

sfcWind:_FillValue = 1.e+20f ;
sfcWind:missing_value = 1.e+20f ;

but none for the coordinate variables

agstephens · 2021-12-09T13:44:10Z

@ellesmith88 thanks. It looks like this is a problem with the data and we should just exclude them. Please generate an output file and attach it here that records the errors, e.g. one per line:

<dsid>:<error string>

Then we can look it up easily later. Thanks

glevava · 2021-12-09T15:18:49Z

I have compared the missing CORDEX files with our replicated data archive. We have those files about with a newer version.

Instead of:
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.psl.v20210111
We have:
c3s-cordex.output.MED-11.ICTP.MOHC-HadGEM2-ES.historical.r1i1p1.ICTP-RegCM4-3.v1.mon.psl.v20210505
@glevava Do you have an updated version of the manifest? https://github.com/cp4cds/c3s_34g_manifests/blob/master/cordex/manifest_C3S_34B_Lot1_CORDEX_20210708_fixed.tgz

I corrected the manifest. This was an old version. v20210505 we all have locally is the right one.
New manifest available here : https://vesg.ipsl.upmc.fr/thredds/fileServer/IPSLFS/glipsl/manifest_C3S_34B_Lot1_CORDEX_20210708_fixed.txt

agstephens · 2021-12-09T16:20:13Z

Thanks @glevava

@ellesmith88 please can you try a rescan based on the new manifest provided by Guillaume.

cehbrecht · 2021-12-10T17:36:27Z

@ellesmith88 please can you try a rescan based on the new manifest provided by Guillaume.

@ellesmith88 I have opened a PR with the fixed version of the cordex manifest:
cp4cds/c3s_34g_manifests#22

@glevava The fixed manifest in this PR differs from yours. The version mismatch was for 36 datasets (see above) not just one. I have manually updated it.

I have added a notebook in the PR to compare the manifest with the scanned files on disk (which should be available at all 3 sites). The fixed manifest contains now only available datasets. But it is missing 855 datasets which are on disk. @glevava are these for a new cordex batch to the CDS?

Notebook: https://github.com/cehbrecht/c3s_34g_manifests/blob/fix-cordex-manifest2/notebooks/cordex_compare_manifest.ipynb

ellesmith88 · 2021-12-13T12:00:59Z

The rescan is in progress, I'm attaching the error output from the first scan.
c3s-cordex-errors.txt

I will generate a new error output after the rescan is complete.

ellesmith88 · 2021-12-13T12:46:51Z

Here is the updated error output from the rescan
c3s-cordex-errors-updated.txt

The missing datasets are now included and the error from issue #11 has been resolved, now there are 23171 datasets successfully scanned out of 23196

agstephens · 2021-12-16T09:13:22Z

@ellesmith88 Please construct and publish the CORDEX catalog to github.

agstephens assigned ellesmith88 Nov 15, 2021

cehbrecht mentioned this issue Dec 10, 2021

fixed cordex manifest cp4cds/c3s_34g_manifests#22

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Investigate scanning CORDEX data with catalog-maker #10

Investigate scanning CORDEX data with catalog-maker #10

agstephens commented Nov 15, 2021

ellesmith88 commented Nov 15, 2021

agstephens commented Nov 15, 2021

cehbrecht commented Nov 16, 2021

glevava commented Nov 16, 2021

cehbrecht commented Nov 16, 2021

agstephens commented Nov 16, 2021

agstephens commented Nov 16, 2021

ellesmith88 commented Nov 22, 2021

agstephens commented Dec 1, 2021

ellesmith88 commented Dec 2, 2021 •

edited

Loading

ellesmith88 commented Dec 6, 2021 •

edited

Loading

cehbrecht commented Dec 8, 2021

ellesmith88 commented Dec 9, 2021 •

edited

Loading

agstephens commented Dec 9, 2021

ellesmith88 commented Dec 9, 2021 •

edited

Loading

agstephens commented Dec 9, 2021

glevava commented Dec 9, 2021 •

edited

Loading

agstephens commented Dec 9, 2021

cehbrecht commented Dec 10, 2021

ellesmith88 commented Dec 13, 2021 •

edited

Loading

ellesmith88 commented Dec 13, 2021

agstephens commented Dec 16, 2021

Investigate scanning CORDEX data with catalog-maker #10

Investigate scanning CORDEX data with catalog-maker #10

Comments

agstephens commented Nov 15, 2021

ellesmith88 commented Nov 15, 2021

agstephens commented Nov 15, 2021

cehbrecht commented Nov 16, 2021

glevava commented Nov 16, 2021

cehbrecht commented Nov 16, 2021

agstephens commented Nov 16, 2021

agstephens commented Nov 16, 2021

ellesmith88 commented Nov 22, 2021

agstephens commented Dec 1, 2021

ellesmith88 commented Dec 2, 2021 • edited Loading

ellesmith88 commented Dec 6, 2021 • edited Loading

cehbrecht commented Dec 8, 2021

ellesmith88 commented Dec 9, 2021 • edited Loading

agstephens commented Dec 9, 2021

ellesmith88 commented Dec 9, 2021 • edited Loading

agstephens commented Dec 9, 2021

glevava commented Dec 9, 2021 • edited Loading

agstephens commented Dec 9, 2021

cehbrecht commented Dec 10, 2021

ellesmith88 commented Dec 13, 2021 • edited Loading

ellesmith88 commented Dec 13, 2021

agstephens commented Dec 16, 2021

ellesmith88 commented Dec 2, 2021 •

edited

Loading

ellesmith88 commented Dec 6, 2021 •

edited

Loading

ellesmith88 commented Dec 9, 2021 •

edited

Loading

ellesmith88 commented Dec 9, 2021 •

edited

Loading

glevava commented Dec 9, 2021 •

edited

Loading

ellesmith88 commented Dec 13, 2021 •

edited

Loading