modis_regrid data_file help #2605
-
|
Hi METplus team, |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 5 replies
-
|
Hi @aleya-kaushik, I see you are looking for information on the format of data that can be passed to the modis_regrid tool with the -data_file argument. A description of the valid data files that can be passed to MET tools can be found in the Input data formats section of the MET Data I/O chapter of the MET User's Guide. In short, MET can accept GRIB1, GRIB2, CF-compliant NetCDF, and NetCDF files that were generated by another MET tool. A good test to see if a data file can be read by MET is to use the Plot-Data-Plane tool. To read files into MET that don't meet these criteria, MET Python Embedding functionality can be used to pass data to the MET tools through a Python script. If you would like us to take a look at your input file to see if it can be read into MET, please provide a sample file using the instructions on discussion #954. |
Beta Was this translation helpful? Give feedback.
-
|
Hi,
Thanks very much for the info. I've uploaded the two files I'm attempting
to use via the ftp instructions you sent (to
/incoming/irap/met_help/Kaushik_data). Could you please take a look? This
is the command I'm trying to run:
modis_regrid \
-data_file
/work2/noaa/co2/kaushik/MODIS/MCD12C1_v061/mcd12c1_MajorIGBP_2020.nc \
-field LC_Prop3 \
-out /work2/noaa/co2/kaushik/MODIS/MCD12Q1_v061/2020_test_regrid.nc \
-scale 1.0 \
-offset 0.0 \
-fill -1 \
/work2/noaa/co2/kaushik/MODIS/MCD12Q1_v061/MCD12Q1_A2020.hdf
Thanks,
Aleya
…On Mon, Jun 3, 2024 at 10:29 AM George McCabe ***@***.***> wrote:
Hi @aleya-kaushik <https://github.com/aleya-kaushik>,
I see you are looking for information on the format of data that can be
passed to the modis_regrid tool with the -data_file argument. A description
of the valid data files that can be passed to MET tools can be found in the Input
data formats
<https://met.readthedocs.io/en/latest/Users_Guide/data_io.html#input-data-formats>
section of the MET Data I/O chapter of the MET User's Guide. In short, MET
can accept GRIB1, GRIB2, CF-compliant NetCDF, and NetCDF files that were
generated by another MET tool. A good test to see if a data file can be
read by MET is to use the Plot-Data-Plane
<https://met.readthedocs.io/en/latest/Users_Guide/plotting.html#plot-data-plane-usage>
tool.
To read files into MET that don't meet these criteria, MET Python
Embedding
<https://met.readthedocs.io/en/latest/Users_Guide/appendixF.html>
functionality can be used to pass data to the MET tools through a Python
script.
If you would like us to take a look at your input file to see if it can be
read into MET, please provide a sample file using the instructions on
discussion #954 <#954>.
—
Reply to this email directly, view it on GitHub
<#2605 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A52NHVGPUU4CT46WBVRDTCLZFSKV5AVCNFSM6AAAAABIVV6RRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TMNJRGIZTE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Dr. Aleya Kaushik
Research Scientist
CIESRDS and CIRES | University of Colorado Boulder
Working in the NOAA Global Monitoring Laboratory
325 Broadway GML | Boulder, CO 80305
Google Voice: (720) 263-6375
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Aleya, Thank you for sending us your files. I was able to get the data file read into the Plot-Data-Plane tool by using the
However, it looks like the One way to get around this is to run another MET tool like Pcp-Combine or Regrid-Data-Plane on the data file to convert it into a MET NetCDF files that can be read easily by all of the MET tools. I ran the following command to accomplish this:
I then ran modis_regrid, passing in the file I generated from Pcp-Combine instead.
This run threw a different error:
It appears that the tool is having issues parsing out any swaths from the MODIS file. I will need to reach out to another METplus team member who is more experienced with this tool for further assistance. Unfortunately, a few members of our team are out on vacation this week, so there will likely be a delay in getting a response. I apologize for the inconvenience, but will follow up as soon as possible. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @aleya-kaushik, Could you give some information about the MODIS file source? I spoke with another METplus team member who suspects that the format of the MODIS files has changed since this tool was developed. If this is the case, then modifications to the tool would be needed to process these data. Another approach would be to write a Python Embedding script to read the data into another MET tool for processing. |
Beta Was this translation helpful? Give feedback.
-
|
Hi,
I downloaded the MODIS data via NASA's Earthdata tool, then processed the
swaths for one variable with the HEG tool described here:
https://hdfeos.org/software/heg.php
I use the second option in the tool ("Hdf-Eos to Stich/Subset"). This way I
now have one annual HDF file which covers the entire domain I need. But I'm
having trouble reading HDF files in Python so I thought to convert it to a
netcdf format which is also gridded by lat/lon because I need to regrid the
500m data to 0.05deg.
If you have any advice for achieving this without using modis_regrid, I'd
appreciate any thoughts!
Thanks,
Aleya
…On Tue, Jun 4, 2024 at 10:45 AM George McCabe ***@***.***> wrote:
Hi @aleya-kaushik <https://github.com/aleya-kaushik>, Could you give some
information about the MODIS file source? I spoke with another METplus team
member who suspects that the format of the MODIS files has changed since
this tool was developed. If this is the case, then modifications to the
tool would be needed to process these data. Another approach would be to
write a Python Embedding script to read the data into another MET tool for
processing.
—
Reply to this email directly, view it on GitHub
<#2605 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A52NHVHHC6UOP7E5CFYEFILZFXVJLAVCNFSM6AAAAABIVV6RRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TMNRXGM3TM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Dr. Aleya Kaushik
Research Scientist
CIESRDS and CIRES | University of Colorado Boulder
Working in the NOAA Global Monitoring Laboratory
325 Broadway GML | Boulder, CO 80305
Google Voice: (720) 263-6375
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @aleya-kaushik, Thank you for the info. Have you tried processing one of the HDF files before running that tool to stitch them together? I reached out to a colleague to see if they have any other advice. |
Beta Was this translation helpful? Give feedback.
-
|
@hsoh-u suggested that the Python NetCDF4 package can read HDF5 files. I tested this and was able to read your file into Python using that package: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for that tip, I had not checked the netCDF4 library. I guess I'll
look into regridding options within python.
I would like to try to use the MET tools on the non-stitched files but I'm
running into an error in the pcp_combine pre-processing step you suggested.
See below. Would it be possible for you to upload the pcp_out.nc data file
you created on your end so I can test the modis_regrid command with one of
the non-stitched files?
Orion-login-4[13] akaushik$ pwd
/apps/contrib/MET/9.1/bin
Orion-login-4[14] akaushik$ pcp_combine -add
/work2/noaa/co2/kaushik/MODIS/MCD12C1_v061/mcd12c1_MajorIGBP_2020.nc
pcp_out.nc -field 'name=Majority_Land_Cover_Type_1"; level="*,*";
file_type=NETCDF_NCCF;'
DEBUG 2: Since the "-field" command line option was used, parsing the
command line arguments as a list of files.
DEBUG 2: Performing derivation command (sum) for 1 files.
DEBUG 1: Reading data (name=Majority_Land_Cover_Type_1"; level="*,*";
file_type=NETCDF_NCCF;) from input file:
/work2/noaa/co2/kaushik/MODIS/MCD12C1_v061/mcd12c1_MajorIGBP_2020.nc
ERROR :
ERROR : yyerror() -> syntax error in file "/tmp/met_config_204384_0"
ERROR :
ERROR : line = 1
ERROR :
ERROR : column = 23
ERROR :
ERROR : text = "; level="
ERROR :
ERROR :
ERROR : name=Majority_Land_Cover_Type_1"; level="*,*";
file_type=NETCDF_NCCF;
ERROR :
…______________________^^^^^^^^_______________________________________
ERROR :
On Tue, Jun 4, 2024 at 11:44 AM George McCabe ***@***.***> wrote:
@hsoh-u <https://github.com/hsoh-u> suggested that the Python NetCDF4
package can read HDF5 files. I tested this and was able to read your file
into Python using that package:
import netCDF4
nc_obj = netCDF4.Dataset('./MCD12Q1_A2020.hdf')
—
Reply to this email directly, view it on GitHub
<#2605 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A52NHVGEZZCGC32HUBGYVDDZFX4HNAVCNFSM6AAAAABIVV6RRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TMNRXHE3DO>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Dr. Aleya Kaushik
Research Scientist
CIESRDS and CIRES | University of Colorado Boulder
Working in the NOAA Global Monitoring Laboratory
325 Broadway GML | Boulder, CO 80305
Google Voice: (720) 263-6375
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @aleya-kaushik , you are missing a quotation mark after |
Beta Was this translation helpful? Give feedback.
-
|
Aha thanks for catching that!
I was able to create the pcp_out file, but the modis_regrid command is
still failing on the non-stitched files with
StringArray::operator[](int) const -> range check error!
I will look into the netCDF4 and pyhdf tools.
Thanks for all your help!
Aleya
…On Tue, Jun 4, 2024 at 12:30 PM George McCabe ***@***.***> wrote:
Hi @aleya-kaushik <https://github.com/aleya-kaushik> , you are missing a
quotation mark after name= and before Majority_Land_Cover_Type_1";.
Adding that character should fix your issue.
—
Reply to this email directly, view it on GitHub
<#2605 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A52NHVDDAVPGH4BR54VUCFLZFYBS3AVCNFSM6AAAAABIVV6RRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TMNRYGM4DK>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Dr. Aleya Kaushik
Research Scientist
CIESRDS and CIRES | University of Colorado Boulder
Working in the NOAA Global Monitoring Laboratory
325 Broadway GML | Boulder, CO 80305
Google Voice: (720) 263-6375
|
Beta Was this translation helpful? Give feedback.
Hi @aleya-kaushik,
I see you are looking for information on the format of data that can be passed to the modis_regrid tool with the -data_file argument. A description of the valid data files that can be passed to MET tools can be found in the Input data formats section of the MET Data I/O chapter of the MET User's Guide. In short, MET can accept GRIB1, GRIB2, CF-compliant NetCDF, and NetCDF files that were generated by another MET tool. A good test to see if a data file can be read by MET is to use the Plot-Data-Plane tool.
To read files into MET that don't meet these criteria, MET Python Embedding functionality can be used to pass data to the MET tools through a Python script.
If you wou…