Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect _FillValue in netCDF output when exporting data from fci_l1c_nc reader #2937

Open
vsanvir opened this issue Oct 17, 2024 · 0 comments

Comments

@vsanvir
Copy link

vsanvir commented Oct 17, 2024

Describe the bug
Incorrect _FillValue in NetCDF output when saving MTG data that has been loaded with other calibration than 'count'. Pixels outside the disk get assigned zeros, while netcdf attributes report _FillValue to be 65535.

To Reproduce

from satpy.scene import Scene
from satpy import find_files_and_readers
from netCDF4 import Dataset

import hdf5plugin

files = find_files_and_readers(base_dir=path_to_data, reader="fci_l1c_nc")
scn = Scene(filenames=files)
scn.load(["vis_06"])
start_time = scn["vis_06"].start_time.strftime("%Y%m%d%H%M%S")
end_time = scn["vis_06"].end_time.strftime("%Y%m%d%H%M%S")
vis_output = f"{output_path}/{start_time}-vis_06.nc"
save_lonlats = False
scn.save_datasets(
    writer="cf",
    datasets=["vis_06"],
    encoding={
        "vis_06": {
            "zlib": True,
            "dtype": "uint16",
            "scale_factor": vis_06_encoding["scale_factor"],
            "add_offset": vis_06_encoding["add_offset"],
            # "_FillValue": 0
        }
    },
    exclude_attrs=["raw_metadata"],
    include_lonlats=save_lonlats,
    filename=vis_output,
)

Expected behavior
I would expect the _FillValue would be set so that the parts outside the disk would contain the fill value in the netcdf output. In the example case zeros are assigned to pixels outside the fulldisk, when the netcdf reports _FillValue to be 65535US.

Also setting _FillValue returns xarray exception "ValueError: failed to prevent overwriting existing key _FillValue in attrs on variable 'vis_06'. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.". I would expect overriding the _FillValue this way to work.

Actual results
Text output of actual results or error messages including full tracebacks if applicable.

Environment Info:

  • OS: OSX
  • Satpy Version: 0.51.0
  • PyResample Version: 1.30.0
  • Readers and writers dependencies (when relevant):
    Readers
    =======
    abi_l1b: ok
    abi_l1b_scmi: ok
    abi_l2_nc: ok
    acspo: ok
    agri_fy4a_l1: ok
    agri_fy4b_l1: ok
    ahi_hrit: ok
    ahi_hsd: ok
    ahi_l1b_gridded_bin: ok
    ahi_l2_nc: ok
    ami_l1b: ok
    amsr2_l1b: ok
    amsr2_l2: ok
    amsr2_l2_gaasp: ok
    amsub_l1c_aapp: ok
    ascat_l2_soilmoisture_bufr: cannot find module 'satpy.readers.ascat_l2_soilmoisture_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'")))
    atms_l1b_nc: ok
    atms_sdr_hdf5: ok
    avhrr_l1b_aapp: ok
    avhrr_l1b_eps: cannot find module 'satpy.readers.eps_l1b' (No module named 'defusedxml')
    avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
    avhrr_l1b_hrpt: ok
    avhrr_l1c_eum_gac_fdr_nc: ok
    caliop_l2_cloud: cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf')
    clavrx: cannot find module 'satpy.readers.clavrx' (No module named 'pyhdf')
    cmsaf-claas2_l2_nc: ok
    electrol_hrit: ok
    epic_l1b_h5: ok
    fci_l1c_nc: ok
    fci_l2_bufr: cannot find module 'satpy.readers.eum_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
    fci_l2_nc: ok
    fy3a_mersi1_l1b: ok
    fy3b_mersi1_l1b: ok
    fy3c_mersi1_l1b: ok
    generic_image: cannot find module 'satpy.readers.generic_image' (No module named 'rioxarray')
    geocat: ok
    gerb_l2_hr_h5: ok
    ghi_l1: ok
    ghrsst_l2: ok
    gld360_ualf2: ok
    glm_l2: ok
    gms5-vissr_l1b: cannot find module 'satpy.readers.gms.gms5_vissr_l1b' (No module named 'numba')
    goci2_l2_nc: ok
    goes-imager_hrit: ok
    goes-imager_nc: ok
    gpm_imerg: ok
    grib: cannot find module 'satpy.readers.grib' (No module named 'pygrib')
    hsaf_grib: cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib')
    hsaf_h5: ok
    hy2_scat_l2b_h5: ok
    iasi_l2: ok
    iasi_l2_cdr_nc: ok
    iasi_l2_so2_bufr: cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'")))
    ici_l1b_nc: ok
    insat3d_img_l1b_h5: cannot find module 'satpy.readers.insat3d_img_l1b_h5' (No module named 'datatree' It can be installed with the xarray-datatree package.)
    jami_hrit: ok
    li_l2_nc: ok
    maia: ok
    meris_nc_sen3: ok
    mersi2_l1b: ok
    mersi3_l1b: ok
    mersi_ll_l1b: ok
    mersi_rm_l1b: ok
    mhs_l1c_aapp: ok
    mimicTPW2_comp: ok
    mirs: ok
    modis_l1b: cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf')
    modis_l2: cannot find module 'satpy.readers.modis_l2' (No module named 'pyhdf')
    modis_l3: cannot find module 'satpy.readers.modis_l3' (No module named 'pyhdf')
    msi_safe: cannot find module 'satpy.readers.msi_safe' (No module named 'defusedxml')
    msi_safe_l2a: cannot find module 'satpy.readers.msi_safe' (No module named 'defusedxml')
    msu_gsa_l1b: ok
    mtsat2-imager_hrit: ok
    mviri_l1b_fiduceo_nc: ok
    mwi_l1b_nc: ok
    mws_l1b_nc: ok
    nucaps: ok
    nwcsaf-geo: ok
    nwcsaf-msg2013-hdf5: ok
    nwcsaf-pps_nc: ok
    oceancolorcci_l3_nc: ok
    olci_l1b: ok
    olci_l2: ok
    omps_edr: ok
    osisaf_nc: ok
    safe_sar_l2_ocn: ok
    sar-c_safe: cannot find module 'satpy.readers.sar_c_safe' (No module named 'defusedxml')
    satpy_cf_nc: ok
    scatsat1_l2b: cannot find module 'satpy.readers.scatsat1_l2b' (cannot import name 'Dataset' from 'satpy.dataset' (/Users/ANVIR/micromamba/envs/satpy/lib/python3.12/site-packages/satpy/dataset/init.py))
    seadas_l2: cannot find module 'satpy.readers.seadas_l2' (No module named 'pyhdf')
    seviri_l1b_hrit: ok
    seviri_l1b_icare: cannot find module 'satpy.readers.seviri_l1b_icare' (No module named 'pyhdf')
    seviri_l1b_native: ok
    seviri_l1b_nc: ok
    seviri_l2_bufr: cannot find module 'satpy.readers.eum_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
    seviri_l2_grib: cannot find module 'satpy.readers.seviri_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
    sgli_l1b: ok
    slstr_l1b: ok
    smos_l2_wind: ok
    tropomi_l2: ok
    vii_l1b_nc: ok
    vii_l2_nc: ok
    viirs_compact: ok
    viirs_edr: ok
    viirs_edr_active_fires: ok
    viirs_edr_flood: cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf')
    viirs_l1b: ok
    viirs_l2: ok
    viirs_sdr: ok
    viirs_vgac_l1c_nc: ok
    virr_l1b: ok

Writers

awips_tiled: ok
cf: ok
geotiff: ok
mitiff: ok
ninjogeotiff: ok
ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
simple_image: ok

Extras

cartopy: No module named 'cartopy'
geoviews: No module named 'geoviews'

Additional context
Discussion thread on Pytroll slack: https://pytroll.slack.com/archives/C0LNH7LMB/p1729147424510329

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant