Saving channel 3A fails with struct.error: argument out of range

[gerritholl]

I'm not sure if this is a problem in pyninjotiff or in pytroll/satpy#1653; the problem occurs with the satpy PR but not with satpy main, but it might nevertheless be a problem in pyninjotiff. I'm putting this here and will close if it turns out if pyninjotiff is nothing to blame.

I have a file, produced with AAPP, for which attempting to write channel 3A with pyninjotiff fails with struct.error: argument out of range.

MCVE

from satpy import Scene
from satpy.utils import debug_on; debug_on()

fn = ["/media/nas/x21308/scratch/AAPP-processed/hrpt_noaa19_20210508_1435_63129.l1b"]
fn_out = "{start_time:%Y%m%d%H%M}-{platform_name}_{sensor}_{area.area_id}_{name}.tiff"

sc = Scene(filenames=fn, reader=["avhrr_l1b_aapp"])
sc.load(["3a"])
ls = sc.resample("nqeuro1km")

ls.save_dataset("3a",
        writer="ninjotiff",
        physic_unit="%",
        ch_min_measurement_unit=0,
        ch_max_measurement_unit=125,
        filename=fn_out,
        sat_id=0,
        chan_id=0,
        data_cat="PORN",
        nbits=8,
        data_source="HRPT")

Expected result

Probably a blank file like for the corresponding GeoTIFF or the same blank file I get with satpy main.

Actual result

I get an empty file 195001010000-NOAA-19_avhrr-3_nqeuro1km_3a.tiff and a struct.error exception:

[DEBUG: 2021-05-10 12:07:06 : satpy.readers.yaml_reader] Reading ('/home/gholl/checkouts/satpy/satpy/etc/readers/avhrr_l1b_aapp.yaml',)
[DEBUG: 2021-05-10 12:07:06 : satpy.readers.yaml_reader] Assigning to avhrr_l1b_aapp: ['/media/nas/x21308/scratch/AAPP-processed/hrpt_noaa19_20210508_1435_63129.l1b']
[DEBUG: 2021-05-10 12:07:06 : satpy.readers.aapp_l1b] Reading time 0:00:00.003673
[DEBUG: 2021-05-10 12:07:06 : satpy.composites.config_loader] Looking for composites config file avhrr-3.yaml
[DEBUG: 2021-05-10 12:07:06 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2021-05-10 12:07:06 : satpy.readers.yaml_reader] No coordinates found for DataID(name='longitude', resolution=1050, modifiers=())
[DEBUG: 2021-05-10 12:07:06 : satpy.readers.yaml_reader] No coordinates found for DataID(name='latitude', resolution=1050, modifiers=())
[INFO: 2021-05-10 12:07:06 : satpy.readers.aapp_l1b] No valid operational coefficients, fall back to pre-launch
[DEBUG: 2021-05-10 12:07:06 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2021-05-10 12:07:06 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2021-05-10 12:07:06 : satpy.scene] Resampling DataID(name='3a', wavelength=WavelengthRange(min=1.58, central=1.61, max=1.64, unit='µm'), resolution=1050, calibration=<calibration.reflectance>, modifiers=())
[INFO: 2021-05-10 12:07:06 : satpy.scene] Not reducing data before resampling.
[INFO: 2021-05-10 12:07:06 : satpy.resample] Using default KDTree resampler
[DEBUG: 2021-05-10 12:07:07 : satpy.resample] Check if ./resample_lut-804d3daba9ceafc721d1968df8e06836cdd1080a.npz exists
[DEBUG: 2021-05-10 12:07:07 : satpy.resample] Computing kd-tree parameters
/home/gholl/checkouts/pyresample/pyresample/kd_tree.py:1025: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  input_coords = input_coords.astype(np.float)
[DEBUG: 2021-05-10 12:07:07 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2021-05-10 12:07:07 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
/home/gholl/checkouts/pyresample/pyresample/kd_tree.py:1047: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  radius=self.radius_of_influence, dtype=np.int,
[DEBUG: 2021-05-10 12:07:07 : satpy.resample] Resampling where-bbdb931ad47efe6421d69d61a825cc34
[DEBUG: 2021-05-10 12:07:07 : satpy.writers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/writers/ninjotiff.yaml']
/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyninjotiff/tifffile.py:154: UserWarning: failed to import the optional _tifffile C extension module.
Loading of some compressed images will be slow.
Tifffile.c can be obtained at http://www.lfd.uci.edu/~gohlke/
  warnings.warn(
[DEBUG: 2021-05-10 12:07:07 : satpy.writers] Enhancement configuration options: [{'name': 'linear_stretch', 'method': <function stretch at 0x7f99d2842670>, 'kwargs': {'stretch': 'crude', 'min_stretch': 0.0, 'max_stretch': 100.0}}, {'name': 'gamma', 'method': <function gamma at 0x7f99d24a7280>, 'kwargs': {'gamma': 1.5}}]
[DEBUG: 2021-05-10 12:07:07 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': 0.0, 'max_stretch': 100.0}
[DEBUG: 2021-05-10 12:07:07 : trollimage.xrimage] Applying gamma 1.5
[INFO: 2021-05-10 12:07:07 : pyninjotiff.ninjotiff] Will generate single band product
/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4()
/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4()
/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4()
[INFO: 2021-05-10 12:07:07 : pyninjotiff.ninjotiff] Creating output file '/media/nas/x21308/plots_and_maps/2021/05/10/195001010000-NOAA-19_avhrr-3_nqeuro1km_3a.tiff'
[INFO: 2021-05-10 12:07:07 : pyninjotiff.ninjotiff] creating tags and data for a resolution 6000x12000
[DEBUG: 2021-05-10 12:07:07 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2021-05-10 12:07:07 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2021-05-10 12:07:07 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2021-05-10 12:07:07 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2021-05-10 12:07:07 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2021-05-10 12:07:07 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
Traceback (most recent call last):
  File "/home/gholl/checkouts/protocode/mwe/avhrr-struct-error.py", line 13, in <module>
    ls.save_dataset("3a",
  File "/home/gholl/checkouts/satpy/satpy/scene.py", line 986, in save_dataset
    return writer.save_dataset(self[dataset_id],
  File "/home/gholl/checkouts/satpy/satpy/writers/ninjotiff.py", line 204, in save_dataset
    return super(NinjoTIFFWriter, self).save_dataset(
  File "/home/gholl/checkouts/satpy/satpy/writers/__init__.py", line 811, in save_dataset
    return self.save_image(img, filename=filename, compute=compute, fill_value=fill_value, **kwargs)
  File "/home/gholl/checkouts/satpy/satpy/writers/ninjotiff.py", line 176, in save_image
    return nt.save(img, filename, data_is_scaled_01=True, compute=compute, **kwargs)
  File "/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyninjotiff/ninjotiff.py", line 560, in save
    return write(data, filename, area_def, ninjo_product_name, **kwargs)
  File "/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyninjotiff/ninjotiff.py", line 667, in write
    return _write(image_data, output_fn, write_rgb=write_rgb, **options)
  File "/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyninjotiff/ninjotiff.py", line 1062, in _write
    return tiffwrite(output_fn, image_data, args, tifargs, factors)
  File "/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyninjotiff/ninjotiff.py", line 1071, in tiffwrite
    tif.save(image_data, **args)
  File "/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyninjotiff/tifffile.py", line 671, in save
    addtag(*t)
  File "/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyninjotiff/tifffile.py", line 569, in addtag
    ifdentry.append(pack(val_format, pack(dtype, value)))
  File "/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyninjotiff/tifffile.py", line 545, in pack
    return struct.pack(byteorder+fmt, *val)
struct.error: argument out of range

Versions

• pyninjotiff v0.3.0
• satpy v0.27.0-85-g93ec12b1 (Set specific start and end times for channels 3A and 3B. satpy#1653)