Echec des exports avec le type numeric

[lpofredc]

Les exports (formats JSON, GeoJSON, GeoPackage) contenant un champ de type numeric (et sans doute aussi decimal) échouent avec l'erreur suivante dans le log du worker. Cela fonctionne correctement pour les exports CSV. Pour passer outre, il faut les convertir (en int ou float par exemple).

export geopackage

[2023-10-17 08:01:18,961: ERROR/ForkPoolWorker-4] Task gn_module_export.tasks.generate_export[32621287-d047-40be-9c10-bacc859dff88] raised unexpected: ValueError("Invalid field type <class 'decimal.Decimal'>")
Traceback (most recent call last):
  File "/home/gnadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/gnadmin/geonature/backend/geonature/celery_app.py", line 12, in __call__
    return self.run(*args, **kwargs)
  File "/home/gnadmin/gnModules/gn_module_export/backend/gn_module_export/tasks.py", line 62, in generate_export
    export_data_file(export_id, file_name, export_url, format, id_role, filters)
  File "/home/gnadmin/gnModules/gn_module_export/backend/gn_module_export/utils_export.py", line 153, in export_data_file
    raise exp
  File "/home/gnadmin/gnModules/gn_module_export/backend/gn_module_export/utils_export.py", line 140, in export_data_file
    export_as_file(
  File "/home/gnadmin/gnModules/gn_module_export/backend/gn_module_export/utils/export.py", line 29, in export_as_file
    _export_as_file(
  File "/home/gnadmin/gnModules/gn_module_export/backend/gn_module_export/utils/export.py", line 59, in _export_as_file
    export_geopackage(
  File "/home/gnadmin/geonature/backend/venv/lib/python3.9/site-packages/utils_flask_sqla_geo/export.py", line 160, in export_geopackage
    f.write(feature)
  File "/home/gnadmin/geonature/backend/venv/lib/python3.9/site-packages/fiona/collection.py", line 367, in write
    self.writerecords([record])
  File "/home/gnadmin/geonature/backend/venv/lib/python3.9/site-packages/fiona/collection.py", line 361, in writerecords
    self.session.writerecs(records, self)
  File "fiona/ogrext.pyx", line 1291, in fiona.ogrext.WritingSession.writerecs
  File "fiona/ogrext.pyx", line 466, in fiona.ogrext.OGRFeatureBuilder.build
ValueError: Invalid field type <class 'decimal.Decimal'>

export json/geojson

[2023-10-17 08:38:34,045: ERROR/ForkPoolWorker-4] Task gn_module_export.tasks.generate_export[cf26de65-3556-42ee-ab94-abc77f269e1e] raised unexpected: TypeError('Object of type Decimal is not JSON serializable')
Traceback (most recent call last):
  File "/home/gnadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/gnadmin/geonature/backend/geonature/celery_app.py", line 12, in __call__
    return self.run(*args, **kwargs)
  File "/home/gnadmin/gnModules/gn_module_export/backend/gn_module_export/tasks.py", line 62, in generate_export
    export_data_file(export_id, file_name, export_url, format, id_role, filters)
  File "/home/gnadmin/gnModules/gn_module_export/backend/gn_module_export/utils_export.py", line 153, in export_data_file
    raise exp
  File "/home/gnadmin/gnModules/gn_module_export/backend/gn_module_export/utils_export.py", line 140, in export_data_file
    export_as_file(
  File "/home/gnadmin/gnModules/gn_module_export/backend/gn_module_export/utils/export.py", line 29, in export_as_file
    _export_as_file(
  File "/home/gnadmin/gnModules/gn_module_export/backend/gn_module_export/utils/export.py", line 75, in _export_as_file
    func_dict[file_format](
  File "/home/gnadmin/geonature/backend/venv/lib/python3.9/site-packages/utils_flask_sqla_geo/export.py", line 90, in export_geojson
    for chunk in json.JSONEncoder().iterencode(feature_collection):
  File "/usr/lib/python3.9/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.9/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/usr/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.9/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Decimal is not JSON serializable

Versions utilisées:
GeoNature 2.13.0
Module d'export: 1.6.0

Pour reproduire, créer une vue avec un champ numeric. Par exemple:

CREATE VIEW gn_exports.test_numeric AS
SELECT ROW_NUMBER() OVER () AS id,
       EXTRACT(YEAR FROM t) AS year,
       geom
FROM GENERATE_SERIES('2000-01-01'::DATE, '2023-01-01'::DATE, '1 year') t
         CROSS JOIN (SELECT geom FROM ref_geo.l_areas WHERE id_type = ref_geo.get_id_area_type('M10') LIMIT 1) g;

Configurer l'export dans le backoffice et faire des tentatives d'export.

[amandine-sahl]

En fait le soucis est lié à Fiona et a la librairie json qui n'encode pas nativement les decimals.
https://marshmallow.readthedocs.io/en/latest/marshmallow.fields.html#marshmallow.fields.Decimal