GBFS Feed results in dagster-pipeline exception

[ThorstenFroehlinghaus]

The donkey_le_locle feed causes the following exception:

ERROR:root:Error retrieving vehicles for system {'id': 'donkey_le_locle', 'url': 'http://lamassu:8080/gbfs-internal/donkey_le_locle/gbfs'}
Traceback (most recent call last):
File "/opt/dagster/app/pipeline/assets/sharing.py", line 103, in vehicles
vehicles = lamassu.get_vehicles_as_frame(feeds, system_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/dagster/app/pipeline/resources/lamassu.py", line 30, in get_vehicles_as_frame
return self._lamassu().get_vehicles_as_frame(feed, system_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/dagster/app/pipeline/sources/lamassu.py", line 208, in get_vehicles_as_frame
merged = pd.merge(free_vehicle_status_df, vehicle_types_df, on=['vehicle_type_id'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pandas/core/reshape/merge.py", line 170, in merge
op = _MergeOperation(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pandas/core/reshape/merge.py", line 794, in init
) = self._get_merge_keys()
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pandas/core/reshape/merge.py", line 1297, in _get_merge_keys
right_keys.append(right._get_label_or_level_values(rk))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pandas/core/generic.py", line 1911, in _get_label_or_level_values
raise KeyError(key)
KeyError: 'vehicle_type_id'

The feed contains invalid vehicle_ids. This, however, should not result in an exception.

[hbruch]

The GBFS feed is invalid and the issue needs to be fixed at the source. The Error reports the issues with this feed. Assuming any kind of default would be inappropriate. The exception highlights the cause of this issue and is IMHO appropriate, as we should not work around bad quality downstream but have it fixed upstream.

[ThorstenFroehlinghaus]

This should be a topic for our retroperspektive. From my point of view, invalid data may occor at any time, and we will not always be able to fix them in time. Errors are not exceptions - we must be able to deal with them.