Influxdb2 full restore error: failed to restore TSM snapshot for shard XXX: 500 Internal Server Error

[tomklapka]

Steps to reproduce:

1. influx backup --host <HOST_URL> --token <TOKEN> --org '<ORG>' /path/to/backup

2. influx restore --host <HOST_URL> --full --token <TOKEN> /path/to/backup

Expected behaviour:
Successful restore of db

Actual behaviour:
InfluxCLI fails on Error: failed to restore TSM snapshot for shard 10954: 500 Internal Server Error

Environment info:

K3s=1.31.3, Official influxdb:2.7.5 docker image.
Influx CLI dev (git: a79a2a1b825867421d320428538f76a4c90aa34c) build_date: 2024-04-16T14:34:32Z

Config:
DOCKER_INFLUXDB_INIT_MODE=setup
DOCKER_INFLUXDB_INIT_ORG=default
DOCKER_INFLUXDB_INIT_BUCKET=default
DOCKER_INFLUXDB_INIT_RETENTION=48h
INFLUXD_BOLT_PATH=/var/lib/influxdb2/influxd.bolt
INFLUXD_ENGINE_PATH=/var/lib/influxdb2
INFLUXD_SESSION_LENGTH=1440
INFLUXD_REPORTING_DISABLED=true
DOCKER_INFLUXDB_INIT_USERNAME=XXX
DOCKER_INFLUXDB_INIT_PASSWORD=XXX
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=XXX

Logs:

influxdb2-restore-error.log

[tomklapka]

Possible duplicates:
#23212
#15323

[davidby-influx]

It looks like the shard being restored is outside of its retention period, so has been reaped.

ts=2025-01-27T19:48:16.524579Z lvl=info msg="Attempting deletion of shard from store" log_id=0uHxwn20000 service=retention op_name=retention_delete_check db_instance=5ad8ce2b635727a6 db_shard_id=10954 db_rp=autogen
ts=2025-01-27T19:48:16.532477Z lvl=info msg="Deleted shard" log_id=0uHxwn20000 service=retention op_name=retention_delete_check db_instance=5ad8ce2b635727a6 db_shard_id=10954 db_rp=autogen

ts=2025-01-27T20:00:59.402699Z lvl=warn msg="internal error not returned to client" log_id=0uHxwn20000 handler=error_logger error="shard 10954 doesn't exist on this server"