Parse legacy S3 bucket objects using Vector.

[neeltom92]

We currently use Vector to send our logs to our log aggregator UI for searching. Our setup includes using SQS for object notifications from an AWS S3 source.

We now have a requirement to ingest historical logs stored in a different S3 bucket, containing data from 2019 to 2023, and send them to our log aggregator UI for indexing and searching.

I only know how to ingest new objects as they get updated. How can we configure Vector to also ingest and parse these legacy logs from the existing S3 bucket and forward them to our log aggregation tool?

[pront]

I don't see anything in the docs that enables this use case. You could use a script to download the historical data and forward them to Vector with HTTP.

[neeltom92]

thanks