AWS S3 to S3 transfer without Dataplane as middleware

[rafaelmag110]

Feature Request

During an AWS S3 to S3 transfer using the Connector, the current transfer behaviour is for the S3 DataSource to get the transfer payload from the source bucket into memory and push the payload into the destination bucket. This creates a bottleneck in the dataplane as the payload can consume all the host memory resources.

As a feature request, i'd like to explore the idea of levering the AWS S3 API to trigger a direct transfer from one bucket to another, without any payload even reaching the dataplane. In this case, the connector would act as a transfer coordinator only.

This only applies to AWS S3 to S3 transfers.

Which Areas Would Be Affected?

S3 DataSource and Sink.
S3 Provisioners would have to be changed aswell to correctly setup the environment for the transfer (permission management).

Why Is the Feature Desired?

Benefits in performance.

Solution Proposal

N/A

[mspiekermann]

@rafaelmag110 we like the idea and it makes sense. Therefore, we transferred the issue into a discussion that we can use for further exploration and sharing of ideas. It would be great if you can start by providing a more detailed descriptions on your feature request of levering the AWS S3 API to trigger a direct transfer from one bucket to another, without any payload even reaching the dataplane.