Stream download large file from S3

[indjev99]

I want to download some large file (10s of Gigabytes or more) from S3 in a streamed fashion. I do NOT want to store the file to disk. I want to occasionally read from it, in a way similar to how one would do when having opened a file with std::ifstream, i.e. occasionally read from it (e.g. with >>) and then do some computations. I was thinking of using a GetObjectRequest and then returning the getBody() of the GetObjectResult. However, 1) I am not sure that reading from that is actually streamed (as in occasionally making new requests to download the file in small chunks which to keep in a buffer) and 2) it doesn't seem possible to return just the stream (e.g. a std::shared_pointerAWS::FStream) and passing around the whole GetObjectResult seems inconvenient. Surely there is a better approach?

[jmklix]

Have you looking into using transfermanger? Here is the C++ docs page and here is an entire sample for how to use it. In the sample it uses a stream that might fit your use case:

            auto downloadHandle = transfer_manager->DownloadFile(BUCKET,
                KEY,
                [&]() { //Define a lambda expression for the callback method parameter to stream back the data.
                    return Aws::New<MyUnderlyingStream>("TestTag", Aws::New<Stream::PreallocatedStreamBuf>("TestTag", buffer.GetUnderlyingData(), BUFFER_SIZE));
                });
            downloadHandle->WaitUntilFinished();// Block calling thread until download is complete.

[indjev99]

I don't want to download the entire file to memory. I want to get something which my program can use as, say a stream, to get data from and it would download whenever it needs more data (possibly asynchronously). E.g. I would get some fileStream and then my program could do stuff like: fileStream >> a; which may or may not trigger a download of a chunk of the file. Ideally, these downloads would be happening asynchronously, anticipating when we need more data.

[jmklix]

@github-actions proposed-answer

I don't think this sdk provides the functionality that you're looking for. It starts downloading the files to memory as soon as you send the request. You could attempt to make your own stream like functionality by using the GetPartNumber when making an s3 request. API docs:

Part number of the object being read. This is a positive integer between 1 and 10,000. Effectively performs a 'ranged' GET request for the part specified. Useful for downloading just a part of an object.