Skip to content

Manages streaming of data to AWS S3 without knowing the size beforehand and without keeping it all in memory or writing to disk.

License

Notifications You must be signed in to change notification settings

IlyaGazman/s3-stream-upload

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Upgrade

  • Upgraded to use Gradle

S3 Stream Upload

This library allows you to efficiently stream data to a location on AWS S3 in Java. The S3 API requires that a content length be set before starting uploading, which is a problem when you want to calculate a large amount of data on the fly. The standard Java AWS SDK will simply buffer all the data in memory so that it can calculate the length, which consumes RAM and delays the upload. You can write the data to a temporary file but disk IO is slow. This library provides an OutputStream that packages data written to it into chunks which are sent in a multipart upload. You can also use several streams and upload the data in parallel.

The entrypoint is the class StreamTransferManager. Read more in the javadoc, including a usage example.

This is available from maven central. You can include it as a dependency in pom.xml:

<dependency>
    <groupId>com.github.alexmojaki</groupId>
    <artifactId>s3-stream-upload</artifactId>
    <version>1.0.1</version>
</dependency>

About

Manages streaming of data to AWS S3 without knowing the size beforehand and without keeping it all in memory or writing to disk.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 100.0%