Skip to content

Latest commit

 

History

History
101 lines (74 loc) · 4.1 KB

awss3.md

File metadata and controls

101 lines (74 loc) · 4.1 KB

AWS S3

In order to use S3 you need to reference NuGet package first. The provider wraps around the standard AWS SDK which is updated regularly, but adds a lot of untrivial workarounds that makes your life painless.

Connecting

Unfortunately, AWS is not as good in permission management as Azure Active Directory, therefore you have the following options:

There are a few overloads in this package, for instance:

IBlobStorage storage = StorageFactory.Blobs.AmazonS3BlobStorage(string accessKeyId,
   string secretAccessKey,
   string sessionToken,
   string bucketName,
   RegionEndpoint regionEndpoint = null);

Please see StorageFactory.Blobs factory entry for more options.

To create with a connection string, first reference the module:

StorageFactory.Modules.UseAwsStorage();

Then construct using the following format:

IBlobStorage storage = StorageFactory.Blobs.FromConnectionString("aws.s3://keyId=...;key=...;bucket=...;region=...");

where:

  • keyId is (optional) access key ID.
  • key is (optional) secret access key.
  • bucket is bucket name.
  • region is an optional value and defaults to EU West 1 if not specified. At the moment of this wring the following regions are supported. However as we are using the official AWS SDK, when region information changes, storage.net gets automatically updated.
    • us-east-1
    • us-east-2
    • us-west-1
    • us-west-2
    • eu-north-1
    • eu-west-1
    • eu-west-2
    • eu-west-3
    • eu-central-1
    • ap-northeast-1
    • ap-northeast-2
    • ap-northeast-3
    • ap-south-1
    • ap-southeast-1
    • ap-southeast-2
    • sa-east-1
    • us-gov-east-1
    • us-gov-west-1
    • cn-north-1
    • cn-northwest-1
    • ca-central-1

If keyId and key are omitted, the AWS SDK's default approach to credential resolution will be used. For example: if running in Lambda, it will assume the Lambda execution role; if there are credentials configured in ~/.aws, it will use those; etc. See AWS SDK Documentation for more details.

AWS CLI Profile Support

If you already have credentials in the local ~/.aws/credentials generated by AWS CLI, you can also use them to connect to a bucket with Storage.Net. Credentials file example:

[default]
aws_access_key_id = ASIAV44FYRYLSBYIPG67
aws_secret_access_key = zhuh8nSUAR0pR5jbi4vhUym95JsC7ay3iiE2v9Jq
aws_session_token = FQoGZXIvY...

[anotherProfile]
aws_access_key_id = ASIAXEKXYP2H7MRN45FD
aws_secret_access_key = 8i6jnOOurRtTiK4CWN+uUD115SYS8++ZuQjO/LtW
aws_session_token = FQoGZXI...

...

To connect using say anotherProfile credentials, construct the instance by:

IBlobStorage bs = StorageFactory.Blobs.AwsS3(
   "anotherProfile",
   "bucketName",
   "region");

AwsCliCredentials class contains credentials file parrsing logic and some utility methods you mind find useful, for instance enumerating available profiles and so on.

Performance & Memory Consumption

By default, S3 SDK is extremely ineffective and I have no idea why. Can it be that .NET developers are not paying enough attention to good quality code or Amazon is just not good in programming? In particular, SDK doesn't deal very well with streaming. When you stream with S3 SDK, it accumulates data in memory and only uploads it on flush!. Whoever designed it this way has probably never built real software. Unfortunately, S3 has a limitation that data length needs to be known beforehand in order to create a file. See issues here and here for more information.

If you ever wonder why your application is slow, say thanks to Amazon engineers.

Native Operations

Native operations are exposed via IAwsS3BlobStorageNativeOperations interface.