allow setting s3 concurrency, proxy, timeout, uploadPartSize, putSizeLimit, version, and verify_bucket_exists #2271
Conversation
jessebot
force-pushed
the
add-missing-s3-variables
branch
from
c355acb to
040b741
Compare
joshtrichards
added
enhancement
feature: auto config (environment variables)
|
@joshtrichards wanted to gently follow up and ask if anything else is needed here? |
|
@J0WI or @joshtrichards can I ask if there's anything else needed here to get this merged into the default branch? Sorry to be a pest. I just want to make sure this gets merged so we can support it in the helm chart properly. |
jessebot
force-pushed
the
add-missing-s3-variables
branch
from
10d6edd to
2b545ea
Compare
…Limit, version, and verify_bucket_exists Signed-off-by: jessebot <[email protected]>
Signed-off-by: Jesse Hitch <[email protected]>
Signed-off-by: Jesse Hitch <[email protected]>
Signed-off-by: Jesse Hitch <[email protected]>
jessebot
force-pushed
the
add-missing-s3-variables
branch
from
d7baf2f to
1581991
Compare
|
@joshtrichards or @J0WI could you take another look at this when you have a moment? |
| - `OBJECTSTORE_S3_CONCURRENCY` defines the maximum number of concurrent multipart uploads | ||
| - `OBJECTSTORE_S3_PROXY` (default: `false`) | ||
| - `OBJECTSTORE_S3_TIMEOUT` (not set by default) | ||
| - `OBJECTSTORE_S3_UPLOADPARTSIZE` (not set by default) | ||
| - `OBJECTSTORE_S3_PUTSIZELIMIT` (not set by default) | ||
| - `OBJECTSTORE_S3_USEMULTIPARTCOPY` (default: `false`) | ||
| - `OBJECTSTORE_S3_COPYSIZELIMIT` (not set by default) | ||
| - `OBJECTSTORE_S3_VERSION` (default: `latest`) | ||
| - `OBJECTSTORE_S3_VERIFY_BUCKET_EXISTS` (default: `true`) Setting this to `false` after confirming the bucket has been created may provide a performance benefit, but may not be possible in multibucket scenarios. |
There was a problem hiding this comment.
@J0WI how's this? I now only set an int value if it is provided, this way, no one has to find out if empty string breaks stuff. I still set default strings and default booleans though, as we do that everywhere else as well and I want to be consistent. I know that if these new int s3 values are not set, it doesn't break anything, so this should be the best of both worlds right? :)
| if $concurrency { | ||
| $CONFIG['objectstore']['arguments']['concurrency'] = $concurrency; | ||
| } | ||
|
|
||
| if $timeout { | ||
| $CONFIG['objectstore']['arguments']['timeout'] = $timeout; | ||
| } | ||
|
|
||
| if $upload_part_size { | ||
| $CONFIG['objectstore']['arguments']['uploadPartSize'] = $upload_part_size; | ||
| } | ||
|
|
||
| if $put_size_limit { | ||
| $CONFIG['objectstore']['arguments']['putSizeLimit'] = $put_size_limit; | ||
| } | ||
|
|
||
| if $copy_size_limit { | ||
| $CONFIG['objectstore']['arguments']['copySizeLimit'] = $copy_size_limit; | ||
| } |
There was a problem hiding this comment.
these are now all conditionally set by the variables defined at the top of the file that check the env vars.
There was a problem hiding this comment.
I like this approach but all tests are failing
There was a problem hiding this comment.
hmmm, I don't know why... I am terrible at php. I'll try to take a look this weekend
Signed-off-by: jessebot <[email protected]>
jessebot
force-pushed
the
add-missing-s3-variables
branch
from
c29fea3 to
e35ea1e
Compare
Adds missing S3 related environment variables. Fixes #2270
Per the docs here's the missing supported S3 parameters:
concurrencydefaults to 5 [Note: This defines the maximum number of concurrent multipart uploads]proxydefaults tofalsetimeoutdefaults to 15uploadPartSizedefaults to 524288000putSizeLimitdefaults to 104857600versiondefaults to latestuseMultipartCopydefaults totruecopySizeLimitdefaults to 5242880000verify_bucket_existsdefaults totrue[Note: Setting this to false after confirming the bucket has been created may provide a performance benefit, but may not be possible in multibucket scenarios.]This also helps us get ready to merge nextcloud/helm#614 downstream 🙏
Let me know if you need anything else :)