Multipart upload fails

[albertjannap]

Description

I am trying to stream a 111MB xlsx file, using putObject, to MinIO and I am using SSE-C encryption

Expected Behavior

I expect that the file will upload without any memory issues or in this case errors about wrong configuration about encryption.

Current Behavior

I receive an error: S3Error: The multipart upload initiate requested encryption. Subsequent part requests must include the appropriate encryption parameters.. For not multipart requests the upload is successful

Possible Solution

In uploadStream(src/internal/client.ts) a request is made to upload a part, I think the configuration for this request is wrong, the headers you configure when you call putObject are not being passed for this request. Maybe it could be a possible solution.

await this.minioClient.putObject(id, name, file.buffer, file.size, {
  'X-Amz-Server-Side-Encryption-Customer-Algorithm': encryptionConfig.SSECustomerAlgorithm,
  'X-Amz-Server-Side-Encryption-Customer-Key': encryptionConfig.SSECustomerKey,
  'X-Amz-Server-Side-Encryption-Customer-Key-MD5': encryptionConfig.SSECustomerKeyMD5,
  'X-Amz-Copy-Source-Server-Side-Encryption-Customer-Algorithm': encryptionConfig.SSECustomerAlgorithm,
  'X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key': encryptionConfig.SSECustomerKey,
  'X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key-Md5': encryptionConfig.SSECustomerKeyMD5,
});

const options: RequestOption = {
  method: 'PUT',
  query: qs.stringify({ partNumber, uploadId }),
  headers: {
    'Content-Length': chunk.length,
    'Content-MD5': md5.toString('base64'),
  },
  bucketName,
  objectName,
}

const response = await this.makeRequestAsyncOmit(options, chunk)

Steps to Reproduce (for bugs)

1. Use putObject to upload a file(buffer) larger then 100 MB
2. Add SSE-C encryption headers to encrypt a file

Context

I am trying to upload this file to temporary store it in a bucket, in a later moment in time I want to process the xlsx file to import the data to Salesforce. This has impact on my work because I need to store the file with the encryption headers.

Regression

Is this issue a regression? No

Your Environment

• Version used: https://github.com/minio/minio/tree/RELEASE.2024-01-31T20-20-33Z
• Operating System and version: Node + NestJS
• Minio-js: 8.0.1

[prakashsvmx]

@albertjannap

Version used: 2024.1.31 (https://northflank.com/docs/v1/application/databases-and-persistence/deploy-databases-on-northflank/deploy-minio-on-northflank)

This is not a MinIO maintained or supported version.

minio --version is the command to check .

please capture mc admin trace -v -ALIAS for us to review. once you use official distribution of MinIO.

What is the minio-js version used ?

[albertjannap]

@prakashsvmx updated environment info

[albertjannap]

@prakashsvmx I made a local version of minio-js and update the headers for these options by adding ...headers

const options: RequestOption = {
  method: 'PUT',
  query: qs.stringify({ partNumber, uploadId }),
  headers: {
    'Content-Length': chunk.length,
    'Content-MD5': md5.toString('base64'),
    ...headers,
  },
  bucketName,
  objectName,
}

Now I don't get the error when I upload my 111MB file but got a new issue, my minio instance was terminated because OOMKilled😅 any idea how I could avoid that

[prakashsvmx]

please capture mc admin trace -v -ALIAS for us to review.

[albertjannap]

I don't have shell access