Uploading to object locked buckets is slower than normal

[Zibbp]

Using the CLI to upload a file to an bucket with object locking/versioning enabled is slower than a bucket with these settings disabled. I'm only able to reproduce this with the CLI, using a SDK does not have this issue.

Expected behavior

Copying a file with mc cp to an object locked/versioned bucket should be just as fast as a bucket without these settings enabled.

Actual behavior

Speeds are ~3x slower copying a file to a object locked/versioned bucket.

Steps to reproduce the behavior

1. Create two buckets - one versioned and object locked, one not.

2. 

3. Generate a large file dd if=/dev/zero of=./random.img bs=4k iflag=fullblock,count_bytes count=10G

4. Copy files with CLI

 # mc cp random.img test/test1/cli
/root/random.img: 10.00 GiB / 10.00 GiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 167.09 MiB/s 1m1s

 # mc cp random.img test/test2/cli
/root/random.img: 10.00 GiB / 10.00 GiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 455.34 MiB/s 22s

As you can see the upload speed to test1 is signficantly slower than the bucket without object locking/versioning enabled (test2).

As I noted above, I cannot reproduce this using the Go SDK. The upload speed and time are identical for the two buckets.

 # /tmp/main -bucket test1 -file /root/random.img
2024/11/11 10:32:47 Uploading /root/random.img to test1
2024/11/11 10:32:47 Object lock status for test1: Enabled
2024/11/11 10:33:09 Successfully uploaded testdata of size 10737418240 bytes
2024/11/11 10:33:09 Time taken: 21.439045457s
2024/11/11 10:33:09 Upload speed: 477.63 MB/s

 # /tmp/main -bucket test2 -file /root/random.img
2024/11/11 10:33:46 Uploading /root/random.img to test2
2024/11/11 10:33:46 Object lock status for test2:
2024/11/11 10:34:09 Successfully uploaded testdata of size 10737418240 bytes
2024/11/11 10:34:09 Time taken: 22.360927093s
2024/11/11 10:34:09 Upload speed: 457.94 MB/s

Go code:

package main

import (
	"context"
	"flag"
	"log"
	"time"

	"github.com/minio/minio-go/v7"
	"github.com/minio/minio-go/v7/pkg/credentials"
)

func main() {
	ctx := context.Background()

	bucketPtr := flag.String("bucket", "", "bucket")
	filePtr := flag.String("file", "", "file")

	flag.Parse()

	endpoint := "minio.domain.com"
	accessKeyID := "admin"
	secretAccessKey := "secret"
	useSSL := true

	minioClient, err := minio.New(endpoint, &minio.Options{
		Creds:  credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
		Secure: useSSL,
	})
	if err != nil {
		log.Fatalln(err)
	}

	log.Printf("Uploading %s to %s\n", *filePtr, *bucketPtr)

	objectName := "testdata"
	filePath := *filePtr
	contentType := "application/octet-stream"

	objectLockEnabled, _, _, _, _ := minioClient.GetObjectLockConfig(ctx, *bucketPtr)

	log.Printf("Object lock status for %s: %v\n", *bucketPtr, objectLockEnabled)

	startTime := time.Now()

	info, err := minioClient.FPutObject(ctx, *bucketPtr, objectName, filePath, minio.PutObjectOptions{ContentType: contentType})
	if err != nil {
		log.Fatalln(err)
	}

	elapsedTime := time.Since(startTime)
	uploadSpeedMBps := (float64(info.Size) / elapsedTime.Seconds()) / (1024 * 1024)

	log.Printf("Successfully uploaded %s of size %d bytes\n", objectName, info.Size)
	log.Printf("Time taken: %v\n", elapsedTime)
	log.Printf("Upload speed: %.2f MB/s\n", uploadSpeedMBps)
}

mc --version

 # mc --version
mc version RELEASE.2024-11-05T11-29-45Z (commit-id=6ac18619cf881074fe6edcc79ab62c9c85da60b9)
Runtime: go1.23.3 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

System information

Machine running the client is a VM with 4 cores and 8GB of memory.

[harshavardhana]

Such marginal changes are expected as the object-locked bucket uses slightly more IOPs; avoid micro benchmarking - real-world aberrations of 1sec differences will be non-existent.

As in this is a silly way to benchmark.

[Zibbp]

Such marginal changes are expected as the object-locked bucket uses slightly more IOPs; avoid micro benchmarking - real-world aberrations of 1sec differences will be non-existent.

As in this is a silly way to benchmark.

The CLI has way more than a 1 second difference, referencing my original post:

 # mc cp random.img test/test1/cli
/root/random.img: 10.00 GiB / 10.00 GiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 167.09 MiB/s 1m1s

 # mc cp random.img test/test2/cli
/root/random.img: 10.00 GiB / 10.00 GiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 455.34 MiB/s 22s

There is a 39 second difference between uploading to the buckets.

As I noted above the Go SDK example is demonstrating this is an issue with the CLI not benchmarking the Go code.