mc mirror fails when replicating a bucket using tiered storage to another bucket on the same host

[MrMeeb]

Expected behavior

Running mc mirror myminio/bucket1 myminio/bucket2 where bucket1 uses tiered storage should create a full replica to bucket2

Actual behavior

Not all data will be replicated between the buckets. In my case, bucket1 has 3493 objects (69.4GiB). where 2372 objects (56.7GiB) have been transitioned to the tiered storage.

When running mc mirror --debug myminio/bucket1 myminio/bucket2, it appears to stall after approximately 2000 objects. During this period, network activity still suggests uploading from the cold tier to the hot tier. Eventually the mc mirror commands finishes with an exit code of 1 with an error to the following effect:

Error

{
 "status": "error",
 "error": {
  "message": "Failed to copy `http://10.200.0.8:9000/gitea-packages/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647`.",
  "cause": {
   "message": "Put \"http://10.200.0.8:9000/gitea-packages-mc-mirror-disable-multipart/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647\": read tcp 172.17.245.99:48308-\u003e10.200.0.8:9000: i/o timeout",
   "error": {
    "Op": "Put",
    "URL": "http://10.200.0.8:9000/gitea-packages-mc-mirror-disable-multipart/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647",
    "Err": {
     "Op": "read",
     "Net": "tcp",
     "Source": {
      "IP": "172.17.245.99",
      "Port": 48308,
      "Zone": ""
     },
     "Addr": {
      "IP": "10.200.0.8",
      "Port": 9000,
      "Zone": ""
     },
     "Err": {}
    }
   }
  },
  "type": "error",
  "trace": [
   {
    "line": 1063,
    "file": "client-s3.go",
    "func": "cmd.(*S3Client).Copy"
   },
   {
    "line": 246,
    "file": "common-methods.go",
    "func": "cmd.copySourceToTargetURL",
    "env": {
     "Tags": [
      "myminio",
      "http://10.200.0.8:9000/gitea-packages-mc-mirror-disable-multipart/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647"
     ]
    }
   },
   {
    "line": 509,
    "file": "common-methods.go",
    "func": "cmd.uploadSourceToTargetURL",
    "env": {
     "Tags": [
      "http://10.200.0.8:9000/gitea-packages/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647"
     ]
    }
   },
   {
    "line": 558,
    "file": "mirror-main.go",
    "func": "cmd.(*mirrorJob).monitorMirrorStatus",
    "env": {
     "Tags": [
      "http://10.200.0.8:9000/gitea-packages/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647"
     ]
    }
   }
  ],
  "sysinfo": {
   "host.arch": "amd64",
   "host.cpus": "16",
   "host.lang": "go1.22.4",
   "host.name": "MINI-PC",
   "host.os": "linux",
   "mem.heap.total": "27 MiB",
   "mem.heap.used": "16 MiB",
   "mem.total": "44 MiB",
   "mem.used": "16 MiB"
  }
 }
}

Steps to reproduce the behavior

1. Create a bucket that uses tiered storage
2. Allow some data to transition to the tiered storage
3. Try and replicate the bucket to another bucket on the same host using mc mirror from a remote client machine
4. Not all data will be replicated. The command will exit complaining of an i/o timeout

mc --version

mc version RELEASE.2024-06-24T19-40-33Z (commit-id=3548007d5bd1ba8e2ed98f35d95c4776dd9a7ff9)
Runtime: go1.22.4 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

System information

My minio deployment consists of two SNSD instances. The primary instance runs on an OCI VPS. Due to limited storage on the VPS, I have a second instance that runs on my LAN. Here I have much more storage but a slower network. This is where I transition things to. The tiers are connected via a Wireguard VPN tunnel that is constrained primarily by my home network connection (1000/120mbps). Due to some overhead, this means the maximum rate at which server1 can pull data from server2 is ~80-100mbps

I originally ran nginx in front of both minio servers, but during my troubleshooting I circumvented it in case it was the cause of this issue. Both servers communicate directly now with each other and clients.

Both minio servers are on RELEASE.2024-06-24T19-40-33Z and run in docker. An example docker compose:

  minio:
    image: minio/minio:RELEASE.2024-06-28T09-06-49Z
    container_name: minio
    restart: unless-stopped
    user: ${PUID}:${PGID}
    depends_on:
      authentik-checker:
        condition: service_completed_successfully
    networks:
      - general
      - nginx
    ports:
      - ${WG_IP}:9000:9000
    command: server --console-address ":9090" /data
    volumes:
      - ${DBULK}/minio:/data
    environment:
      - MINIO_ROOT_USER=rootuser
      - MINIO_ROOT_PASSWORD=${MINIO_ROOT_PW}
      - MINIO_SERVER_URL=https://minio.domain.com
      - MINIO_BROWSER_REDIRECT_URL=https://minio.domain.com/web/ui

I have captured the output of mc mirror --debug --json running various different tests. I'm happy to share if useful, but would need instructions on which details to remove from them, since all headers are of course visible.

Things that work

1. Server-side replication
2. rclone sync myminio/bucket1 myminio/bucket2
3. mc mirror myminio/bucket1 myminio/bucket2 when all objects are on the hot tier
4. mc mirror myminio/bucket1 ./localfolder
5. mc cp myminio/bucket1 ./localfolder

Things I have tried that had no effect

1. Removing nginx
2. Enabling traffic shaping on my LAN to ensure the limited upload bandwidth is not overwhelmed (without this, the network would be overwhelmed and significant packet loss would occur, resetting the connection. I thought this was the root cause initially)
3. Using --disable-multipart
4. Running mc mirror directly in the minio container (docker compose exec minio mc mirror...) - this hung after about 1100 objects, but otherwise behaved as described above

mc in all instances is being run from in Debian WSL on my LAN. My tests initially started on the Windows client, but I moved to Linux to check it wasn't limited to just the Windows client.

While I first thought this was an issue with my minio deployment, now I don't believe it is