mc admin replicate status gives different results for users

[Lisiadito]

Expected behavior

That the status is the same for both sites.

Actual behavior

This is what I get when I query the status for the site from which I initiated the site replication:

./mc admin replicate status stage_minio
Bucket replication status:
●  4/4 Buckets in sync

Policy replication status:
●  7/7 Policies in sync

User replication status:
●  0/2 Users in sync

User            | NEWPROD         | STAGE
BHZKD8XTDCTO... | ✗  in-sync      | ✗  in-sync
orYyTaBZq9Pa... | ✗  in-sync      | ✗  in-sync

And this is the result from the site which I try to replicate to:

./mc admin replicate status prod_minio
Bucket replication status:
●  4/4 Buckets in sync

Policy replication status:
●  7/7 Policies in sync

User replication status:
●  2/2 Users in sync

Group replication status:
No Groups present

First of all the result for the users is different and an additional question would be why do I get the result that the buckets are in sync even though not all objects are replicated into the new site?

Steps to reproduce the behavior

• Create a cluster, add a few buckets, objects, policies and users
• Create a second empty cluster
• Initialize the site replication via GUI (minio console)
• Query the status via mc

mc --version

• mc version RELEASE.2023-01-11T03-14-16Z (commit-id=14c2e506fa78b53fb6db88bcf87d8f6d3fb6989e)
  Runtime: go1.19.4 linux/amd64
  Copyright (c) 2015-2023 MinIO, Inc.
  License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

System information

• Runs in a docker swarm on CentOS machines

[harshavardhana]

what is the server version here?

[Lisiadito]

@harshavardhana server version is RELEASE.2022-11-17T23-20-09Z

[harshavardhana]

@harshavardhana server version is RELEASE.2022-11-17T23-20-09Z

Please upgrade to latest @Lisiadito

[Lisiadito]

even on the latest server version I get the same output

[poornas]

would be why do I get the result that the buckets are in sync even though not all objects are replicated into the new site?
mc admin replicate status reports on whether bucket metadata is in sync - not the status of objects within the bucket. You should look at mc replicate status alias/bucket for that info

[poornas]

Can you share the output of ./mc admin replicate status stage_minio --json - are the users that didn't sync service accounts? site replication should heal missing users automatically every 10s. Please double check if it has healed in your stage instance after you last checked

[Lisiadito]

@poornas
here the output:

./mc admin replicate status stage_minio --json
{
 "Enabled": true,
 "MaxBuckets": 4,
 "MaxUsers": 2,
 "MaxGroups": 0,
 "MaxPolicies": 7,
 "Sites": {
  "56784047-4bb6-4edb-bd30-82c419ff7036": {
   "endpoint": "http://our.domain.com",
   "name": "newprod",
   "deploymentID": "56784047-4bb6-4edb-bd30-82c419ff7036"
  },
  "f5b65608-0d03-46f8-88c7-67b69ee36d0d": {
   "endpoint": "http://127.0.0.1:9000",
   "name": "stage",
   "deploymentID": "f5b65608-0d03-46f8-88c7-67b69ee36d0d"
  }
 },
 "StatsSummary": {
  "56784047-4bb6-4edb-bd30-82c419ff7036": {
   "ReplicatedBuckets": 4,
   "ReplicatedTags": 0,
   "ReplicatedBucketPolicies": 4,
   "ReplicatedIAMPolicies": 7,
   "ReplicatedUsers": 2,
   "ReplicatedGroups": 0,
   "ReplicatedLockConfig": 0,
   "ReplicatedSSEConfig": 0,
   "ReplicatedVersioningConfig": 4,
   "ReplicatedQuotaConfig": 0,
   "ReplicatedUserPolicyMappings": 2,
   "ReplicatedGroupPolicyMappings": 0,
   "TotalBucketsCount": 4,
   "TotalTagsCount": 0,
   "TotalBucketPoliciesCount": 4,
   "TotalIAMPoliciesCount": 7,
   "TotalLockConfigCount": 0,
   "TotalSSEConfigCount": 0,
   "TotalVersioningConfigCount": 4,
   "TotalQuotaConfigCount": 0,
   "TotalUsersCount": 2,
   "TotalGroupsCount": 0,
   "TotalUserPolicyMappingCount": 2,
   "TotalGroupPolicyMappingCount": 0
  },
  "f5b65608-0d03-46f8-88c7-67b69ee36d0d": {
   "ReplicatedBuckets": 4,
   "ReplicatedTags": 0,
   "ReplicatedBucketPolicies": 4,
   "ReplicatedIAMPolicies": 7,
   "ReplicatedUsers": 2,
   "ReplicatedGroups": 0,
   "ReplicatedLockConfig": 0,
   "ReplicatedSSEConfig": 0,
   "ReplicatedVersioningConfig": 4,
   "ReplicatedQuotaConfig": 0,
   "ReplicatedUserPolicyMappings": 2,
   "ReplicatedGroupPolicyMappings": 0,
   "TotalBucketsCount": 4,
   "TotalTagsCount": 0,
   "TotalBucketPoliciesCount": 4,
   "TotalIAMPoliciesCount": 7,
   "TotalLockConfigCount": 0,
   "TotalSSEConfigCount": 0,
   "TotalVersioningConfigCount": 4,
   "TotalQuotaConfigCount": 0,
   "TotalUsersCount": 2,
   "TotalGroupsCount": 0,
   "TotalUserPolicyMappingCount": 2,
   "TotalGroupPolicyMappingCount": 0
  }
 },
 "BucketStats": {},
 "PolicyStats": {},
 "UserStats": {
  "BHZKD8XTDCTORZO9DNZO": {
   "56784047-4bb6-4edb-bd30-82c419ff7036": {
    "DeploymentID": "",
    "PolicyMismatch": false,
    "UserInfoMismatch": true,
    "HasUser": true,
    "HasPolicyMapping": true
   },
   "f5b65608-0d03-46f8-88c7-67b69ee36d0d": {
    "DeploymentID": "",
    "PolicyMismatch": false,
    "UserInfoMismatch": true,
    "HasUser": true,
    "HasPolicyMapping": true
   }
  },
  "orYyTaBZq9PaNRx47ksc": {
   "56784047-4bb6-4edb-bd30-82c419ff7036": {
    "DeploymentID": "",
    "PolicyMismatch": false,
    "UserInfoMismatch": true,
    "HasUser": true,
    "HasPolicyMapping": true
   },
   "f5b65608-0d03-46f8-88c7-67b69ee36d0d": {
    "DeploymentID": "",
    "PolicyMismatch": false,
    "UserInfoMismatch": true,
    "HasUser": true,
    "HasPolicyMapping": true
   }
  }
 },
 "GroupStats": {}
}

The users do sync just the output of mc suggests that they are not in sync thats whats confuse me.

[poornas]

These seem to be service accounts - can you do mc admin user svcacct info stage_minio BHZKD8XTDCTORZO9DNZO --json and mc admin user svcacct info prod_minio BHZKD8XTDCTORZO9DNZO --json - and paste the output? If this is not a service acct, you can drop the svcacct in above command.

Can you share server logs - it should report if there was a failure in healing these users. There seems to be some mismatch in policy/status/ secret key that could not be healed. I will try to repro this locally, but any info you share will be helpful. Thanks

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.