Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]Failed to vscale the etcd cluster #8833

Closed
tianyue86 opened this issue Jan 20, 2025 · 2 comments
Closed

[BUG]Failed to vscale the etcd cluster #8833

tianyue86 opened this issue Jan 20, 2025 · 2 comments
Assignees
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone

Comments

@tianyue86
Copy link

Describe the env
Kubernetes: v1.31.1-aliyun.1
KubeBlocks: 1.0.0-beta.23
kbcli: 1.0.0-beta.9

To Reproduce
Steps to reproduce the behavior:

  1. Create etcd cluster with yaml below
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: etcd-xrzbpj
  namespace: default
spec:
  terminationPolicy: WipeOut
  componentSpecs:
    - name: etcd
      componentDef: etcd-3-1.0.0-alpha.0
      tls: false
      replicas: 3
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  1. check cluster status: correct
kbcli cluster list-instances etcd-xrzbpj --namespace default
NAME                 NAMESPACE   CLUSTER       COMPONENT   STATUS    ROLE       ACCESSMODE   AZ                 CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                                   CREATED-TIME                 
etcd-xrzbpj-etcd-0   default     etcd-xrzbpj   etcd        Running   leader     ReadWrite    cn-zhangjiakou-c   100m / 100m          512Mi / 512Mi           data:20Gi   cn-zhangjiakou.10.0.0.145/10.0.0.145   Jan 20,2025 17:11 UTC+0800   
etcd-xrzbpj-etcd-1   default     etcd-xrzbpj   etcd        Running   follower   Readonly     cn-zhangjiakou-c   100m / 100m          512Mi / 512Mi           data:20Gi   cn-zhangjiakou.10.0.0.128/10.0.0.128   Jan 20,2025 17:11 UTC+0800   
etcd-xrzbpj-etcd-2   default     etcd-xrzbpj   etcd        Running   follower   Readonly     cn-zhangjiakou-c   100m / 100m          512Mi / 512Mi           data:20Gi   cn-zhangjiakou.10.0.0.144/10.0.0.144   Jan 20,2025 17:11 UTC+0800 
  1. vscale the cluster
    kbcli cluster vscalekbcli cluster vscale etcd-xrzbpj --auto-approve --force=true --components etcd --cpu 200m --memory 0.6Gi --namespace default
    OpsRequest etcd-xrzbpj-verticalscaling-lbcc9 created successfully, you can view the progress:
    kbcli cluster describe-ops etcd-xrzbpj-verticalscaling-lbcc9 -n default

  2. The ops is always in Running status

kbcli cluster list-ops etcd-xrzbpj --status all  --namespace default
NAME                                NAMESPACE   TYPE              CLUSTER       COMPONENT   STATUS    PROGRESS   CREATED-TIME                 
etcd-xrzbpj-verticalscaling-lbcc9   default     VerticalScaling   etcd-xrzbpj   etcd        Running   0/3        Jan 20,2025 17:15 UTC+0800


kbcli cluster describe-ops etcd-xrzbpj-verticalscaling-lbcc9 -n default
Spec:
  Name: etcd-xrzbpj-verticalscaling-lbcc9	NameSpace: default	Cluster: etcd-xrzbpj	Type: VerticalScaling	

Command:
  kbcli cluster vscale etcd-xrzbpj --cpu=200m --memory=644245094400m --components=etcd --namespace=default

Last Configuration:
COMPONENT   REQUEST-CPU   REQUEST-MEMORY   LIMIT-CPU   LIMIT-MEMORY   
etcd        100m          512Mi            100m        512Mi          

Status:
  Start Time:         Jan 20,2025 17:15 UTC+0800
  Duration:           9m55s
  Status:             Running
  Progress:           0/3
                      OBJECT-KEY               STATUS    DURATION    MESSAGE   
                      Pod/etcd-xrzbpj-etcd-0   Pending   <Unknown>             
                      Pod/etcd-xrzbpj-etcd-1   Pending   <Unknown>             
                      Pod/etcd-xrzbpj-etcd-2   Pending   <Unknown>             

Conditions:
LAST-TRANSITION-TIME         TYPE                 REASON                     STATUS   MESSAGE                                                                                                        
Jan 20,2025 17:15 UTC+0800   WaitForProgressing   WaitForProgressing         True     wait for the controller to process the OpsRequest: etcd-xrzbpj-verticalscaling-lbcc9 in Cluster: etcd-xrzbpj   
Jan 20,2025 17:15 UTC+0800   Validated            ValidateOpsRequestPassed   True     OpsRequest: etcd-xrzbpj-verticalscaling-lbcc9 is validated                                                     
Jan 20,2025 17:15 UTC+0800   VerticalScaling      VerticalScalingStarted     True     Start to vertical scale resources in Cluster: etcd-xrzbpj                                                      

Warning Events: <none>

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

@tianyue86 tianyue86 added the kind/bug Something isn't working label Jan 20, 2025
@tianyue86 tianyue86 added this to the Release 1.0.0 milestone Jan 20, 2025
@tianyue86
Copy link
Author

mongodb has the same problem.

kbcli cluster describe-ops mongo-lkyzoz-verticalscaling-d4mkp -n default
Spec:
  Name: mongo-lkyzoz-verticalscaling-d4mkp	NameSpace: default	Cluster: mongo-lkyzoz	Type: VerticalScaling	

Command:
  kbcli cluster vscale mongo-lkyzoz --cpu=200m --memory=644245094400m --components=mongodb --namespace=default

Last Configuration:
COMPONENT   REQUEST-CPU   REQUEST-MEMORY   LIMIT-CPU   LIMIT-MEMORY   
mongodb     100m          512Mi            100m        512Mi          

Status:
  Start Time:         Jan 21,2025 17:09 UTC+0800
  Duration:           20m
  Status:             Running
  Progress:           0/3
                      OBJECT-KEY                   STATUS    DURATION    MESSAGE   
                      Pod/mongo-lkyzoz-mongodb-1   Pending   <Unknown>             
                      Pod/mongo-lkyzoz-mongodb-2   Pending   <Unknown>             
                      Pod/mongo-lkyzoz-mongodb-0   Pending   <Unknown>             

Conditions:
LAST-TRANSITION-TIME         TYPE                 REASON                     STATUS   MESSAGE                                                                                                          
Jan 21,2025 17:09 UTC+0800   WaitForProgressing   WaitForProgressing         True     wait for the controller to process the OpsRequest: mongo-lkyzoz-verticalscaling-d4mkp in Cluster: mongo-lkyzoz   
Jan 21,2025 17:09 UTC+0800   Validated            ValidateOpsRequestPassed   True     OpsRequest: mongo-lkyzoz-verticalscaling-d4mkp is validated                                                      
Jan 21,2025 17:09 UTC+0800   VerticalScaling      VerticalScalingStarted     True     Start to vertical scale resources in Cluster: mongo-lkyzoz                                                       

Warning Events: <none>

@tianyue86 tianyue86 added the severity/major Great chance user will encounter the same problem label Jan 21, 2025
@wangyelei
Copy link
Contributor

fixed at #8845

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Development

No branches or pull requests

3 participants