Deploy -> Kubernetes -> APIManager ( deployment job failing in helm installation task )

[srikanth-6520]

Hello @gandham-santhosh , we have tried to deploy "APIManager" job with tag "release-5.1.0",

Jenkins Logs:

Started by user [Rajasimman S]
Rebuilds build #[7]
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager@script/74766eba9651ee97692d587da848d14cae2c225f81df644eb5a529f729d5e85a to read kubernetes/pipelines/deploy_core/Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager@script/74766eba9651ee97692d587da848d14cae2c225f81df644eb5a529f729d5e85a/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0^{commit} # timeout=10
Checking out Revision bee2ee6f39baa087eba22646d7919493552b10e1 (origin/release-5.1.0)
git config core.sparsecheckout # timeout=10
git checkout -f bee2ee6f39baa087eba22646d7919493552b10e1 # timeout=10
Commit message: "Merge pull request #3793 from project-sunbird/gandham-santhosh-patch-1"
First time build. Skipping changelog.
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager@libs/b4ac4d1235c67cfedda9d7bf40936b35bc6929460ef19107abe1b183748d4145/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh

• git clean -fxd
  Removing ansible/inventory/env/hosts
  Removing ansible/inventory/env/keys/
  Removing ansible/inventory/env/secrets.yml
  Removing kubernetes/ansible/roles/sunbird-deploy/templates/
  Removing kubernetes/helm_charts/core/apimanager/values.yaml
  Removing metadata.json
  Skipping repository private/
  Removing private@tmp/
  [Pipeline] checkout
  The recommended git tool is: NONE
  No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0^{commit} # timeout=10
Checking out Revision bee2ee6f39baa087eba22646d7919493552b10e1 (origin/release-5.1.0)
git config core.sparsecheckout # timeout=10
git checkout -f bee2ee6f39baa087eba22646d7919493552b10e1 # timeout=10
Commit message: "Merge pull request #3793 from project-sunbird/gandham-santhosh-patch-1"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/alex/dev/Kubernetes/APIManager
  [Pipeline] sh
• echo Deploy/alex/dev/Kubernetes/APIManager
  [Pipeline] sh
• echo Deploy/alex/dev/Kubernetes/APIManager
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] echo
  Found environment variable named hub_org with value as: 961571523562.dkr.ecr.ap-south-1.amazonaws.com/prod
  [Pipeline] copyArtifacts
  Copied 1 artifact from "ArtifactUpload » dev » Core » APIManager" build number 3
  [Pipeline] sh
• jq -r .image_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  image_tag not specified, using the image_tag specified in metadata.json.
  [Pipeline] sh
• jq -r .image_tag metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] sh

• echo Deploy/alex/dev/Kubernetes/APIManager
  [Pipeline] sh
• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=Kubernetes, jobName=APIManager, absolute_job_path=ArtifactUpload/dev/Core/APIManager, agent=built-in, image_name=kong, image_tag=0.14.1, currentWs=/var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/kubernetes/ansible/deploy_core_service.yml, ansibleExtraArgs=--extra-vars "chart_path=/var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/kubernetes/helm_charts/core/apimanager release_name=apimanager image_tag=0.14.1 image_name=kong role_name=sunbird-deploy" --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential bitbucket-cred

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/private/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url bitbucket.org:verlab/veranda-alex-sb-private.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from bitbucket.org:verlab/veranda-alex-sb-private.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_SSH to set credentials Veranda Bitbucket Credential
Verifying host key using known hosts file
git fetch --no-tags --progress --depth=1 -- bitbucket.org:verlab/veranda-alex-sb-private.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0-alex-dev^{commit} # timeout=10
Checking out Revision e3a1a39925c663954fd8df09eab0b75e5c7f2fb3 (origin/release-5.1.0-alex-dev)
git config core.sparsecheckout # timeout=10
git checkout -f e3a1a39925c663954fd8df09eab0b75e5c7f2fb3 # timeout=10
Commit message: "sunbird_private_storage_account_name & account_key updated"
git rev-list --no-walk e3a1a39925c663954fd8df09eab0b75e5c7f2fb3 # timeout=10
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/private/ansible/inventory/dev/Kubernetes/common.yml /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/private/ansible/inventory/dev/Kubernetes/hosts /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/private/ansible/inventory/dev/Kubernetes/keys /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/private/ansible/inventory/dev/Kubernetes/secrets.yml /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/kubernetes/ansible/deploy_core_service.yml --extra-vars chart_path=/var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/kubernetes/helm_charts/core/apimanager release_name=apimanager image_tag=0.14.1 image_name=kong role_name=sunbird-deploy --vault-password-file /var/lib/jenkins/secrets/vault-pass
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/alex/dev/Kubern
etes/APIManager/ansible/inventory/env/hosts with yaml plugin: Syntax Error
while loading YAML. found unexpected ':' The error appears to be in '/var/li
b/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/ansible/inventory/env
/hosts': line 1, column 5, but may be elsewhere in the file depending on the
exact syntax problem. The offending line appears to be: [all:vars] ^
here

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/alex/dev/Kubern
etes/APIManager/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/w
orkspace/Deploy/alex/dev/Kubernetes/APIManager/ansible/inventory/env/hosts:150:
Section [core:children] includes undefined group: postgresql-slave

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernete
s/APIManager/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
alex/dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(sunbird_time_zone). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
alex/dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(nginx_per_ip_connection_limit). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
alex/dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(sunbird_portal_player_cdn_enabled). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
alex/dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(dial_liveness_readiness). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
alex/dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 924, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
alex/dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 934, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
alex/dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 947, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
alex/dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 957, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

PLAY [local] *******************************************************************

TASK [check for kong db migration] *********************************************

TASK [kong-migration : check if tables are existing in kong postgres database] ***
[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host localhost should use
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the
discovered platform python for this host. See https://docs.ansible.com/ansible/
2.8/reference_appendices/interpreter_discovery.html for more information. This
feature will be removed in version 2.12. Deprecation warnings can be disabled
by setting deprecation_warnings=False in ansible.cfg.
changed: [localhost]

TASK [kong-migration : run kong migrations for first run if tables are empty] ***
skipping: [localhost]

TASK [kong-migration : fail when migration failed] *****************************
skipping: [localhost]

TASK [sunbird-deploy : template values.yaml file] ******************************
changed: [localhost]

TASK [sunbird-deploy : copy and template env file from swarm role] *************
changed: [localhost] => (item=../../../../ansible/roles/stack-api-manager/templates/api-manager.env)

TASK [sunbird-deploy : create env configmap] ***********************************
changed: [localhost]

TASK [sunbird-deploy : create aplication configmap] ****************************
changed: [localhost]

TASK [sunbird-deploy : create xml configmap] ***********************************
changed: [localhost]

TASK [Check opa testcases coverage is 100%] ************************************
skipping: [localhost]

TASK [Load role to decrypt private keys, copy public keys to helm chart] *******
skipping: [localhost] => (item=[{'key_to_register': 'access', 'values_to_pass': {'basepath': '/keys/', 'keyprefix': 'accessv1_key', 'keystart': 1, 'keycount': '10'}}, 'decrypt.yml'])
skipping: [localhost] => (item=[{'key_to_register': 'access', 'values_to_pass': {'basepath': '/keys/', 'keyprefix': 'accessv1_key', 'keystart': 1, 'keycount': '10'}}, 'generate-keys.yml'])
skipping: [localhost] => (item=[{'key_to_register': 'access', 'values_to_pass': {'basepath': '/keys/', 'keyprefix': 'accessv1_key', 'keystart': 1, 'keycount': '10'}}, 'copy-to-helm.yml'])

TASK [sunbird-deploy : Create the token public key file] ***********************
skipping: [localhost]

TASK [sunbird-deploy : Replace ansible variables with values] ******************
skipping: [localhost] => (item=main.rego)
skipping: [localhost] => (item=common.rego)

TASK [sunbird-deploy : Create OPA bundle] **************************************
skipping: [localhost]

TASK [sunbird-deploy : Copy the OPA bundle to helm chart directory] ************
skipping: [localhost]

TASK [sunbird-deploy : helm install] *******************************************
fatal: [localhost]: FAILED! => {
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m apimanager /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/kubernetes/helm_charts/core/apimanager -n alexdev",
"delta": "0:10:00.698669",
"end": "2023-05-22 20:29:31.520928",
"rc": 1,
"start": "2023-05-22 20:19:30.822259"
}

STDOUT:

Release "apimanager" does not exist. Installing it now.

STDERR:

Error: release apimanager failed, and has been uninstalled due to atomic being set: timed out waiting for the condition

MSG:

non-zero return code

PLAY RECAP *********************************************************************
localhost : ok=6 changed=6 unreachable=0 failed=1 skipped=8 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• git name-rev --name-only HEAD
• rev
• cut -d / -f1
• rev
  [Pipeline] sh
• echo Deploy/alex/dev/Kubernetes/APIManager
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/alex/dev/Kubernetes/APIManager
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[santhosh-tg]

@srikanth-6520 Please check the pod logs to see why the deployment is failing. The deployment rollbacks after 10 minutes. So, you need to check before that
cmd": "helm upgrade --install --atomic --timeout 10m apimanager /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/APIManager/kubernetes/helm_charts/core/apimanager -n alexdev",

[santhosh-tg]

@srikanth-6520 Is the issue still open ?

[srikanth-6520]

@santhosh-tg issue fixed , thank you.