Deploy---> alex---> dev---> managed-learn/---> ml-survey-service

[srikanth-6520]

Hi Team ,
we are getting an error , while deploying ml-survey-service jenkins job, below we have provided jenkins console.

Started by user [Kolluri Srikanth]
Rebuilds build #2
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service@script/74766eba9651ee97692d587da848d14cae2c225f81df644eb5a529f729d5e85a to read kubernetes/pipelines/deploy_core/Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service@script/74766eba9651ee97692d587da848d14cae2c225f81df644eb5a529f729d5e85a/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0^{commit} # timeout=10
Checking out Revision bee2ee6f39baa087eba22646d7919493552b10e1 (origin/release-5.1.0)
git config core.sparsecheckout # timeout=10
git checkout -f bee2ee6f39baa087eba22646d7919493552b10e1 # timeout=10
Commit message: "Merge pull request #3793 from project-sunbird/gandham-santhosh-patch-1"
First time build. Skipping changelog.
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service@libs/b4ac4d1235c67cfedda9d7bf40936b35bc6929460ef19107abe1b183748d4145/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh

• git clean -fxd
  Removing ansible/inventory/env/hosts
  Removing ansible/inventory/env/keys/
  Removing ansible/inventory/env/secrets.yml
  Removing kubernetes/helm_charts/core/ml-survey-service/values.yaml
  Removing metadata.json
  Skipping repository private/
  Removing private@tmp/
  [Pipeline] checkout
  The recommended git tool is: NONE
  No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0^{commit} # timeout=10
Checking out Revision bee2ee6f39baa087eba22646d7919493552b10e1 (origin/release-5.1.0)
git config core.sparsecheckout # timeout=10
git checkout -f bee2ee6f39baa087eba22646d7919493552b10e1 # timeout=10
Commit message: "Merge pull request #3793 from project-sunbird/gandham-santhosh-patch-1"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/alex/dev/managed-learn/ml-survey-service
  [Pipeline] sh
• echo Deploy/alex/dev/managed-learn/ml-survey-service
  [Pipeline] sh
• echo Deploy/alex/dev/managed-learn/ml-survey-service
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] echo
  Found environment variable named hub_org with value as: 961571523562.dkr.ecr.ap-south-1.amazonaws.com/prod
  [Pipeline] copyArtifacts
  Copied 1 artifact from "ArtifactUpload » dev » managed-learn » ml-survey-service" build number 3
  [Pipeline] sh
• jq -r .image_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  image_tag not specified, using the image_tag specified in metadata.json.
  [Pipeline] sh
• jq -r .image_tag metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] sh

• echo Deploy/alex/dev/managed-learn/ml-survey-service
  [Pipeline] sh
• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=managed-learn, jobName=ml-survey-service, absolute_job_path=ArtifactUpload/dev/managed-learn/ml-survey-service, agent=built-in, image_name=ml-survey-service, image_tag=release-5.1.0_RC2_9c7c7d18_1, currentWs=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/kubernetes/ansible/deploy_core_service.yml, ansibleExtraArgs=--extra-vars "chart_path=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/kubernetes/helm_charts/core/ml-survey-service release_name=ml-survey-service image_tag=release-5.1.0_RC2_9c7c7d18_1 image_name=ml-survey-service role_name=sunbird-deploy" --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential bitbucket-cred

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/private/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url bitbucket.org:verlab/veranda-alex-sb-private.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from bitbucket.org:verlab/veranda-alex-sb-private.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_SSH to set credentials Veranda Bitbucket Credential (https://bitbucket.org/verlab/veranda-alex-sb-private)
Verifying host key using known hosts file
git fetch --no-tags --progress --depth=1 -- bitbucket.org:verlab/veranda-alex-sb-private.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0^{commit} # timeout=10
Checking out Revision 181df246f5e15f714a812b34f890d4b7b51a4bb9 (origin/release-5.1.0)
git config core.sparsecheckout # timeout=10
git checkout -f 181df246f5e15f714a812b34f890d4b7b51a4bb9 # timeout=10
Commit message: "we have added "ml_cloud_config" variable"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/private/ansible/inventory/dev/managed-learn/common.yml /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/private/ansible/inventory/dev/managed-learn/hosts /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/private/ansible/inventory/dev/managed-learn/keys /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/private/ansible/inventory/dev/managed-learn/secrets.yml /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/kubernetes/ansible/deploy_core_service.yml --extra-vars chart_path=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/kubernetes/helm_charts/core/ml-survey-service release_name=ml-survey-service image_tag=release-5.1.0_RC2_9c7c7d18_1 image_name=ml-survey-service role_name=sunbird-deploy --vault-password-file /var/lib/jenkins/secrets/vault-pass
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/ansible/inventory/env/hosts with yaml plugin: Syntax Error while
loading YAML. found unexpected ':' The error appears to be in
'/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/ansible/inventory/env/hosts': line 1, column 5, but may be elsewhere in
the file depending on the exact syntax problem. The offending line appears to
be: [all:vars] ^ here

[WARNING]: * Failed to parse
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/ansible/inventory/env/hosts with ini plugin:
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/ansible/inventory/env/hosts:150: Section [core:children] includes
undefined group: postgresql-slave

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/alex/dev/managed-
learn/ml-survey-service/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

[WARNING]: While constructing a mapping from
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/kubernetes/ansible/roles/sunbird-deploy/defaults/main.yml, line 3,
column 1, found a duplicate dict key (sunbird_time_zone). Using last defined
value only.

[WARNING]: While constructing a mapping from
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/kubernetes/ansible/roles/sunbird-deploy/defaults/main.yml, line 3,
column 1, found a duplicate dict key (nginx_per_ip_connection_limit). Using
last defined value only.

[WARNING]: While constructing a mapping from
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/kubernetes/ansible/roles/sunbird-deploy/defaults/main.yml, line 3,
column 1, found a duplicate dict key (sunbird_portal_player_cdn_enabled). Using
last defined value only.

[WARNING]: While constructing a mapping from
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/kubernetes/ansible/roles/sunbird-deploy/defaults/main.yml, line 3,
column 1, found a duplicate dict key (dial_liveness_readiness). Using last
defined value only.

[WARNING]: While constructing a mapping from
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/kubernetes/ansible/roles/sunbird-deploy/defaults/main.yml, line 924,
column 5, found a duplicate dict key (failureThreshold). Using last defined
value only.

[WARNING]: While constructing a mapping from
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/kubernetes/ansible/roles/sunbird-deploy/defaults/main.yml, line 934,
column 5, found a duplicate dict key (failureThreshold). Using last defined
value only.

[WARNING]: While constructing a mapping from
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/kubernetes/ansible/roles/sunbird-deploy/defaults/main.yml, line 947,
column 5, found a duplicate dict key (failureThreshold). Using last defined
value only.

[WARNING]: While constructing a mapping from
/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-
service/kubernetes/ansible/roles/sunbird-deploy/defaults/main.yml, line 957,
column 5, found a duplicate dict key (failureThreshold). Using last defined
value only.

PLAY [local] *******************************************************************

TASK [check for kong db migration] *********************************************
skipping: [localhost]

TASK [sunbird-deploy : template values.yaml file] ******************************
[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host localhost should use
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the
discovered platform python for this host. See https://docs.ansible.com/ansible/
2.8/reference_appendices/interpreter_discovery.html for more information. This
feature will be removed in version 2.12. Deprecation warnings can be disabled
by setting deprecation_warnings=False in ansible.cfg.
changed: [localhost]

TASK [sunbird-deploy : copy and template env file from swarm role] *************
changed: [localhost] => (item=../../../../ansible/roles/stack-sunbird/templates/ml-survey-service.env)

TASK [sunbird-deploy : create env configmap] ***********************************
changed: [localhost]

TASK [sunbird-deploy : create aplication configmap] ****************************
changed: [localhost]

TASK [sunbird-deploy : create xml configmap] ***********************************
changed: [localhost]

TASK [Check opa testcases coverage is 100%] ************************************
skipping: [localhost]

TASK [Load role to decrypt private keys, copy public keys to helm chart] *******

TASK [mount-keys : Decrypt the data first] *************************************
changed: [localhost] => (item=1)
changed: [localhost] => (item=2)
changed: [localhost] => (item=3)
changed: [localhost] => (item=4)
changed: [localhost] => (item=5)
changed: [localhost] => (item=6)
changed: [localhost] => (item=7)
changed: [localhost] => (item=8)
changed: [localhost] => (item=9)
changed: [localhost] => (item=10)

TASK [mount-keys : Generate public keys for private keys] **********************
changed: [localhost] => (item=1)
changed: [localhost] => (item=2)
changed: [localhost] => (item=3)
changed: [localhost] => (item=4)
changed: [localhost] => (item=5)
changed: [localhost] => (item=6)
changed: [localhost] => (item=7)
changed: [localhost] => (item=8)
changed: [localhost] => (item=9)
changed: [localhost] => (item=10)

TASK [mount-keys : Register list of files as a variable] ***********************
ok: [localhost]

TASK [mount-keys : Create a temporary file] ************************************
changed: [localhost]

TASK [mount-keys : Insert the public keys as key:value pairs into the file] ****
changed: [localhost] => (item=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/keys/accessv1_key10)
changed: [localhost] => (item=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/keys/accessv1_key1)
changed: [localhost] => (item=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/keys/accessv1_key8)
changed: [localhost] => (item=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/keys/accessv1_key5)
changed: [localhost] => (item=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/keys/accessv1_key7)
changed: [localhost] => (item=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/keys/accessv1_key9)
changed: [localhost] => (item=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/keys/accessv1_key3)
changed: [localhost] => (item=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/keys/accessv1_key4)
changed: [localhost] => (item=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/keys/accessv1_key6)
changed: [localhost] => (item=/var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/ansible/inventory/env/keys/accessv1_key2)

TASK [mount-keys : Set a variable with the file contents] **********************
ok: [localhost]

TASK [mount-keys : Register a variable for keycloak public key in PEM format] ***
changed: [localhost]

TASK [mount-keys : Remove the temporary file] **********************************
changed: [localhost]

TASK [mount-keys : Create the keys directory in case user is overriding the private_key_path] ***
changed: [localhost]

TASK [mount-keys : Copy keys directory to chart path] **************************
changed: [localhost] => (item=1)
changed: [localhost] => (item=2)
changed: [localhost] => (item=3)
changed: [localhost] => (item=4)
changed: [localhost] => (item=5)
changed: [localhost] => (item=6)
changed: [localhost] => (item=7)
changed: [localhost] => (item=8)
changed: [localhost] => (item=9)
changed: [localhost] => (item=10)

TASK [sunbird-deploy : Create the token public key file] ***********************
changed: [localhost]

TASK [sunbird-deploy : Replace ansible variables with values] ******************
skipping: [localhost] => (item=main.rego)
skipping: [localhost] => (item=common.rego)

TASK [sunbird-deploy : Create OPA bundle] **************************************
skipping: [localhost]

TASK [sunbird-deploy : Copy the OPA bundle to helm chart directory] ************
skipping: [localhost]

TASK [sunbird-deploy : helm install] *******************************************
fatal: [localhost]: FAILED! => {
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m ml-survey-service /var/lib/jenkins/workspace/Deploy/alex/dev/managed-learn/ml-survey-service/kubernetes/helm_charts/core/ml-survey-service -n alexdev",
"delta": "0:10:00.676598",
"end": "2023-05-30 15:42:39.877263",
"rc": 1,
"start": "2023-05-30 15:32:39.200665"
}

STDOUT:

Release "ml-survey-service" does not exist. Installing it now.

STDERR:

Error: release ml-survey-service failed, and has been uninstalled due to atomic being set: timed out waiting for the condition

MSG:

non-zero return code

PLAY RECAP *********************************************************************
localhost : ok=16 changed=14 unreachable=0 failed=1 skipped=5 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• git name-rev --name-only HEAD
• rev
• cut -d / -f1
• rev
  [Pipeline] sh
• echo Deploy/alex/dev/managed-learn/ml-survey-service
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/alex/dev/managed-learn/ml-survey-service
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[tsprasath]

@srikanth-6520 Please share the pods logs or the events of the deployment

[tsprasath]

aks30 Kindly help @srikanth-6520 on ML service

[heungheung]

have you setup

ml_core_internal_access_token: ""  ### Generated by openssl rand -hex 10

in your private inventory? inventory/dev/Core/common.yml

[srikanth-6520]

@heungheung yes we have already defined the variable in private repo with empty value.

[heungheung]

no. it shouldn't be an empty value.

run the command, generate a value and use it as the internal access token

[srikanth-6520]

@heungheung we have updated the token , now the job got success , thank you.