Deploy -> Datapipeline -> AnalyticsCore

[srikanth-6520]

Hello @santhosh-tg @gandham-santhosh , we are getting an error while deploying the "AnalyticsCore" job with tag "release-5.1.0" ,

Jenkins Logs:

Started by user [sandeepr]
Rebuilds build #[5]
Checking out git https://github.com/project-sunbird/sunbird-data-pipeline.git into /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore@script/8585a5a3f074076bd537559dc5c44af39f26ac77fb54137179a27fb09804ed12 to read pipelines/deploy/data-products/Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore@script/8585a5a3f074076bd537559dc5c44af39f26ac77fb54137179a27fb09804ed12/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-data-pipeline.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-data-pipeline.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-data-pipeline.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0^{commit} # timeout=10
Checking out Revision e7d0f2b5189d9a74f8a5d0631f913039aeca6c56 (origin/release-5.1.0)
git config core.sparsecheckout # timeout=10
git checkout -f e7d0f2b5189d9a74f8a5d0631f913039aeca6c56 # timeout=10
Commit message: "Merge pull request #1599 from anandp504/release-5.1.0"
git rev-list --no-walk e7d0f2b5189d9a74f8a5d0631f913039aeca6c56 # timeout=10
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore@libs/b4ac4d1235c67cfedda9d7bf40936b35bc6929460ef19107abe1b183748d4145/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh

• git clean -fxd
  Skipping repository private/
  [Pipeline] checkout
  The recommended git tool is: NONE
  No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-data-pipeline.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-data-pipeline.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-data-pipeline.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0^{commit} # timeout=10
Checking out Revision e7d0f2b5189d9a74f8a5d0631f913039aeca6c56 (origin/release-5.1.0)
git config core.sparsecheckout # timeout=10
git checkout -f e7d0f2b5189d9a74f8a5d0631f913039aeca6c56 # timeout=10
Commit message: "Merge pull request #1599 from anandp504/release-5.1.0"
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/alex/dev/DataPipeline/AnalyticsCore
  [Pipeline] sh
• echo Deploy/alex/dev/DataPipeline/AnalyticsCore
  [Pipeline] sh
• echo Deploy/alex/dev/DataPipeline/AnalyticsCore
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] copyArtifacts
  Copied 2 artifacts from "ArtifactUpload » dev » DataPipeline » AnalyticsCore" build number 3
  [Pipeline] sh
• jq -r .artifact_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  artifact_version not specified, using the artifact_version specified in metadata.json.
  [Pipeline] sh
• jq -r .artifact_version metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] stage
[Pipeline] { (get artifact)
[Pipeline] sh

• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] echo
Option chosen is JenkinsJob, using the artifacts copied
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy artifact)
[Pipeline] sh

• unzip lpa_core_artifacts.zip:release-5.0.0_RC2
  Archive: lpa_core_artifacts.zip:release-5.0.0_RC2
  inflating: analytics-framework-2.0.jar
  inflating: scruid_2.11-2.4.0.jar
  inflating: scruid_2.12-2.4.0.jar
  inflating: scruid_2.12-2.5.0.jar
• [ -f batch-models-2.0.jar ]
• [ -f job-manager-2.0.jar ]
• [ -f analytics-framework-2.0.jar ]
• echo analytics-framework-2.0.jar exist
  analytics-framework-2.0.jar exist
• mv analytics-framework-2.0.jar ansible
• [ -f scruid_2.12-2.5.0.jar ]
• echo scruid_2.12-2.5.0.jar exist
  scruid_2.12-2.5.0.jar exist
• mv scruid_2.12-2.5.0.jar ansible
  [Pipeline] echo
  {env=dev, module=DataPipeline, jobName=AnalyticsCore, absolute_job_path=ArtifactUpload/dev/DataPipeline/AnalyticsCore, agent=built-in, artifact_name=lpa_core_artifacts.zip, artifact_version=release-5.0.0_RC2, currentWs=/var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore, artifact=lpa_core_artifacts.zip:release-5.0.0_RC2, ansiblePlaybook=--extra-vars "remote=spark" /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/ansible/lpa_data-products_deploy.yml, ansibleExtraArgs= --tags framework,framework-spark-cluster --vault-password-file /var/lib/jenkins/secrets/vault-pass}
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=DataPipeline, jobName=AnalyticsCore, absolute_job_path=ArtifactUpload/dev/DataPipeline/AnalyticsCore, agent=built-in, artifact_name=lpa_core_artifacts.zip, artifact_version=release-5.0.0_RC2, currentWs=/var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore, artifact=lpa_core_artifacts.zip:release-5.0.0_RC2, ansiblePlaybook=--extra-vars "remote=spark" /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/ansible/lpa_data-products_deploy.yml, ansibleExtraArgs= --tags framework,framework-spark-cluster --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential bitbucket-cred

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/private/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url bitbucket.org:verlab/veranda-alex-sb-private.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from bitbucket.org:verlab/veranda-alex-sb-private.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_SSH to set credentials Veranda Bitbucket Credential (https://bitbucket.org/verlab/veranda-alex-sb-private)
Verifying host key using known hosts file
git fetch --no-tags --progress --depth=1 -- bitbucket.org:verlab/veranda-alex-sb-private.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0-alex-dev^{commit} # timeout=10
Checking out Revision 062b1bd1f2b8ffab1a5d3db393387cc34f7c5b23 (origin/release-5.1.0-alex-dev)
git config core.sparsecheckout # timeout=10
git checkout -f 062b1bd1f2b8ffab1a5d3db393387cc34f7c5b23 # timeout=10
Commit message: "we have added sunbird_private_storage_account_key variable"
git rev-list --no-walk 062b1bd1f2b8ffab1a5d3db393387cc34f7c5b23 # timeout=10
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/private/ansible/inventory/dev/DataPipeline/common.yml /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/private/ansible/inventory/dev/DataPipeline/hosts /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/private/ansible/inventory/dev/DataPipeline/secrets.yml /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/hosts --extra-vars remote=spark /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AnalyticsCore/ansible/lpa_data-products_deploy.yml --tags framework,framework-spark-cluster --vault-password-file /var/lib/jenkins/secrets/vault-pass
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: Found both group and host with same name: raw-coordinator

PLAY [spark] *******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [172.16.1.34]

TASK [data-products-deploy : Ensure azure blob storage container exists] *******
fatal: [172.16.1.34]: FAILED! => {
"changed": true,
"cmd": [
"az",
"storage",
"container",
"create",
"--name",
"telemetry-data-store"
],
"delta": "0:01:24.190419",
"end": "2023-05-25 07:11:40.867857",
"rc": 1,
"start": "2023-05-25 07:10:16.677438"
}

STDERR:

ERROR: <urllib3.connection.HTTPSConnection object at 0x7feaf7989180>: Failed to establish a new connection: [Errno -2] Name or service not known

MSG:

non-zero return code

PLAY RECAP *********************************************************************
172.16.1.34 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• rev
• rev
• • gitcut name-rev -d --name-only / HEAD -f1

[Pipeline] sh

• echo Deploy/alex/dev/DataPipeline/AnalyticsCore
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/alex/dev/DataPipeline/AnalyticsCore
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[chitranshu-keshav]

@santhosh-tg @gandham-santhosh can you please get back on this

[heungheung]

From discussion 505, I understand that you are deploying Sunbird-Ed on Azure. That's why you hit this error - you shouldn't need to run az command(s) in the Ansible role.

You can use
https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/release-5.1.0/ansible/roles/data-products-deploy/tasks/main.yml#L4
as example and add when conditional for ALL of the az commands. At the same time, you will need to add the same commands for aws s3 storage. The command will be something like this
https://github.com/project-sunbird/sunbird-devops/blob/release-5.1.0/ansible/roles/aws-cloud-storage/tasks/upload.yml#L7

If you have no intention to extend the code by yourself, you might need to wait for the Sunbird community to get the code updated. IMHO, try to fork the repo and extend the code so that you can quickly do your own changes. Once you've test your code, you can contribute back by creating a PR.

[santhosh-tg]

@srikanth-6520 Is this still an issue ? If not, please mark it as answered.