Deploy -> Datapipeline -> AdhocScripts

[srikanth-6520]

Hello @santhosh-tg @gandham-santhosh , we are getting an error while deploying the "Adhoscripts" job with tag "release-5.1.0" ,

Jenkins Logs:

Started by user [Kolluri Srikanth]
Rebuilds build #[7]
Checking out git https://github.com/project-sunbird/sunbird-data-pipeline.git into /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts@script/8585a5a3f074076bd537559dc5c44af39f26ac77fb54137179a27fb09804ed12 to read pipelines/deploy/datascience/Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts@script/8585a5a3f074076bd537559dc5c44af39f26ac77fb54137179a27fb09804ed12/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-data-pipeline.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-data-pipeline.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-data-pipeline.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0^{commit} # timeout=10
Checking out Revision e7d0f2b5189d9a74f8a5d0631f913039aeca6c56 (origin/release-5.1.0)
git config core.sparsecheckout # timeout=10
git checkout -f e7d0f2b5189d9a74f8a5d0631f913039aeca6c56 # timeout=10
Commit message: "Merge pull request #1599 from anandp504/release-5.1.0"
git rev-list --no-walk e7d0f2b5189d9a74f8a5d0631f913039aeca6c56 # timeout=10
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts@libs/b4ac4d1235c67cfedda9d7bf40936b35bc6929460ef19107abe1b183748d4145/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh

• git clean -fxd
  Removing ansible/.portal.yml.swp
  Removing ansible/dataproducts.tar.gz
  Removing dataproducts_artifacts.zip:release-4.5.1_RC1
  Removing metadata.json
  Skipping repository private/
  Removing private@tmp/
  [Pipeline] checkout
  The recommended git tool is: NONE
  No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-data-pipeline.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-data-pipeline.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-data-pipeline.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0^{commit} # timeout=10
Checking out Revision e7d0f2b5189d9a74f8a5d0631f913039aeca6c56 (origin/release-5.1.0)
git config core.sparsecheckout # timeout=10
git checkout -f e7d0f2b5189d9a74f8a5d0631f913039aeca6c56 # timeout=10
Commit message: "Merge pull request #1599 from anandp504/release-5.1.0"
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/alex/dev/DataPipeline/AdhocScripts
  [Pipeline] sh
• echo Deploy/alex/dev/DataPipeline/AdhocScripts
  [Pipeline] sh
• echo Deploy/alex/dev/DataPipeline/AdhocScripts
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] copyArtifacts
  Copied 2 artifacts from "ArtifactUpload » dev » DataPipeline » AdhocScripts" build number 4
  [Pipeline] sh
• jq -r .artifact_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  artifact_version not specified, using the artifact_version specified in metadata.json.
  [Pipeline] sh
• jq -r .artifact_version metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] stage
[Pipeline] { (get artifact)
[Pipeline] sh

• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] echo
Option chosen is JenkinsJob, using the artifacts copied
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy artifact)
[Pipeline] sh

• unzip dataproducts_artifacts.zip:release-4.5.1_RC1
  Archive: dataproducts_artifacts.zip:release-4.5.1_RC1
  inflating: dataproducts.tar.gz
• mv dataproducts.tar.gz ansible
  [Pipeline] echo
  {env=dev, module=DataPipeline, jobName=AdhocScripts, absolute_job_path=ArtifactUpload/dev/DataPipeline/AdhocScripts, agent=built-in, artifact_name=dataproducts_artifacts.zip, artifact_version=release-4.5.1_RC1, currentWs=/var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts, artifact=dataproducts_artifacts.zip:release-4.5.1_RC1, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/ansible/portal.yml, ansibleExtraArgs=--extra-vars "date=" --vault-password-file /var/lib/jenkins/secrets/vault-pass --tags common,daily_metrics,content_aggregates,content_progress,content_creation,gps_learning,dialcode_exception_etb_reports,district_monthly,district_weekly,landing_page,cmo_dashboards}
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=DataPipeline, jobName=AdhocScripts, absolute_job_path=ArtifactUpload/dev/DataPipeline/AdhocScripts, agent=built-in, artifact_name=dataproducts_artifacts.zip, artifact_version=release-4.5.1_RC1, currentWs=/var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts, artifact=dataproducts_artifacts.zip:release-4.5.1_RC1, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/ansible/portal.yml, ansibleExtraArgs=--extra-vars "date=" --vault-password-file /var/lib/jenkins/secrets/vault-pass --tags common,daily_metrics,content_aggregates,content_progress,content_creation,gps_learning,dialcode_exception_etb_reports,district_monthly,district_weekly,landing_page,cmo_dashboards}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential bitbucket-cred

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/private/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url bitbucket.org:verlab/veranda-alex-sb-private.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from bitbucket.org:verlab/veranda-alex-sb-private.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_SSH to set credentials Veranda Bitbucket Credential (https://bitbucket.org/verlab/veranda-alex-sb-private)
Verifying host key using known hosts file
git fetch --no-tags --progress --depth=1 -- bitbucket.org:verlab/veranda-alex-sb-private.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-5.1.0^{commit} # timeout=10
Checking out Revision 9897299bc8b1e7816cdb3ef026ca4b62bf35460f (origin/release-5.1.0)
git config core.sparsecheckout # timeout=10
git checkout -f 9897299bc8b1e7816cdb3ef026ca4b62bf35460f # timeout=10
Commit message: "sunbird_public_storage_account_name variable added"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/private/ansible/inventory/dev/DataPipeline/common.yml /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/private/ansible/inventory/dev/DataPipeline/hosts /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/private/ansible/inventory/dev/DataPipeline/secrets.yml /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/ansible/portal.yml --extra-vars date= --vault-password-file /var/lib/jenkins/secrets/vault-pass --tags common,daily_metrics,content_aggregates,content_progress,content_creation,gps_learning,dialcode_exception_etb_reports,district_monthly,district_weekly,landing_page,cmo_dashboards
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: Found both group and host with same name: raw-coordinator

PLAY [spark] *******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [172.16.1.34]

TASK [portal-dashboard : Copy the library to library path] *********************
ok: [172.16.1.34]

TASK [portal-dashboard : Installing package] ***********************************
changed: [172.16.1.34]

TASK [portal-dashboard : execute_content_creation] *****************************
fatal: [172.16.1.34]: FAILED! => {}

MSG:

The task includes an option with an undefined variable. The error was: 'search_lb_ip' is undefined

The error appears to be in '/var/lib/jenkins/workspace/Deploy/alex/dev/DataPipeline/AdhocScripts/ansible/roles/portal-dashboard/tasks/main.yml': line 19, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

• name: execute_content_creation
  ^ here

PLAY RECAP *********************************************************************
172.16.1.34 : ok=3 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• git name-rev --name-only HEAD
• rev
• cut -d / -f1
• rev
  [Pipeline] sh
• echo Deploy/alex/dev/DataPipeline/AdhocScripts
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/alex/dev/DataPipeline/AdhocScripts
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[chitranshu-keshav]

@santhosh-tg @gandham-santhosh can you please get back on this

[heungheung]

Please check the description when you build it in Jenkins. If you are just setting up Sunbird-Ed, using common as tag is enough because it is for "deploy" the required library.

The other tags are actually for "generate" the respective reports. Until you have the setup completed and data, you probably won't have any report to generate.

=========== The above is the answer to your original question >> for setup, just use the tag common

If you have everything ready and want to generate a report, you need the other tag. From the Jenkins output you've shared, you were actually trying to generate the content creation report, using the task
https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/release-5.1.0/ansible/roles/portal-dashboard/tasks/main.yml#L19-L25

When running the shell command in line 20

source {{virtualenv_path}}/bin/activate && dataproducts content_creation --data_store_location={{data_path}} --execution_date={{date}} --content_search={{content_search_api}} --org_search={{org_search_api}}

one of the argument is API endpoint content_search_api which is defined in
https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/release-5.1.0/ansible/roles/portal-dashboard/defaults/main.yml#L6
You have this error because you didn't have the default value defined for search_lb_ip

The solution is to define this variable in common.yml of DP.

@santhosh-tg I don't think this search api LB is in the documentation / in the sample ansible inventory.

@srikanth-6520 not sure if you are doing the deployment in Azure. I remember the adhoc scripts only have the Azure specific code. If you are using other CSP. You might need to do some modification inside the dataproducts libs.

[sandeep-r1]

@heungheung as per sunbird ed document we have used release-4.5.1_RC1 to build AdhocScripts. Kindly correct if i am using a wrong tag and we are not getting any error while building the AdhocScripts

cc: @srikanth-6520

[santhosh-tg]

@santhosh-tg I don't think this search api LB is in the documentation / in the sample ansible inventory.

Yes @heungheung Have raised PR for the ansible inventory project-sunbird/sunbird-devops#3828 @gandham-santhosh Please review and merge.

[srikanth-6520]

@heungheung as per you are suggestion , we have used "common" tag in deployment & it got success as of now, thank you.

[santhosh-tg]

@srikanth-6520 Please mark this as answered if the issues are resolved.