Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run mvn with the project's pom.xml in hybrid_execution.sh #12064

Merged
merged 1 commit into from
Feb 6, 2025

Conversation

NvTimLiu
Copy link
Collaborator

@NvTimLiu NvTimLiu commented Feb 5, 2025

This script may run outside the project root path, so we use 'mvn -f $project_root' to target the project's pom.xml

To fix below integration test failure because of the wrong walue of RAPIDS_HYBRID_VER:

RAPIDS_HYBRID_VER='null object or invalid expression'

 + '[' -z '/tmp/rapids-4-spark-hybrid_2.12-null object or invalid expression.jar,/tmp/gluten-velox-bundle-1.2.0-spark3.2_2.12-ubuntu_22.04_x86_64.jar,/tmp/gluten-thirdparty-lib-1.2.0-ubuntu_22.04-x86_64.jar' ']'
 + export 'PYSP_TEST_spark_jars=/home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/rapids-4-spark_2.12-25.02.0-SNAPSHOT-cuda11.jar,/home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/rapids-4-spark-integration-tests_2.12-25.02.0-SNAPSHOT-spark320.jar,/home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/parquet-hadoop-1.12.1-tests.jar,/tmp/rapids-4-spark-hybrid_2.12-null object or invalid expression.jar,/tmp/gluten-velox-bundle-1.2.0-spark3.2_2.12-ubuntu_22.04_x86_64.jar,/tmp/gluten-thirdparty-lib-1.2.0-ubuntu_22.04-x86_64.jar'
 + PYSP_TEST_spark_jars='/home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/rapids-4-spark_2.12-25.02.0-SNAPSHOT-cuda11.jar,/home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/rapids-4-spark-integration-tests_2.12-25.02.0-SNAPSHOT-spark320.jar,/home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/parquet-hadoop-1.12.1-tests.jar,/tmp/rapids-4-spark-hybrid_2.12-null object or invalid expression.jar,/tmp/gluten-velox-bundle-1.2.0-spark3.2_2.12-ubuntu_22.04_x86_64.jar,/tmp/gluten-thirdparty-lib-1.2.0-ubuntu_22.04-x86_64.jar'
 + export PYSP_TEST_spark_rapids_sql_parquet_useHybridReader=true
 ....

 + exec python /home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/integration_tests/runtests.py --rootdir /home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/integration_tests /home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/integration_tests/src/main/python -n 4 -v -rfE '' --std_input_path=/home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/integration_tests/src/test/resources --color=yes --test_type nightly '' --junitxml=TEST-pytest-1738791165075728929.xml -m hybrid_test
Exception in thread "main" org.apache.spark.SparkException: Failed to get main class in JAR with error 'File file:/home/jenkins/agent/workspace/jenkins-rapids_it-AQE-pre_release-github-381/jars/integration_tests/target/run_dir-20250205213245-fX0Z/object does not exist'.  Please specify one with --class.
	at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:972)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:486)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

To follow up: #12054

This script may run outside the project root path, so we use 'mvn -f $project_root' to target the project's pom.xml

Signed-off-by: timl <[email protected]>
@NvTimLiu NvTimLiu self-assigned this Feb 5, 2025
@NvTimLiu NvTimLiu requested a review from a team as a code owner February 5, 2025 10:58
@NvTimLiu NvTimLiu added the build Related to CI / CD or cleanly building label Feb 5, 2025
@NvTimLiu
Copy link
Collaborator Author

NvTimLiu commented Feb 5, 2025

build

@NvTimLiu
Copy link
Collaborator Author

NvTimLiu commented Feb 6, 2025

integration test CI got PASS:

image

@NvTimLiu NvTimLiu merged commit 602b16a into NVIDIA:branch-25.02 Feb 6, 2025
52 of 53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants