Unsupported scan IcebergScan

[majetideepak]

How do I fix the fallback? I am using the Velox backend.

WARN GlutenFallbackReporter: Validation failed for plan: BatchScan spark_catalog.tpcds_sf100_parquet_iceberg.store_returns[QueryId=8], due to: Unsupported scan IcebergScan(table=spark_catalog.tpcds_sf100_parquet_iceberg.store_returns, branch=null, type=struct<1: sr_returned_date_sk: optional int, 2: sr_return_time_sk: optional int, 3: sr_item_sk: optional int, 4: sr_customer_sk: optional int, 5: sr_cdemo_sk: optional int, 6: sr_hdemo_sk: optional int, 7: sr_addr_sk: optional int, 8: sr_store_sk: optional int, 9: sr_reason_sk: optional int, 10: sr_ticket_number: optional long, 11: sr_return_quantity: optional int, 12: sr_return_amt: optional decimal(7, 2), 13: sr_return_tax: optional decimal(7, 2), 14: sr_return_amt_inc_tax: optional decimal(7, 2), 15: sr_fee: optional decimal(7, 2), 16: sr_return_ship_cost: optional decimal(7, 2), 17: sr_refunded_cash: optional decimal(7, 2), 18: sr_reversed_charge: optional decimal(7, 2), 19: sr_store_credit: optional decimal(7, 2), 20: sr_net_loss: optional decimal(7, 2)>, filters=[], runtimeFilters=[], caseSensitive=false), original Spark plan is class org.apache.spark.sql.execution.datasources.v2.BatchScanExec(List()).

[majetideepak]

@liujiayi771 can you please help? Thanks.

[liujiayi771]

This error msg may occur when the service loader for IcebergTransformerProvider cannot be found. Is the iceberg profile enabled when compiling Gluten(-Piceberg)? In more recent versions, when the iceberg profile is enabled, the gluten-iceberg will be automatically shaded into the gluten-velox-bundle-xxx.jar. In earlier versions, it was necessary to manually add the gluten-iceberg-xxx.jar to the classpath of the Spark driver and executor.

[majetideepak]

I did enable the profile. I build gluten as
EXTRA_MAVEN_OPTIONS=" -Piceberg " ./dev/buildbundle-veloxbe.sh --spark_version=3.4 --run_setup_script=OFF --enable_s3=ON

[liujiayi771]

Can you use jar -tf gluten-velox-bundle-xxx.jar | grep IcebergTransformerProvider to check if IcebergTransformerProvider is present in the gluten-velox-bundle-xxx.jar file?

[majetideepak]

EXTRA_MAVEN_OPTIONS has to be specified to the mvn command later and it is not passed. Thanks, @liujiayi771