Is there an easier way to register hudi tables in HMS?

[alberttwong]

Per https://xtable.apache.org/docs/hms#register-the-target-table-in-hive-metastore, if you have hudi files, you have to use hudi-hive-sync. However, to use hudi-hive-sync, you need to install an instance of Hadoop and Hive. That's almost 1GB of files just to use the utility. Is there an easier way?

Also the docs says to git clone the entire hudi project. Is there a way just to get the hudi-hive-sync utility by itself?

[alberttwong]

I wasn't able to get around the installation of Hadoop and Hive but you don't have to compile the Hudi-Hive-Sync

For convience I'm using pyspark (via Apache Ivy) to download the dependancies and missing files for hudi-hive-sync/run_sync_tool.sh and then modifying the run_sync_tool script to add the ivy downloaded jars to classpath (~/.ivy2/jars)

pyspark --packages org.apache.hudi:hudi-hive-sync-bundle:0.14.1,com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-client:2.10.2,org.apache.hadoop:hadoop-aws:2.10.2
pyspark --packages org.apache.parquet:parquet-avro:1.10.1,com.esotericsoftware:kryo-shaded:4.0.2