IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2

Like IMPALA-8369, this patch adds a compatibility shim in fe so that
Impala can interoperate with Hive 3.1.2. we need adds a new
Metastoreshim class under compat-apache-hive-3 directory. These shim
classes implement method which are different in cdp-hive-3 vs
apache-hive-3 and are used by front end code. At the build time, based
on the environment variable IMPALA_HIVE_DIST_TYPE one of the two shims
is added to as source using the fe/pom.xml build plugin.

Some codes that directly use Hive 4 APIs need to be ignored in
compilation, eg. fe/src/main/java/org/apache/impala/catalog/metastore/.
Use Maven profile to ignore some codes, profile will automatically
activated based on the IMPALA_HIVE_DIST_TYPE.

Testing:
1. Code compiles and runs against both HMS-3 and ASF-HMS-3
2. Ran full-suite of tests against HMS-3
3. Running full-tests against ASF-HMS-3 will need more work
supporting Tez in the mini-cluster (for dataloading) and HMS
transaction support. This will be on-going effort and test failures
on ASF-Hive-3 will be fixed in additional sub-tasks.

Notes:
1. Patch uses a custom build of Apache Hive to be deployed in
mini-cluster. This build has the fixes for HIVE-21569, HIVE-20038.
This hack will be added to the build script in additional sub-tasks.

Change-Id: I9f08db5f6da735ac431819063060941f0941f606
Reviewed-on: http://gerrit.cloudera.org:8080/17774
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>

bin/jenkins/build-all-flag-combinations.sh

@@ -44,6 +44,8 @@ CONFIGS=(
   "-skiptests -noclean -asan"
   "-skiptests -noclean -tsan"
   "-skiptests -noclean -ubsan -so -ninja"
+  # USE_APACHE_HIVE=true build:
+  "-skiptests -noclean -use_apache_hive"
 )
 
 FAILED=""
@@ -63,7 +65,14 @@ function onexit {
 trap onexit EXIT
 
 for CONFIG in "${CONFIGS[@]}"; do
-  DESCRIPTION="Options $CONFIG"
+  CONFIG2=${CONFIG/-use_apache_hive/}
+  if [[ "$CONFIG" != "$CONFIG2" ]]; then
+    CONFIG=$CONFIG2
+    export USE_APACHE_HIVE=true
+  else
+    export USE_APACHE_HIVE=false
+  fi
+  DESCRIPTION="Options $CONFIG USE_APACHE_HIVE=$USE_APACHE_HIVE"
 
   if [[ $# == 1 && $1 == "--dryrun" ]]; then
     echo $DESCRIPTION

bin/rat_exclude_files.txt

@@ -130,6 +130,7 @@ testdata/AllTypesErrorNoNulls/*.txt
 *.avsc
 *.parq
 *.parquet
+testdata/cluster/hive/*.diff
 testdata/cluster/node_templates/cdh5/etc/hadoop/conf/*.xml.tmpl
 testdata/cluster/node_templates/common/etc/kudu/*.conf.tmpl
 testdata/cluster/node_templates/common/etc/hadoop/conf/*.xml.tmpl

buildall.sh

@@ -419,10 +419,8 @@ bootstrap_dependencies() {
     "$IMPALA_HOME/bin/bootstrap_toolchain.py"
     echo "Toolchain bootstrap complete."
   fi
-  # HIVE-22915
   if [[ "${USE_APACHE_HIVE}" = true ]]; then
-    rm $HIVE_HOME/lib/guava-*jar
-    cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-*.jar $HIVE_HOME/lib/
+    "$IMPALA_HOME/testdata/bin/patch_hive.sh"
   fi
 }
 

fe/pom.xml

@@ -756,7 +756,7 @@ under the License.
                         -->
                 <source>${project.basedir}/generated-sources/gen-java</source>
                 <source>${project.build.directory}/generated-sources/cup</source>
-                <source>${project.basedir}/src/compat-hive-${hive.major.version}/java</source>
+                <source>${project.basedir}/src/compat-${hive.dist.type}-${hive.major.version}/java</source>
                </sources>
             </configuration>
           </execution>
@@ -1053,6 +1053,38 @@ under the License.
       </build>
     </profile>
 
+    <!-- Profile which is automatically activated based on the value of the
+         IMPALA_HIVE_DIST_TYPE environment -->
+    <profile>
+      <id>apache-hive-3</id>
+      <activation>
+        <property>
+          <name>env.IMPALA_HIVE_DIST_TYPE</name>
+          <value>apache-hive</value>
+        </property>
+      </activation>
+      <build>
+        <plugins>
+          <plugin>
+            <groupId>org.apache.maven.plugins</groupId>
+            <artifactId>maven-compiler-plugin</artifactId>
+            <version>3.3</version>
+            <configuration>
+              <excludes>
+                <exclude>**/org/apache/impala/catalog/metastore/*.java</exclude>
+                <exclude>**/org/apache/impala/catalog/CatalogHmsAPIHelper.java</exclude>
+                <exclude>**/org/apache/impala/catalog/CompactionInfoLoader.java</exclude>
+              </excludes>
+              <testExcludes>
+                <testExclude>**/org/apache/impala/catalog/metastore/*.java</testExclude>
+                <testExclude>**/org/apache/impala/testutil/Catalog*.java</testExclude>
+              </testExcludes>
+            </configuration>
+          </plugin>
+        </plugins>
+      </build>
+    </profile>
+
     <!-- Profile which is automatically activated when building from
          within Eclipse based on the presence of the m2e.version
          property -->