HIVE-28727: Iceberg: Null partitions make major compactions fail

[okumin]

What changes were proposed in this pull request?

Let IcebergTableUtil list NULL partitions so that Hive can successfully compact a table that includes such partitions.

Why are the changes needed?

OPTIMIZE TABLE ... REWRITE DATA or ALTER TABLE ... COMPACT 'major' fail on the current master branch when a null partition is included.

Does this PR introduce any user-facing change?

At least, such compaction works on 4.0.1. So, I think we can say this is not a user-facing change. I suspect HIVE-28256 introduced the issue though my confidence is far from 100%

Is the change a dependency upgrade?

No.

How was this patch tested?

I updated an existing qtest.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[okumin]

I wonder why two {"company_id":null,"dept_id":null,"team_id":null} remain. I think it is minor and out of scope of HIVE-28727

[okumin]

@difin Could you please have a look? You are one of the most familiar engineers with Apache Icebergs' compaction.

[deniskuzZ]

Could we simplify this with

StructProjection projection = StructProjection.create(sourceKeyType, targetKeyType)
    .wrap(sourceKey);
PartitionData keyTemplate = new PartitionData(targetKeyType);
return keyTemplate.copyFor(projection);

[deniskuzZ]

above PartitionData toPartitionData(StructLike key, Types.StructType keyType) could be replaced with

PartitionData keyTemplate = new PartitionData(keyType);
return keyTemplate.copyFor(key);

[deniskuzZ]

since you are doing refactor here, could we also leverage IcebergTableUtil#toPartitionData in HiveIcebergStorageHandler#getPartishSummary? see

try (Closeable toClose = partitionStatsRecords) {
  PartitionStats partitionStats = Iterables.tryFind(partitionStatsRecords, stats -> {
    PartitionSpec spec = table.specs().get(stats.specId());
    
    PartitionData data  = IcebergTableUtil.toPartitionData(stats.partition(), 
        partitionType, spec.partitionType());
    return spec.partitionToPath(data).equals(partish.getPartition().getName());

  }).orNull();

[deniskuzZ]

hi @okumin, WDYT, does it make sense? Tried locally and test with your updates passed.