[HUDI-7696] Refactoring functions in HoodieTableMetadataUtil

[bibhu107]

Change Logs

Two functions in HoodieTableMetadataUtil, convertFilesToPartitionStatsRecords and convertMetadataToPartitionStatsRecords, are being evaluated for unification.
These functions perform similar tasks of creating statistics for records in a partition. The proposal aims to combine them into a single function, improving code readability and maintainability.

Reference: #10352 (comment)

Impact

This change will primarily affect the internal implementation of HoodieTableMetadataUtil. While it's not expected to impact the public API directly.

There are simillar other functions like mentioned below :

• convertFilesToColumnStatsRecords and convertMetadataToColumnStatsRecords
• convertFilesToBloomFilterRecords and convertMetadataToBloomFilterRecords

Unification challenge:
To merge these function pairs, we would need to transform HoodieCommitMetadata into a format compatible with the unified function. This transformation process could potentially introduce inefficiencies.

Risk level

Low

Documentation Update

N/A

As this is an internal code refactoring, no user-facing documentation updates are required.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

cc - @codope

[codope]

@bibhu107 Thanks for your contribution. After going through the code, I think unifying the two methods is neither easy nor desirable. However, some common logic can be extracted. For example, if we introduce a method below that can be reused in the two methods, then the logic for collecting, grouping, and aggregating column metadata is extracted out. Let me know what you think.

private static Stream<HoodieRecord> collectAndProcessColumnMetadata(
    List<List<HoodieColumnRangeMetadata<Comparable>>> fileColumnMetadata,
    String partitionPath) {
  
  // Step 2: Flatten and Group by Column Name
  Map<String, List<HoodieColumnRangeMetadata<Comparable>>> columnMetadataMap = fileColumnMetadata.stream()
      .flatMap(List::stream)
      .collect(Collectors.groupingBy(HoodieColumnRangeMetadata::getColumnName, Collectors.toList())); // Group by column name
  
  // Step 3: Aggregate Column Ranges
  Stream<HoodieColumnRangeMetadata<Comparable>> partitionStatsRangeMetadata = columnMetadataMap.entrySet().stream()
      .map(entry -> FileFormatUtils.getColumnRangeInPartition(partitionPath, entry.getValue()));
  
  // Create Partition Stats Records
  return HoodieMetadataPayload.createPartitionStatsRecords(partitionPath, partitionStatsRangeMetadata.collect(Collectors.toList()), false).stream();
}

[bibhu107]

Makes sense @codope. Have made necessary changes.

[hudi-bot]

CI report:

• 6d3f0f7 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build