[FEATURE] Supporting storing column metadata in Gravitino

[jerryshao]

Describe the feature

Currently, column is managed out of Gravitino, for example, column in a Hive table is gotten from HMS, column in an Iceberg table is gotten from Iceberg's metadata.

The current design choice avoids the duplication of maintaining column metadata both in Gravitino and underlying sources. But in the meantime, it brings some limitations like tagging on columns is hard to achieve in Gravitino.

So we should figure out a solution to handle this scenario: support some operations in Gravitino while keep consistency between two services for column.

Attach the design doc (https://docs.google.com/document/d/1X9kFeckDMrzqHo5fcOO55bnU0GIf2DM0KxheH4Nry54/edit?usp=sharing)

Tasks:

• [Subtask] Define the column and column entity in Gravitino #4795
• [Subtask] Design and implement the column storage layout for columns. #4867
• [Subtask] Implement the column management logic in Gravitino. #5086
• [Subtask] Support adding tags on column #4599
• [Subtask] Support column position for column entity #5159

Motivation

No response

Describe the solution

No response

Additional context

No response

[jerryshao]

Here attaches the design doc https://docs.google.com/document/d/1X9kFeckDMrzqHo5fcOO55bnU0GIf2DM0KxheH4Nry54/edit?usp=sharing