Create a wrapper around YAML metadata

[Marigold]

We use YAML files extensively to manage metadata. This wasn't always the case, and over time we've accumulated a lot of technical debt from managing metadata in YAML files. It would be great to have everything centralized and refactored into a single class.

Proposal (in progress)

Create a class YAMLMeta with methods like

class YAMLMeta:

  def save(...)
  def load(...)
  def update(...)
  def merge(...)
  def validate(...)
  def render(dim_dict):
     # render Jinja with values given in dim_dict, should replace `render_yaml_file`
  ...

Since we have several types of YAML configs (for dataset metadata, MDIMs, snapshots, etc.), it would make sense to have a base class with subclasses for each case.

Overview of libraries for working with YAML files

• dynamic_yaml – used to load YAML files and evaluate placeholders.
• jinja2 – used to render Jinja templates with dimension values (it's not run on the entire YAML file, but on text inside fields like title).
• ruamel.yaml – used for editing YAML files while preserving comments and overall structure.
• yaml_dump – a function used instead of yaml.dump with better defaults.

There's also an interesting library, OmegaConf, which could potentially replace dynamic_yaml. It offers neat features, like accessing attributes via both .field and ["field"], and merging two YAML files.