How to get descriptions saved as comments?

[Tobychev]

In pydantic you can use the Field function to attach a description to the model properties, and I wonder if there is a way that pydantic-yaml could save that description as a comment?

[NowanIlfideme]

Hi, currently that isn't possible through pydantic-yaml. It could be possible to do through the YAML backend we use, ruamel.yaml (see here for details), but currently the loading/dumping is done through JSON first, which doesn't support comments.

This is functionality I want as well, but so far the only way I've figured out would be writing a custom YAML engine (possibly on top of ruamel.yaml), which is... somewhere in the pipeline, but not now. 😅

[Tobychev]

I guess it could be possible to take the json serialisation that pydantic provides and annotate it with the metadata in a format ruamel.yaml accepts? Technically it woudn't be a completely custom engine?

[NowanIlfideme]

Well, yes, you can look through the ModelField classes and add comments. However, you need to be able to map the model fields to the JSON Path in the resulting mess of dict/list/scalars from the JSON.

There's also the consideration of how you'd annotate it...
Do you apply the comment to only the field value? If it's a list, where do you anchor the documentation? Should you output description as a comment for all fields, or just a subset? I have some opinions on this, but for usability it should really be configurable.

If you have a simple use-case right now (i.e. just one or more nested Pydantic models with fields), it is actually not that difficult to implement - you can use model.dict() (or the equivalent in Pydantic v2, not sure - model_dict()?), pass it to a ruamel.yaml CommentedMap, then add comments from the Pydantic fields. Or - even better - you can write it with pydantic-yaml to a temporary buffer, read with ruamel.yaml with a round-trip-capable serializer, and add comments through that interface.

Making this work in the general case is an issue, since you can have containers (lists, dicts, list-like, dict-like, ...) of Pydantic and non-Pydantic objects. And all can have field validations.

I hope I will find something in pydantic-core that will help with this, once I get to it. I'll be on vacation soon, so mayyyyybe...

[ipcoder]

Up-voting this!
:-)

[apirogov]

I'd love to see that! A one-line function to dump a "default YAML config file" with comments directly from a model to me would be a killer feature when building tools :)

[scottstanie]

Hi, I believe I started looking into this before this project got going: https://github.com/opera-adt/dolphin/blob/main/src/dolphin/workflows/_yaml_model.py
I created the YamlModel for my project which has to_yaml and from_yaml to load and save.

Since that part is already here, perhaps someone would be able to take the _add_comments function from here https://github.com/opera-adt/dolphin/blob/f9f2f87e83140f43550c0842aa140ac9f01cab43/src/dolphin/workflows/_yaml_model.py#L136

This for Pydantic V1, and i'm working on updating things (if there were any breaking changes) to V2 shortly.

[richmr]

Hi there. Came across this discussion after having the same functional goal of writing my models and having them make self-documenting YAML templates. I spent some time looking at the pydantic-yaml code and the underlying ruamel.yaml library. I'm sure a plugin or subclass could make this happen, but it looked like a lot to learn and sort of reverse-engineer.

Instead I wrote this gist that works at the file level and leverages the original models used to make the template file.

I'm sure there are a TON of use cases the gist doesn't cover but its working for my needs.

Hope it helps!

[fer-rum]

Indeed, Adding comments to the YAML file from a fields' description (or docstring) would be amazing.
The approach of @richmr already looks quite promising.