🚚 Stop copying yaml content when a new language is added

[boryanagoncharenko]

This is a draft PR meant to collect feedback for a solution of issue #5187.

Description of the problem
When a new language is added in Weblate, the prefill add-on copies the English content into the new language yaml files. This means that when the English content changes, we manually need make the change in all language that do not have the content translated. Instead of this, we would like to keep in the yaml files only the content that has actually been translated.

Yaml files
All newly added content/yaml files are not prefilled with the current English translation and only contain the translated value for the corresponding language. Currently, only the om language has partial content to illustrate the solution. The content of each yaml file is merged with the en.yaml counterpart when the yaml is loaded.

• Schema validation. The schema definitions have explicit required fields and only the en.yaml files are checked against them. A python script generates new schema definitions (*.generated.schema.json) that lift the required fields constraints. The generation is part of the doit backend task and the schemas are gitignored. The generated schemas are used for all non-en.yaml files.
• Correctness check added in [CHORE] Add a tool to check the length of arrays in YAMLs #4020. We have a script that checks if there are different array lengths between English and other languages. This check is amended to ensure the structure of the language yaml file is a subset of the English file. Now it check that:
  • The nodes in the 2 yaml files are of the same type, specifically arrays and dictionaries.
  • An array in a language file should not have more elements than the English file.
  • A dictionary in the language file should not have keys that are not present in the English file.
• Note that the merging mechanism silently discards the content of the language file if any of the correctness rules are not met. The assumption is that the English file will have the correct structure. This meant to be inline with the silent empty dictionary if an IOError occurs while loading the yaml file. However, we could have a different approach here: fail and just load the fallback? Or at least log that something went wrong?

Po files
Just like the yaml files, all newly added .po files will not be prefilled with English and, again, the om language is used as an example. The merging of every language with English happens through a custom function that defines a fallback. The regular gettext function is substituted with the custom one.

Deduplication of existing languages
The om language is added to this PR for exemplary purposes and is not an officially supported language. Since all current languages have a version of en.yaml files, the merging will not yield immediate results. It will take effect only when a new language is added. Perhaps it is wise to run a basic form of deduplication of the current files and test immediately. Note that complete deduplication of the yaml files is currently very hard to achieve because every yaml file copies a different version of the Enligsh content. However, we could simplify the problem by only removing the duplication that matches the current English version.

[rix0rrr]

Love it! ❤️

[mergify]

Thank you for contributing! Your pull request is now going on the merge train (choo choo! Do not click update from main anymore, and be sure to allow changes to be pushed to your fork).

[mergify]

Thank you for contributing! Your pull request is now going on the merge train (choo choo! Do not click update from main anymore, and be sure to allow changes to be pushed to your fork).

[mergify]

Thank you for contributing! Your pull request is now going on the merge train (choo choo! Do not click update from main anymore, and be sure to allow changes to be pushed to your fork).

[mergify]

Thank you for contributing! Your pull request is now going on the merge train (choo choo! Do not click update from main anymore, and be sure to allow changes to be pushed to your fork).

[mergify]

Thank you for contributing! Your pull request is now going on the merge train (choo choo! Do not click update from main anymore, and be sure to allow changes to be pushed to your fork).