Is it possible to retain the order of the object when serializing

[devantler]

I am using YAMLDotnet to build a devantler/dotnet-kubernetes-generator, which is a library to generate Kubernetes API resources as well as custom resources. I am building this library as I need it for a new feature (generate Kubernetes resources with ksail gen <name-of-resource>) in my OS project devantler/ksail.

When serializing objects, the order is changed such that the serialized object is in lexical order. I would like to retain the order such that for example Kubernetes Secrets look like this:

---
apiVersion: v1
kind: Secret
metadata:
  name: secret
  namespace: default
type: Opaque
data: {}

And not this:

---
apiVersion: v1
data: {}
kind: Secret
metadata:
  name: secret
  namespace: default
type: Opaque

Is it possible to change the serialization to keep the order of the object:

var model = new V1Secret
{
  ApiVersion = "v1",
  Kind = "Secret",
  Metadata = new V1ObjectMeta
  {
    Name = "secret",
    NamespaceProperty = "default"
  },
  Type = "Opaque",
  Data = new Dictionary<string, byte[]>
  {
  }
};

[EdwardCooke]

You may need to use a custom yaml converter. I suspect you’re using the kubernetes client library classes so putting on the yamlmember attribute isn’t a possibility.

The yaml converters allow you to serialize and deserialize exactly the way you want.

[devantler]

I am using this wrapper around YamlDotNet, so the Kubernetes client library classes are no issue: https://github.com/IvanJosipovic/YamlDotNet.System.Text.Json

Thank's I will see if I can cook up a converter for the purpose. K8s yaml looks pretty ew, when its mixed like that 😬

[EdwardCooke]

My guess, based on your example, is that it has something to do with this line in that library that is causing you problems:

.WithTypeConverter(new SystemTextJsonYamlTypeConverter(sortAlphabetically))

Our library typically serializes in the order of the properties as they are declared in the class by default. But there's no guarantees. It's really in whatever order they are returned in the reflection calls or dictionary/list.

[devantler]

Thanks! I will play around with it a bit when time allows, and see if I can find a solution I like. I am mostly concerned with the root object returning the boilerplate at the top, in this order:

apiVersion: <api-version>
kind: <kind>
metadata:
  <metadata-elements>

Given that is the order I create the non-serialized objects in, I believe disabling the alphabetical sort is fine :-) It might still cause elements of list and dictionaries to come in a random order, but that is a non-issue I believe.

[devantler]

Hmm, without the other library I still get a funny order:

ISerializer _serializer = new SerializerBuilder()
    .ConfigureDefaultValuesHandling(DefaultValuesHandling.OmitNull)
    .WithNamingConvention(CamelCaseNamingConvention.Instance).Build();

var model = new V1Secret
{
  ApiVersion = "v1",
  Kind = "Secret",
  Metadata = new V1ObjectMeta
  {
    Name = "secret",
    NamespaceProperty = "default"
  },
  Type = "Opaque",
  Immutable = true,
  Data = new Dictionary<string, byte[]>
  {
    ["key"] = Encoding.UTF8.GetBytes("value")
  },
  StringData = new Dictionary<string, string>
  {
    ["key"] = "value"
  }
};

string yaml = _serializer.Serialize(model);

await File.WriteAllTextAsync(outputPath, yaml, cancellationToken).ConfigureAwait(false);

Output:

---
apiVersion: v1
data:
  key:
  - 118
  - 97
  - 108
  - 117
  - 101
immutable: true
kind: Secret
metadata:
  name: secret
  namespaceProperty: default
stringData:
  key: value
type: Opaque

The sortAlphabetically bool did not have an effect in the other library.

Our library typically serializes in the order of the properties as they are declared in the class by default. But there's no guarantees. It's really in whatever order they are returned in the reflection calls or dictionary/list.

It seems that is what is causing my issues. I will cook something up to solve it in my library, and if it is not too specific to my use case, I can consider doing a PR if you want :-)

[EdwardCooke]

Since you’re only caring about keeping those few at the top then you might be able to do something with a custom type inspector. Might be easier. The type inspector returns the properties and fields to be serialiazed and it is done in the order returned. I can probably put a basic example together if you want.

[devantler]

Its fine, I like the challenge. But I would like to ask if I get stuck? 😊

[EdwardCooke]

Got bored, here you go. This will take all objects under the k8s.Models namespace and output ApiVersion, Kind, and Metadataat the top, with a specific example ofSecretoutputting the order ofTypeandData`. Other properties and classes are handled through normal means.

using k8s.Models;
using System.Diagnostics.CodeAnalysis;
using YamlDotNet.Serialization;

var model = new V1Secret
{
    ApiVersion = "v1",
    Kind = "Secret",
    Metadata = new V1ObjectMeta
    {
        Name = "secret",
        NamespaceProperty = "default"
    },
    Type = "Opaque",
    Data = new Dictionary<string, byte[]>
    {
    }
};

var serializer = new SerializerBuilder()
    .WithTypeInspector(inner=> new KubernetesTypeInspector(inner))
    .ConfigureDefaultValuesHandling(DefaultValuesHandling.OmitNull)
    .Build();

var yaml = serializer.Serialize(model);
Console.WriteLine(yaml);
Console.WriteLine("Proof of non kubernetes object still working");

yaml = serializer.Serialize(new NonK8sType { Name = "name", Namespace = "namespace" });
Console.WriteLine(yaml);

class NonK8sType
{
    public string Name { get; set; }
    public string Namespace { get; set; }
}

class KubernetesTypeInspector : ITypeInspector
{
    public ITypeInspector Inner { get; }

    public KubernetesTypeInspector(ITypeInspector inner)
    {
        Inner = inner;
    }

    public string GetEnumName(Type enumType, string name) => Inner.GetEnumName(enumType, name);

    public string GetEnumValue(object enumValue) => Inner.GetEnumValue(enumValue);

    public IEnumerable<IPropertyDescriptor> GetProperties(Type type, object? container)
    {
        var properties = Inner.GetProperties(type, container);
        var skipped = new List<string>();
        if (type.FullName?.StartsWith("k8s.Models") ?? true)
        {
            var apiVersion = properties.SingleOrDefault(p => p.Name == "ApiVersion");
            if (apiVersion != null)
            {
                skipped.Add("ApiVersion");
                yield return apiVersion;
            }
            var kind = properties.SingleOrDefault(p => p.Name == "Kind");
            if (kind != null)
            {
                skipped.Add("Kind");
                yield return kind;
            }
            var metadata = properties.SingleOrDefault(p => p.Name == "Metadata");
            if (metadata != null)
            {
                skipped.Add("Metadata");
                yield return metadata;
            }

            //special case for secrets
            if (type.FullName == typeof(V1Secret).FullName)
            {
                // do type, then data
                var typeProperty = properties.SingleOrDefault(p => p.Name == "Type");
                if (typeProperty != null)
                {
                    skipped.Add("Type");
                    yield return typeProperty;
                }
                var data = properties.SingleOrDefault(p => p.Name == "Data");
                if (data != null)
                {
                    skipped.Add("Data");
                    yield return data;
                }
            }
        }

        foreach (var property in properties)
        {
            if (!skipped.Contains(property.Name))
            {
                yield return property;
            }
        }
    }

    public IPropertyDescriptor GetProperty(Type type, object? container, string name, [MaybeNullWhen(true)] bool ignoreUnmatched, bool caseInsensitivePropertyMatching) => Inner.GetProperty(type, container, name, ignoreUnmatched, caseInsensitivePropertyMatching);
}

Outputs:

ApiVersion: v1
Kind: Secret
Metadata:
  Name: secret
  NamespaceProperty: default
Type: Opaque
Data: {}

Proof of non kubernetes object still working
Name: name
Namespace: namespace

[devantler]

Thanks man, do you want a shoutout on the devantler/dotnet-kubernetes-generator project? I will implement the inspector in the BaseKubernetesGenerator in Devantler.KubernetesGenerator.Core :-)

[EdwardCooke]

On a note on the above example, if you reverse Name and Namespace property in the NonK8sType you'll notice that the output is reversed for that object. That's what I was trying to explain when I was saying that the order of the properties/fields in the class definition will determine the output order of the yaml.

[devantler]

Ahh like that, so what I am probably seeing on the k8s projects models is that the classes properties are mixed around. As I have not seen the issue on the Kubernetes generators that use my own models where apiVersion and kind is at the top. Thanks for the example!

[devantler]

Thanks for the inspiration, I made a few changes to make it a bit more concise and generic:

using System.Diagnostics.CodeAnalysis;
using YamlDotNet.Serialization;

namespace Devantler.KubernetesGenerator.Core;

/// <summary>
/// A type inspector that prioritizes properties of Kubernetes resources, so they come first, and so they are ordered according to conventions.
/// Thanks to @EdwardCooke for the initial implementation.
/// </summary>
/// <param name="inner"></param>
sealed class KubernetesTypeInspector(ITypeInspector inner) : ITypeInspector
{
  public ITypeInspector Inner { get; } = inner;

  public string GetEnumName(Type enumType, string name) => Inner.GetEnumName(enumType, name);

  public string GetEnumValue(object enumValue) => Inner.GetEnumValue(enumValue);

  public IEnumerable<IPropertyDescriptor> GetProperties(Type systemType, object? container)
  {
    var properties = Inner.GetProperties(systemType, container).ToList();
    var skipped = new List<string>();
    if (systemType.FullName?.StartsWith("k8s.Models", StringComparison.OrdinalIgnoreCase) ?? true)
    {
      foreach (string? propName in new[] { "apiVersion", "kind", "metadata", "type" })
      {
        var property = properties.SingleOrDefault(p => p.Name == propName);
        if (property != null)
        {
          skipped.Add(propName);
          yield return property;
        }
      }
    }

    foreach (var property in properties)
    {
      if (!skipped.Contains(property.Name))
      {
        yield return property;
      }
    }
  }

  public IPropertyDescriptor GetProperty(Type type, object? container, string name, [MaybeNullWhen(true)] bool ignoreUnmatched, bool caseInsensitivePropertyMatching) => Inner.GetProperty(type, container, name, ignoreUnmatched, caseInsensitivePropertyMatching);
}

I will add you as a contributor on the next release of devantler/dotnet-kubernetes-generator

Still need to do snapshot unit tests for about 20-30 k8s models yikes. I hope I can get it done tommorrow, but its boring work.. :-)

[EdwardCooke]

Depending on what you use for your test framework, if you used xunit, you could do one test with theories where you pass in the type. Just a bunch of attributes and one reused test code. We do it in this library and I’ve done it in many many others. If I recall latest has the same thing but it’s been years since I’ve used it. Nunit also has the same type of thing.

[devantler]

Thank's I have used Theories quite a bit before, but for dotnet-kubernetes-generators, it does not fit my structure.

I decided to do one class per generated type, and I prefer to create a unit test class for each src class method. This way I have a test class with multiple tests for each and every method in my src code. This makes it easy for me to find and extend code coverage for methods that lack it, while also avoiding having enormous test classes for the src classes with many methods.

So in short theories won't fit well for this specific project, as each test class only has one or two test methods.

See here: dotnet-kubernetes-generators/Devantler.KubernetesGenerator.Native.Tests/ClusterTests

[devantler]

It is a bit more verbose in some cases, but I feel that it adds a lot to readability and maintainability when a project is more mature :-) Its the initial investment that is a pita, as I have to add sooo many classes upfront.

[EdwardCooke]

YamlDotNet/YamlDotNet.Test/Serialization/DateOnlyConverterTests.cs

Line 168 in 485daaa

[InlineData("d", "01/11/2017")]