BinaryFormatter Migration Guide

docs/fundamentals/toc.yml

@@ -861,6 +861,18 @@ items:
                     href: ../standard/serialization/xml-schema-definition-tool-xsd-exe.md
           - name: Binary serialization
             items:
+              - name: BinaryFormatter migration guide
+                items:
+                  - name: Overview
+                    href: ../standard/serialization/binaryformatter-migration-guide/overview.md
+                  - name: Choosing a serializer
+                    href: ../standard/serialization/binaryformatter-migration-guide/choosing-a-serializer.md
+                  - name: Reading NRBF payloads
+                    href: ../standard/serialization/binaryformatter-migration-guide/reading-nrbf-payloads.md
+                  - name: Compatibility package
+                    href: ../standard/serialization/binaryformatter-migration-guide/compatibility-package.md
+                  - name: BinaryFormatter functionality reference
+                    href: ../standard/serialization/binaryformatter-migration-guide/functionality-reference.md
               - name: BinaryFormatter security guide
                 href: ../standard/serialization/binaryformatter-security-guide.md
               - name: BinaryFormatter event source

docs/standard/serialization/binaryformatter-migration-guide/choosing-a-serializer.md

@@ -0,0 +1,98 @@
+---
+title: "BinaryFormatter Migration Guide: Choosing a serializer"
+description: "Compare the capabilities and trade-offs of serializers to choose a replacement for BinaryFormatter."
+ms.date: 5/31/2024
+no-loc: [BinaryFormatter, Serialization]
+dev_langs:
+  - CSharp
+  - VB
+helpviewer_keywords:
+  - "BinaryFormatter"
+  - "serializing objects"
+  - "serialization"
+  - "objects, serializing"
+---
+
+# Choosing a serializer
+
+Choosing a serializer format boils down to two questions:
+
+- Is compact binary representation important for your scenario? If so, you need to switch to a different binary serializer. If not, you can consider using JSON and XML serialization.
+- Can you modify the types that are being serialized by annotating them with attributes, adding new constructors, making the types public and changing fields to properties? If not, using the modern serializers might require more work (like implementing custom converters or resolvers).
+
+| Feature                                        | BinaryFormatter | DataContractSerializer | System.Text.Json        | MessagePack              |
+|------------------------------------------------|-----------------|------------------------|-------------------------|--------------------------|
+| Compact binary representation                  | ✔️              | ❌                      | ❌                      |  ✔️                       |
+| Human readable                                 | ❌️              | ✔                      | ✔                      |  ❌                       |
+| Performance                                    | ❌️              | ❌                      | ✔️                      |  ✔️✔️                     |
+| `[Serializable]` support                       | ✔️              | ✔️                      | ❌                      |  ❌                       |
+| Serializing public types                       | ✔️              | ✔️                      | ✔️                      |  ✔️                       |
+| Serializing non-public types                   | ✔️              | ✔️                      | ✔️                      |  ✔️ (resolver required)   |
+| Serializing fields                             | ✔️              | ✔️                      | ✔️ (opt in)             |  ✔️ (attribute required)  |
+| Serializing properties                         | ✔️<sup>*</sup>  | ✔️                      | ✔️                      |  ✔️ (attribute required)  |
+| Deserializing readonly members                 | ✔️              | ✔️                      | ✔️ (attribute required) |  ✔️                       |
+| Polymorphic type hierarchy                     | ✔️              | ✔️                      | ✔️ (attribute required) |  ✔️ (attribute required)  |
+
+### XML
+
+The .NET base class libraries provide two XML serializers [XmlSerializer](../introducing-xml-serialization.md) and [DataContractSerializer](../../../fundamentals/runtime-libraries/system-runtime-serialization-datacontractserializer.md). There are some subtle differences between these two, but for the purpose of the migration we are going to focus only on `DataContractSerializer`. Why? Because it **fully supports the serialization programming model that was used by the `BinaryFormatter`**. So all the types that are already marked as `[Serializable]` and/or implement `ISerializable` can be serialized with `DataContractSerializer`. Where is the catch? Known types must be specified up-front (that is why it's secure). So you need to know them and be able to get the `Type` **even for private types**.
+
+```cs
+DataContractSerializer serializer = new(
+    type: input.GetType(),
+    knownTypes: new Type[]
+    {
+        typeof(MyType1),
+        typeof(MyType2)
+    });
+```
+
+It's not required to specify most popular collections or primitive types like `string` or `DateTime` (the serializer has it's own default allowlist), but there are exceptions like `DateTimeOffset`. You can read about the supported types in the [dedicated doc](../../../framework/wcf/feature-details/types-supported-by-the-data-contract-serializer.md).
+
+[Partial trust](../../../framework/wcf/feature-details/partial-trust.md) is a Full .NET Framework feature that was not ported to .NET (Core). If your code runs on a Full .NET Framework and uses this feature, please read about the [limitations](../../../framework/wcf/feature-details/types-supported-by-the-data-contract-serializer.md#limitations-of-using-certain-types-in-partial-trust-mode) that may apply to such scenario.
+
+### JSON
+
+[System.Text.Json](../system-text-json/overview.md) is strict by default and avoids any guessing or interpretation on the caller's behalf, emphasizing deterministic behavior. The library is intentionally designed this way for performance and security. From the migration perspective, it's crucial to know the following facts:
+
+- By default, **fields aren't serialized**, but they can be [included on demand](../system-text-json/fields.md), which is a must-have for types that use fields that are not exposed by properties. The simplest solution that does not require modifying the types is to use the global setting to include fields.
+
+```cs
+JsonSerializerOptions options = new()
+{
+    IncludeFields = true
+};
+```
+
+- By default, System.Text.Json **ignores private fields and properties**. You can enable use of a non-public accessor on a property by using the `[JsonInclude]` attribute. Including private fields requires some [non-trivial extra work](../system-text-json/custom-contracts.md#example-serialize-private-fields).
+- It **[can not deserialize readonly fields](/dotnet/api/system.text.json.jsonserializeroptions.ignorereadonlyfields?view#remarks)** or properties, but `[JsonConstructor]` attribute can be used to indicate that given constructor should be used to create instances of the type on deserialization. And obviously the constructor can set the readonly fields and properties.
+- It [supports serialization and deserialization of most built-in collections](../system-text-json/supported-collection-types.md). The exceptions:
+  - multi-dimensional arrays,
+  - `BitArray`,
+  - `LinkedList<T>`,
+  - `Dictionary<TKey, TValue>`, where `TKey` is not a primitive type,
+  - `BlockingCollection<T>` and `ConcurrentBag<T>`,
+  - most of the collections from [System.Collections.Specialized](../system-text-json/supported-collection-types.md#systemcollectionsspecialized-namespace) and [System.Collections.ObjectModel](../system-text-json/supported-collection-types.md#systemcollectionsobjectmodel-namespace) namespaces.
+- Under [certain condtions](../system-text-json/supported-collection-types.md#custom-collections-with-deserialization-support), it supports serialization and deserialization of custom generic collections.
+- Other types [without built-in support](../system-text-json/migrate-from-newtonsoft.md#types-without-built-in-support) are: `DataSet`, `DataTable`, `DBNull`, `TimeZoneInfo`, `Type`, `ValueTuple`. However, you can write a custom converter to support these types.
+- It [supports polymorphic type hierarchy serialization and deserialization](../system-text-json/polymorphism.md) that have been explicitly opted in via the `[JsonDerivedType]` attribute or via custom resolver.
+- The `[JsonIgnore]` attribute on a property causes the property to be omitted from the JSON during serialization.
+- To preserve references and handle circular references in System.Text.Json, set `JsonSerializerOptions.ReferenceHandler` to `Preserve`.
+- To override the default behavior you can [write custom converters](../system-text-json/converters-how-to.md).
+
+### Binary
+
+.NET Team is deprecating the `BinaryFormatter`, but at the same time we currently have no plans to implement a new binary serializer. It puts us in a situation, where we can't recommend any serializer that we own. But luckily for all of us, the .NET Open Source Ecosystem provides many great binary serializers. Some of them:
+
+[protobuf-net](https://github.com/protobuf-net/protobuf-net) is "a contract based serializer for .NET code, that happens to write data in the _protocol buffers_ serialization format engineered by Google". Because protobuf-net is a cross-language protocol, it has some subtle issues with .NET-specific concepts like [nulls and empty arrays](https://stackoverflow.com/questions/21631428/protobuf-net-deserializes-empty-collection-to-null-when-the-collection-is-a-prop/21632160#21632160). Since this document is about a migration from `BinaryFormatter`, we can safely assume that we don't need cross-language support and the complexity it brings. That is why in this particular scenario of binary serialization, we recommend MessagePack.
+
+MessagePack provides a highly efficient binary serialization format, resulting in smaller message sizes compared to JSON and XML. It's very [performant](https://github.com/MessagePack-CSharp/MessagePack-CSharp?tab=readme-ov-file#performance), ships with built-in support for LZ4 compression and a full set of general-purpose expressive data types:
+
+- By default, only public types are serializable. Private and internal structs and classes can be serialized only when `StandardResolverAllowPrivate.Options` is provided as an argument to `MessagePackSerializer.Serialize` and `MessagePackSerializer.Deserialize` methods.
+- MessagePack requires each serializable type to be annotated with `[MessagePackObject]` attribute. It's possible to avoid that by using the [ContractlessStandardResolver](https://github.com/MessagePack-CSharp/MessagePack-CSharp?tab=readme-ov-file#object-serialization), but it may cause issues with versioning in the future.
+- Every serializable non-static field and a property needs to be annotated with `[Key]` attribute. If you annotate the type with `[MessagePackObject(keyAsPropertyName: true)]` attribute, then members do not require explicit annotations. In such case, to ignore certain public members the `[IgnoreMember]` attribute needs to be used.
+- To serialize private members, use [DynamicObjectResolverAllowPrivate](https://github.com/MessagePack-CSharp/MessagePack-CSharp?tab=readme-ov-file#object-serialization).
+- `System.Runtime.Serialization` annotations can be used instead of MessagePack annotations. `[DataContract]` instead of`[MessagePackObject]`, `[DataMember]` instead of `[Key]` and `[IgnoreDataMember]` instead of `[IgnoreMember]`. It can be very useful if you want to avoid having dependency on MessagePack in the library that defines serializable types.
+- It supports readonly/immutable types and members. The serializer will try to use the public constructor with the best matched argument list. It can be specified in an explicit way by using `[SerializationConstructor]` attribute.
+- The serializer supports most frequently used built-in types and collections provided by the .NET base class libraries. You can find the full list in [official docs](https://github.com/MessagePack-CSharp/MessagePack-CSharp?tab=readme-ov-file#built-in-supported-types). It has [extension points](https://github.com/MessagePack-CSharp/MessagePack-CSharp?tab=readme-ov-file#extensions) that allow for customization.
+- The library provides [Typless API](https://github.com/MessagePack-CSharp/MessagePack-CSharp?tab=readme-ov-file#typeless) similar to `BinaryFormatter`, but it should not be used as it's not [secure](https://github.com/MessagePack-CSharp/MessagePack-CSharp?tab=readme-ov-file#security) and would defeat the purpose of migrating from `BinaryFormatter`.

docs/standard/serialization/binaryformatter-migration-guide/compatibility-package.md

@@ -0,0 +1,31 @@
+---
+title: "BinaryFormatter Migration Guide: Compatibility Package"
+description: "Using the BinaryFormatter compatibility package."
+ms.date: 5/31/2024
+no-loc: [BinaryFormatter, Serialization]
+dev_langs:
+  - CSharp
+  - VB
+helpviewer_keywords:
+  - "BinaryFormatter"
+  - "serializing objects"
+  - "serialization"
+  - "objects, serializing"
+---
+
+# Using the compatibility package
+
+All the .NET 9+ users who for some reason can not migrate away from `BinaryFormatter` can just install the [System.Runtime.Serialization.Formatters](https://www.nuget.org/packages/System.Runtime.Serialization.Formatters) NuGet package and set the `System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization` AppContext switch to `true`.
+
+```xml
+<PropertyGroup>
+  <TargetFramework>net9.0</TargetFramework>
+  <EnableUnsafeBinaryFormatterSerialization>true</EnableUnsafeBinaryFormatterSerialization>
+</PropertyGroup>
+
+<ItemGroup>
+  <PackageReference Include="System.Runtime.Serialization.Formatters" Version="9.0.0-*" />
+</ItemGroup>
+```
+
+**It's unsafe and not recommended**.

docs/standard/serialization/binaryformatter-migration-guide/functionality-reference.md

@@ -0,0 +1,88 @@
+---
+title: "BinaryFormatter Migration Guide: Functionality Reference"
+description: "A reference for BinaryFormatter's functionality that may need to be considered during migrations."
+ms.date: 5/31/2024
+no-loc: [BinaryFormatter, Serialization, WinForms]
+dev_langs:
+  - CSharp
+  - VB
+helpviewer_keywords:
+  - "BinaryFormatter"
+  - "serializing objects"
+  - "serialization"
+  - "objects, serializing"
+---
+
+# BinaryFormatter functionality reference
+
+The `BinaryFormatter` was first introduced with the initial release of the .NET Framework in 2002. It's very likely that engineers who are assigned to the migration task may not have the necessary knowledge or experience to work with the `BinaryFormatter`. This can lead to errors, delays, or failures.
+Therefore, it is crucial for engineers to understand how the old technology works before they start the migration.
+
+`BinaryFormatter` can serialize any `object` that is annotated with `[Serializable]` attribute or implements `ISerializable` interface.
+
+### Member names
+
+In most common scenario the type is just annotated with `[Serializable]` attribute and the serializer uses reflection to serialize **all fields** (both public and not) that are not annotated with `[NonSerialized]` attribute. In case of C# auto properties, they are backed by fields generated by the C# compiler, so the names of the serialized fields are compiler-generated (and not very human friendly to say politely).
+
+Let's use one of the C# decompilers like [https://sharplab.io/](https://sharplab.io/) or [ILSpy](https://github.com/icsharpcode/ILSpy) to see what field gets generated for following simple property.
+
+```cs
+[Serializable]
+internal class PropertySample
+{
+    public string Name { get; set; }
+}
+```
+
+Is translated by the C# compiler to:
+
+```cs
+[Serializable]
+internal class PropertySample
+{
+    private string <Name>k__BackingField;
+
+    public string Name
+    {
+        get
+        {
+            return <Name>k__BackingField;
+        }
+        set
+        {
+            <Name>k__BackingField = value;
+        }
+    }
+}
+```
+
+And in this case `<Name>k__BackingField` is **the name of the member that `BinaryFormatter` uses in the serialized payload**. It's impossible to use `nameof` or any other C# operator to get this name.
+
+[ISerializable](/dotnet/api/system.runtime.serialization.iserializable) interface comes with [GetObjectData(SerializationInfo info, StreamingContext context)](/dotnet/api/system.runtime.serialization.iserializable.getobjectdata) method that allows the users to control the names, by using one of the [SerializationInfo.AddValue](/dotnet/api/system.runtime.serialization.serializationinfo.addvalue) methods.
+
+```cs
+// please note lack of any special attribute
+public void GetObjectData(SerializationInfo info, StreamingContext context)
+{
+    info.AddValue("Name", this.Name);
+}
+```
+
+If such customization has been applied, the information needs to be provided during deserialization as well. It's possible by using the **serialization constructor** where all values are read from `SerializationInfo` by using one of the `Get` methods it provides.
+
+```cs
+private PropertySample(SerializationInfo info, StreamingContext context)
+{
+    this.Name = info.GetString("Name");
+}
+```
+
+**Note:** The `nameof` operator was not used here on purpose, as the payload can be persisted and the property can get renamed at some point of time. So even if it gets renamed (let's say to `FirstName` because we decided to also introduce `LastName` property), to remain backward compatibility the serialization should still use the old name that could have been persisted somewhere.
+
+### Serialization Binder
+
+On top of that, it's recommended to use [SerializationBinder](/dotnet/api/system.runtime.serialization.serializationbinder) to control class loading and mandate what class to load and therefore minimize security vulnerabilities (so only allowed types get loaded, even if the attacker modifies the payload to deserialize and load something else).
+
+Using this type requires inheriting from it and overriding the [Type BindToType(string assemblyName, string typeName)](/dotnet/api/system.runtime.serialization.serializationbinder.bindtotype#system-runtime-serialization-serializationbinder-bindtotype(system-string-system-string)) method.
+
+If given codebase uses custom type that derives from `SerializationBinder` it's likely that the list of serializable types is a **closed set**. If not, finding the list of all types that can get serialized and deserialized is going to require studying all the usages of `BinaryFormatter` in source code. **Knowing that list is crucial to determining how to move further with the migration**.