[Bug]: Custom User Mappings not being applied during migration

[LudekStipal]

Version

• I confirm that I am using the latest version

Source Version

Azure DevOps Server 2022

Target Version

Azure DevOps Service

Relevant configuration

No response

Relevant log output

No response

What happened?

@satano - thank you for your great work fixing User export and loading the Mapping file (#2085). I believe there remains one issue during the User mappings in Migration, and I would appreciate your support here: we need to move 300+ Projects from OnPremise to Cloud, and having the correct Identities assigned to Users is very important.

In the current version 16.0.8, the Users are read correctly from AD (DevOps Server) and userExport.json file written by TfsExportUsersForMappingProcessor, creating entries:

"FriendlyName": null,

After I add the Mappings to correct DevOps Services Identities, the TfsWorkItemMigrationProcessor is able to load this file and use Mappings in MapUserIdentityField function (MigrationTools.Clients.TfsObjectModel\Tools\TfsUserMappingTool.cs).

However, the migrated (newly created / updated) Work Items in DevOps Services still have the old FriendlyName (no actual Identity) in AssignedTo and other fields.

I was able to track this down to a location in source code where it happens, but I am not sure what the correct "fix" is.

PopulateWorkItem Function (MigrationTools.Clients.TfsObjectModel\Processors\TfsWorkItemMigrationProcessor.cs) receives oldWorkItemData and newWorkItemData, and converts them to oldWorkItem and newWorkitem:

private void PopulateWorkItem(WorkItemData oldWorkItemData, WorkItemData newWorkItemData, string destType)
{
  var oldWorkItem = oldWorkItemData.ToWorkItem();
  var newWorkItem = newWorkItemData.ToWorkItem();

User mapping is performed on the convered oldWorkItem:

foreach (Field f in oldWorkItem.Fields)
{
  CommonTools.UserMapping.MapUserIdentityField(f);

However, once the Values are copied from oldWorkItem into newWorkItem, there is this condition:

switch (f.FieldDefinition.FieldType)
{
    case FieldType.String:
        CommonTools.StringManipulator.ProcessorExecutionWithFieldItem(null, oldWorkItemData.Fields[f.ReferenceName]);
        newWorkItem.Fields[f.ReferenceName].Value = oldWorkItemData.Fields[f.ReferenceName].Value;
        break;
    default:
        newWorkItem.Fields[f.ReferenceName].Value = oldWorkItem.Fields[f.ReferenceName].Value;
        break;
}

And as IdentityFields are of type String, the first case is used, which reads the value from oldWorkItemData - the original, unconverted and unmapped entity.

When I commented out the "case FieldType.String" (so that each field used the "default" condition), the mappings were applied correctly, and I see the correct Identity with picture in Target work item.

So the solution seems to be forcing the "case FieldType.String:" branch to also use updated / mapped values form oldWorkItem, not the oldWorkItemData...

Debug in Visual Studio

• Visual Studio Debug

[satano]

This is because of StringManipulator tool. It has been discussed recently in #2525. It's because the default implementation of StringManipulator modifies the strings in a way, that basically it removes all non-ascii characters. The easiest way is to disable string manipulator completely and enable it with custom regex if you will experience some troubles during migration. In CommonTools configure:

"StringManipulatorTool": {
    "Enabled": false,
    "Manipulators": [
        {
            "$type": "RegexStringManipulator",
            "Description": "Remove invalid characters from the string",
            "Enabled": "True",
            "Pattern": "[^\\S \\n\\r\\t]+",
            "Replacement": ""
        }
    ],
    "MaxStringLength": "1000000"
}

Regex in this cofiguration is already customized to preserve local characters (in fact this will preserve any word character).

[LudekStipal]

Hi @satano , thanks for the tip, I tried disabling the StringManipulator with the latest source code, however the issue seems to persist - as even if the StringManipulator does nothing, the next line always updates the "new" value with improper "old" one.

Since the "case FieldType.String" condition still exists:

the newWorkItem is always "updated" with unmapped "oldWorkItemData", not the replaced (correct) "oldWorkItem" as would be the case in "default" case (instead of case FieldType.String):