Revamp Documentation (1) for v16 (#2313)

We need to update a bunch of things to enable better understanding and
close off all the reactors:

- [x] Update readme
- [x] Update index
- [x] Add Installation
- [x] Add Version Control
- [x] Permissions

Gemfile

@@ -3,7 +3,7 @@
 source "https://rubygems.org"
 
 # gem "rails"
-gem 'jekyll-relative-links'
+gem 'jekyll-relative-links' 
 gem 'jekyll-seo-tag'
 gem 'jekyll-sitemap'
 gem 'jekyll-toc'
@@ -14,4 +14,5 @@ gem 'jekyll-optional-front-matter'
 gem 'jekyll-redirect-from'
 gem 'rouge'
 gem 'kramdown'
-gem 'pygments.rb'
+gem 'pygments.rb'
+gem 'wdm'

Gemfile.lock

@@ -111,6 +111,7 @@ GEM
       unicode-display_width (>= 1.1.1, < 3)
     unicode-display_width (2.5.0)
     uri (0.13.0)
+    wdm (0.2.0)
     webrick (1.8.1)
 
 PLATFORMS
@@ -130,6 +131,7 @@ DEPENDENCIES
   kramdown
   pygments.rb
   rouge
+  wdm
 
 BUNDLED WITH
    2.5.17

README.md

@@ -1,6 +1,6 @@
-![azure-devops-migration-tools](https://socialify.git.ci/nkdAgility/azure-devops-migration-tools/image?description=1&descriptionEditable=Azure%20DevOps%20Migration%20Tools%20allow%20you%20to%20migrate%20Work%20Items%2C%20Plans%20%26%20Suits%2C%20%26%20Pipelines%20from%20one%20Azure%20DevOps%20%2F%20TFS%20to%20another!&font=Inter&forks=1&name=1&owner=1&pattern=Signal&stargazers=1&theme=Light)
-![GitHub release](https://img.shields.io/github/release/nkdAgility/vsts-sync-migration.svg)
-![GitHub pre-release](https://img.shields.io/github/v/release/nkdagility/vsts-sync-migration?include_prereleases)
+![Azure DevOps Migration Tools from Naked Agility with Martin Hinshelwood](https://github.com/user-attachments/assets/997cc49f-cbe9-4f22-a8e1-49b529d0dff0)
+![GitHub release](https://img.shields.io/github/v/release/nkdAgility/azure-devops-migration-tools)
+![GitHub pre-release](https://img.shields.io/github/v/release/nkdAgility/azure-devops-migration-tools?include_prereleases)
 
 [![Build Status](https://dev.azure.com/nkdagility/AzureDevOps-Tools/_apis/build/status%2FMigrationTools-CIv2?branchName=main)](https://dev.azure.com/nkdagility/AzureDevOps-Tools/_build/latest?definitionId=115&branchName=main)
 ![Azure DevOps tests](https://img.shields.io/azure-devops/tests/nkdagility/AzureDevOps-Tools/115?compact_message&style=plastic&logo=azuredevops&label=Tests)
@@ -19,84 +19,78 @@ Created and maintained by [Martin Hinshelwood](https://www.linkedin.com/in/marti
 
 # Azure DevOps Migration Tools
 
-The Azure DevOps Migration Tools allow you to bulk edit and migrate data between Team Projects on both Microsoft Team Foundation Server (TFS) and Azure DevOps Services. Take a look at the  [documentation](https://nkdagility.com/docs/azure-devops-migration-tools/) to find out how. This project is published as [code on GitHub](https://github.com/nkdAgility/azure-devops-migration-tools/) as well as a Winget package a `nkdAgility.AzureDevOpsMigrationTools`.
+The Azure DevOps Migration Tools allow you to bulk edit and migrate data between Team Projects on both Microsoft Team Foundation Server (TFS) and Azure DevOps Services. Take a look at the  [documentation](https://nkdagility.com/learn/azure-devops-migration-tools/) to find out how. This project is published as [code on GitHub](https://github.com/nkdAgility/azure-devops-migration-tools/) as well as a Winget package a `nkdAgility.AzureDevOpsMigrationTools`.
 
 **Ask Questions on Github: https://github.com/nkdAgility/azure-devops-migration-tools/discussions**
 
-## Some Data from the last 30 days (as of 05/03/2024)
-
-| Category  | Metric | Notes |
-| ------------- | ------------- | ------------- |
-| Work Items | **1m** | A single Work Item may have many revisions that we need to migrate |
-| Work Item Revisions | **23m** | A single Work Item may have many revisions that we need to migrate |
-| RelatedLinkCount | **11m** | Each work item may have many links or none. |
-| Git Commit Links  | **1.3m** |  |
-| Attachments | **1.2m**  | Total number of attachments migrated |
-| Test Suits | 52k | total suits migrated | 
-| Test Cases Mapped | **1.4m** | Total test cases mapped into Suits |
-| Migration Run Ave  | **14 minutes** | Includes dry-runs as well.  |
-| Migration Run Total   |  **19bn Seconds** | Thats **316m hours** or **13m days** of run time in the last 30 days. |
-| Average Work item Migration Time  | **22s** | Work Item (includes all revisions, links, and attachments for the work item) |
-
-Exceptions shipped to Application Insights and [Elmah.io](https://elmah.io) for analysis and improvement.
-
+## Compatability
 
+These tools run on Windows and support connecting to Team Foundation Server 2013+, Azure DevOps Server, & Azure DevOps Services. They support both hosted and on-premise instances and can move data between any two.
 
-## What can you do with this tool?
+- Supports all versions of TFS 2013+ and all versions of Azure DevOps.
+- You can migrate from any TFS/Azure DevOps source to any TFS/Azure DevOps target.
 
-- Migrate `Work Items`, `Test Plans & Suites`, `Teams`, `Shared Queries`, `Pipelines`, and `Processes` between different `Team Projects` or `Organizations`.
-- Perform bulk edits on `Work Items` across an entire `Project`.
+## What do you get?
 
-**Important:** This tool is intended for experienced users familiar with TFS/Azure DevOps object models and debugging in Visual Studio. It was developed by over 30 contributors from the Azure DevOps community to handle various scenarios and edge cases. 
+- *Move* Work Items, Test Plans & Suits, and Pipelines between projects, collections, and even organizations.
+- *Merge* multiple projects into a single project even from different organizations.
+- *Split* one project into several projects even between projects, collections, and even organizations.
+- *Change* Process process from Agile to Scrum or any other template.
+- *Bulk edit* Work Items.
 
-**Support Options:** Community support is available on [GitHub](https://github.com/nkdAgility/azure-devops-migration-tools/discussions). For paid support, consider our [Azure DevOps Migration Services](https://nkdagility.com/capabilities/azure-devops-migration-services/).
+## What does this tool do?
 
-### What versions of Azure DevOps & TFS do you support?
+For the most part we support moving data between ((Azure DevOps Server | Team Foundation Server | Azure DevOps Services) <=> (Azure DevOps Server | Team Foundation Server | Azure DevOps Services)) for any version greater than 2013. 
 
-- Supports all versions of TFS 2013+ and all versions of Azure DevOps.
-- You can migrate from any TFS/Azure DevOps source to any TFS/Azure DevOps target.
+- `Work Items` (including links and attachments) with custom mappings for fields and types
+	- Copy Work Items between locations with history
+	- Bulk Edit in place of Work Items (Great for cleaning up data, process template changes)
+	- Optionaly includes `Teams`, `Shared Queries`
+- `Test Plans & Suites` 
+	- Copy Test Plans & Suites between locations
+	- Includes `Configurations`, `Shared Steps`, `Shared Parameters`
+- `Pipelines`
+	- Copy Pipelines between locations
+	- excludes XAML & Classic Builds & Release
+- `Processes`
+	- Copy Processes between locations
 
-### Typical Uses of this tool
+**Note**: 'Locations' includes `Projects`, `Collections`, `Organizations`
 
-- Merge multiple projects into a single project.
-- Split one project into several projects.
-- Change Process Templates.
-- Bulk edit Work Items.
-- Migrate Test Suites & Test Plans.
-- Migrate Builds & Pipelines (new).
-- Migrate between different language versions of TFS/Azure DevOps (new in v9.0).
-- Migrate Processes.
+**Important:** This tool is intended for experienced users familiar with TFS/Azure DevOps object models and debugging in Visual Studio. It was developed by 100+ contributors from the Azure DevOps community to handle various scenarios and edge cases. _Not all cases are supported_.
 
-**Note:** If you're migrating an entire collection to Azure DevOps Services, consider using Microsoft's [Azure DevOps Migration Service](https://azure.microsoft.com/services/devops/migrate/). If you need to change the Process Template, do so before moving to Azure DevOps Services.
+**Support Options:** Community support is available on [GitHub](https://github.com/nkdAgility/azure-devops-migration-tools/discussions). For paid support, consider our [Azure DevOps Migration Services](https://nkdagility.com/capabilities/azure-devops-migration-services/).
 
 ## Quick Links
 
- - [Video Overview](https://www.youtube.com/watch?v=RCJsST0xBCE)
- - [Getting Started](https://nkdagility.com/learn/azure-devops-migration-tools/getting-started/)
- - [Installing](https://nkdagility.com/learn/azure-devops-migration-tools/installation/)
-
- - [Documentation](https://nkdagility.com/learn/azure-devops-migration-tools/)
- - [Questions on Usage](https://github.com/nkdAgility/azure-devops-migration-tools/discussions)
-
-## Minimum Permission Requirements
+- [Documenation](https://nkdagility.com/docs/azure-devops-migration-tools/)
+- [Installation](https://nkdagility.com/learn/azure-devops-migration-tools/installation/)
+- [Permissions](https://nkdagility.com/learn/azure-devops-migration-tools/permissions/)
+- [Getting Started](https://nkdagility.com/learn/azure-devops-migration-tools/getting-started/)
+- [Configuration Reference](https://nkdagility.com/learn/azure-devops-migration-tools/Reference/)
+- [Community Support](https://github.com/nkdAgility/azure-devops-migration-tools/discussions)
+- [Commercial Support](https://nkdagility.com/capabilities/azure-devops-migration-services/)
 
-At this time the documented minimum required permissions for running the tools are:
+The documentation for the preview is on [Preview](https://nkdagility.com/docs/azure-devops-migration-tools/preview/)]
 
-- Account in both the source and target projects with "Project Collection Administrator" rights
-- PAT with "full access" for both the Source and the Target
-
-Note: I have been informed by the Azure DevOps product team information that ObjectModel API only works with full scoped PATs, so it won't work with any PAT that has specific scopes. 
+## Some Data from the last 30 days (as of 05/03/2024)
 
-### Advanced Unsupported Permission Options
+| Category  | Metric | Notes |
+| ------------- | ------------- | ------------- |
+| Work Items | **1m** | A single Work Item may have many revisions that we need to migrate |
+| Work Item Revisions | **23m** | A single Work Item may have many revisions that we need to migrate |
+| RelatedLinkCount | **11m** | Each work item may have many links or none. |
+| Git Commit Links  | **1.3m** |  |
+| Attachments | **1.2m**  | Total number of attachments migrated |
+| Test Suits | 52k | total suits migrated | 
+| Test Cases Mapped | **1.4m** | Total test cases mapped into Suits |
+| Migration Run Ave  | **14 minutes** | Includes dry-runs as well.  |
+| Migration Run Total   |  **19bn Seconds** | Thats **316m hours** or **13m days** of run time in the last 30 days. |
+| Average Work item Migration Time  | **22s** | Work Item (includes all revisions, links, and attachments for the work item) |
 
-We have seen that the tools may work with less permissions however the following has not been full tested and is not currently supported:
+Exceptions shipped to Application Insights and [Elmah.io](https://elmah.io) for analysis and improvement.
 
-- Project and Team (Read, write, & manage)
-- Work Items (Read, Write & Manage)
-- Identity (Read & Manage)
-- Security (Manage)
 
-If you do try this out then please let us know how you get on!
 
 ## Advanced tools
 
@@ -126,6 +120,7 @@ We use these tools with our customers, and for fun, to do real world migrations
 
 ## Change Log
 
+- [v16.0.0](https://github.com/nkdAgility/azure-devops-migration-tools/releases/tag/v16.0.0) - Enhanced configuration with support for command line, and environmental variable overrides.
 - [v15.1.7](https://github.com/nkdAgility/azure-devops-migration-tools/releases/tag/v15.1.7) - The latest release brings a host of enhancements and fixes designed to improve user experience and configuration options. Noteworthy features include a new GitHub Actions workflow for automatic updates to pull request titles, enhanced management of area and iteration paths using regex mapping, and a more streamlined query format for migration configurations. Users can now enjoy greater flexibility in configuring attachment processing, including options for export paths and size limits. Additionally, updates to authentication methods and improved logging for user retrieval processes have been implemented. The release also addresses various bugs and makes adjustments to enhance overall functionality.
 - [v14.4.7](https://github.com/nkdAgility/azure-devops-migration-tools/releases/tag/v14.4.7) - The latest major release brings a host of user-focused enhancements and improvements. Key changes include the adoption of Winget as the primary installation method, making it easier for users to get started. The main executable has been renamed to `devopsmigration.exe`, and new configuration options enhance customization capabilities, including parallel builds and test case timeouts. The command for initializing configurations has been updated for greater flexibility, and logging improvements provide better insights during migration operations. Subsequent updates have refined version detection, improved command line arguments, and introduced new configuration files to prevent blank issues. Enhanced logging and error handling further improve user experience, while package upgrades and better handling of specific fields streamline migration processes. Overall, these updates aim to enhance functionality, usability, and reliability for users.
 - [v13.2.1](https://github.com/nkdAgility/azure-devops-migration-tools/releases/tag/v13.2.1) - The latest updates bring a range of enhancements and new features aimed at improving user experience and functionality. A key addition is the `WorkItemMigrationContext` processor, which facilitates the migration of work items, including their history and attachments, between Azure DevOps instances. Users will find clearer documentation and a new configuration file to simplify work item type and field mappings. The introduction of the `ExportUsersForMapping` feature allows for easy JSON file exports for field mapping, while security is bolstered with an updated authentication mode. Users can now disable telemetry collection during migration, and various improvements have been made to migration behavior and configuration settings, enhancing the overall robustness and integrity of the migration tools.

_config.yml

@@ -16,6 +16,8 @@ collections_dir: collections
 collections:
  reference:
    output: true
+ howto:
+    output: true
 
 exclude:
  - "**/*template*.md"

appsettings.json

@@ -420,6 +420,13 @@
         "MaxGracefulFailures": 0,
         "SkipRevisionWithInvalidIterationPath": false,
         "SkipRevisionWithInvalidAreaPath": false
+      },
+      "ExportUsersForMappingProcessor": {
+        "Enabled": true,
+        "WIQLQuery": "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = @TeamProject AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan','Shared Steps','Shared Parameter','Feedback Request') ORDER BY [System.ChangedDate] desc",
+        "OnlyListUsersInWorkItems": true,
+        "SourceName": "Source",
+        "TargetName": "Target"
       }
     }
   }

configuration.json

@@ -10,6 +10,7 @@
         "Collection": "https://dev.azure.com/nkdagility-preview/",
         "Project": "migrationSource1",
         "AllowCrossProjectLinking": false,
+        "ReflectedWorkItemIDFieldName": "Custom.ReflectedWorkItemId",
         "Authentication": {
           "AuthenticationMode": "AccessToken",
           "AccessToken": "",