WinGet Source COM Api #4813

Madhusudhan-MSFT · 2024-09-16T18:52:46Z

WinGet Source COM API Support

This update includes the following changes:

Implementation of WinGet COM Package Catalog Management APIs:
- PackageManager.AddPackageCatalogAsync: Allows adding a new Package Catalog repository to the Windows Package Manager access list. Administrative rights are required to execute this API..
- PackageManager.RemovePackageCatalogAsync: Enables the removal of an existing Package Catalog from the Windows Package Manager access list. Administrative rights are required to execute this API.
  - By default, the 'PreserveData' field is set to false, removing both the PackageCatalog registration data and system artifacts.
  - When 'PreserveData' is set to true, only the PackageCatalog registration data is removed, while system artifacts remain.
- PackageCatalogReference.RefreshPackageCatalogAsync: Allows updating an existing Package Catalog repository.
Necessary C# WinRT Projection to invoke the above API calls from the test classes.
PackageCatalogInterop E2E Inproc and OutOfproc Test Cases to validate:
- AddPackageCatalogAsync
- RemovePackageCatalogAsync
- RefreshPackageCatalogAsync
Added a CallbackDispatcherSink struct that handles and dispatches progress callbacks for the scenario described above, ensuring progress is reported to the caller.
Updated Package.appxmanifest and Microsoft.Management.Deployment.InProc.dll.manifest to include the necessary COM RuntimeClass CLSIDs related to the aforementioned APIs

How Validated:

Compiled AppInstaller.sln
Deployed AppInstallerCLIPackage
Copied Microsoft.Management.Deployment.winmd next to dot.exe process that runs test cases
Executed InProc tests locally and verified all tests pass.

related: #4170

I have signed the Contributor License Agreement.
This pull request is related to an issue.

Microsoft Reviewers: Open in CodeFlow

- Update IDL for new runtime classes and methods. - Add files and classes for package catalog operations (add, remove, reset, update). - Update `PackageManager` with async methods for catalog operations. [Compiles successfully]

Introduced new interfaces in the `Microsoft.Management.Deployment` namespace within the `PackageManager.idl` file. Added interfaces for handling collections of `AddPackageCatalogOptions`, `AddPackageCatalogResult`, `RemovePackageCatalogResult`, `ResetPackageCatalogResult`, and `UpdatePackageCatalogResult`.

- Update `Package.appxmanifest` with a new entry. - Add a new entry in `Microsoft.Management.Deployment.InProc.dll.manifest`. - Define new `constexpr CLSID` values for `AddPackageCatalogOptions` in `Factory.cpp`. - Update `s_nameCLSIDPairs` array to include `AddPackageCatalogOptions`.

- Add entry to type dictionary in ClassesDefinition.cs. - Introduce CreateAddPackageCatalogOptions method in WinGetProjectionFactory.cs.

src/Microsoft.Management.Deployment/PackageManager.idl

…nager.idl - Renamed `CatalogName` to `Name` and `m_catalogName` to `m_name` in `AddPackageCatalogOptions`. - Removed `Arguments` property and its methods. - Added new boolean property `Explicit` with getter and setter. Updated `PackageManager.idl` to reflect these changes: - Renamed `CatalogName` to `Name` with updated comments. - Removed `Arguments` property. - Added `Explicit` property with explanatory comments. - Updated `RemovePackageCatalogAsync` description to use "catalog". - Removed several `IVector` and `IVectorView` interfaces related to package catalog options and results.

…thods - Simplified progress reporting for *CatalogPackageAsync operations in PackageManager by using a double type. - Updated method signatures and removed related enums and structures in PackageManager.cpp, PackageManager.h, and PackageManager.idl.

- Introduce new async methods in `PackageCatalogReference` for removing, resetting, and updating package catalogs. - These methods are defined in `PackageCatalogReference.h` and implemented as placeholders in `PackageCatalogReference.cpp`. - Remove corresponding methods from `PackageManager`, shifting responsibility to `PackageCatalogReference`. Update `PackageManager.idl` to reflect these changes: - Add enums and result classes for new operations. - Include new async methods in `PackageCatalogReference`. - Remove outdated methods from `PackageManager`.

- Refactored RemovePackageCatalogAsync and UpdatePackageCatalogAsync to exclude the catalogName parameter in PackageCatalogReference.cpp, PackageCatalogReference.h, and PackageManager.idl. - This modification eliminates the necessity of passing a catalog name, as the required context is now inherently loaded from PackageCatalogInfor within PackageCatalogReference. Note: These operations are not designed to support actions on multiple PackageCatalog/Source, such as ResetAll, UpdateAll, or RemoveAll. These mass operations are intended only for human interactions. For such operations, consumers are expected to recreate 'all' with a simple set of calls.

- Remove progress reporting from ResetPackageCatalogAsync. The underlying implementation for ResetPackageCatalog does not support progress in Repository::DropSource(..). Therefore, the reset API is now an async operation without progress. - The return type of the `ResetPackageCatalogAsync` method in the `PackageCatalogReference` class has been changed from `winrt::Windows::Foundation::IAsyncOperationWithProgress<winrt::Microsoft::Management::Deployment::ResetPackageCatalogResult, double>` to `winrt::Windows::Foundation::IAsyncOperation<winrt::Microsoft::Management::Deployment::ResetPackageCatalogResult>`. This change removes the progress reporting capability from the asynchronous operation. - Corresponding changes have been made in `PackageCatalogReference.h` and `PackageManager.idl` to reflect the new return type. - This simplifies the method's signature and potentially its implementation.

…ageCatalogAsync. - The reset operation is a subset of the remove operation but does not include cleanup. - The remove operation ensures that everything the PackageCatalog writes to the system is cleaned up, in addition to removing the package catalog entry from the winget package catalog list. Conversely, the reset operation only removes the entry without performing any cleanup.

… into user/masudars/WinGet_Source_COM_API

- Increased the size of the s_nameCLSIDPairs array from 9 to 10 elements in Factory.cpp within the Microsoft::Management::Deployment::OutOfProc namespace. - Added a new NameCLSIDPair entry for Microsoft.Management.Deployment.AddPackageCatalogOptions. Updated the last entry for Microsoft.Management.Deployment.RepairOptions to include a trailing comma.

… upcoming commit - Removed the implementation details for handling the removal and update of package catalogs in the `PackageCatalogReference` class. - Added TODO comments indicating future implementation plans.

- Introduced a new COM class `RemovePackageCatalogOptions` with corresponding CLSIDs and properties. - Updated project files and headers to include the new class. - Added RemovePackageCatalogAsync method to handle the new functionality in `PackageManager`. - Renamed `UpdatePackageCatalog` to `RefreshPackageCatalog` and - Removed the old `UpdatePackageCatalogResult` class and introduced `RefreshPackageCatalogResult`. - Added new properties to `AddPackageCatalogOptions`.

Corrected a typo in the comment for `AcceptSourceAgreements` from "PacakageCatalog" to "PackageCatalog". This change improves the readability and accuracy of the code documentation.

Reformatted the RemovePackageCatalogAsync method declaration in PackageManager.h for improved readability.

- Two new error codes, `SourceAgreementsNotAccepted` and `AuthenticationError`, were added to the `AddPackageCatalogResult` - enumeration in the `Microsoft.Management.Deployment` namespace. - The `InvalidOptions` error code was added to the `RemovePackageCatalogResult` enumeration in the same namespace. - The comment for the `RemovePackageCatalogAsync` method was updated to clarify that it unregisters a package catalog and eliminates system artifacts based on the provided options.

- Introduce CallbackDispatcherSink in AppInstaller namespace to manage and dispatch progress updates to registered callbacks. - Implement methods for initializing, updating, and ending progress, as well as adding and firing callbacks. - Ensure thread-safe access with a mutex. - Minor formatting changes included.

Introduced new functionalities for handling package catalog operations, including adding, refreshing, and removing package catalogs. These operations now feature detailed exception handling, progress reporting, and validation checks. Key changes include: - Added `WINGET_CATALOG_CATCH_STORE` macro in ExecutionContext.h. - Implemented `RefreshPackageCatalogAsync` in PackageCatalogReference.cpp. - Implemented `AddPackageCatalogAsync` and `RemovePackageCatalogAsync` in PackageManager.cpp. - Added various helper functions for status mapping and result creation. - Adjusted includes and formatting for consistency in RemovePackageCatalogOptions.h.

- The PackageCatalogInterop class handles package catalog operations for end-to-end tests, including adding, removing, and validating package catalogs. - It addresses edge cases like invalid options, duplicate names, and unaccepted source agreements.

…com/Madhusudhan-MSFT/winget-cli into user/masudars/WinGet_Source_COM_API

…method - Refactored status handling by introducing a template function `GetPackageCatalogOperationStatus` in `Converters.h`, replacing individual status conversion functions in `PackageCatalogReference.cpp` and `PackageManager.cpp`. - Simplified progress reporting by modifying callbacks to report only the current progress value. - Moved new functions `CheckForDuplicateSource`, `CreateSourceFromOptions`, and `GetMatchingSource` in `PackageManager.cpp` close all the private methods. - Included `<optional>` header to support `std::optional`. - Cleaned up redundant code and improved comments for better clarity.

The indentation of the `struct RepairOptions` declaration has been adjusted. Specifically, the indentation level has been decreased by one level, aligning it with the `[uuid(WINGET_OUTOFPROC_COM_CLSID_RepairOptions)]` attribute. This change improves code readability and consistency.

…kageCatalog COM APIs (as it is targetted for 1.10 release) Updated contract version from 11 to 12 in `winrt::Microsoft::Management::Deployment::implementation` and `Microsoft.Management.Deployment` namespaces. Changes include: - Updated comments and attributes in `PackageCatalogReference.h`, `PackageManager.h`, and `PackageManager.idl`. - Modified `contractversion` attribute in `PackageManager.idl` to 12. - Adjusted contract attributes for enums, runtime classes, and methods to version 12. These updates ensure the codebase aligns with the new contract version, incorporating any new features, improvements, or fixes.

Corrected the term "Packagecatalog" to "Package Catalog" in the comment for the RemovePackageCatalogAsync method in the Microsoft.Management.Deployment namespace.

Updated the comment for the RemoveNonExistingPackageCatalog method to enhance readability and grammatical correctness. Changed "Remove non existing package catalog." to "Remove a non-existent package catalog." to make the purpose of the method clearer.

Updated the comment for the RemoveNonExistingPackageCatalog method to improve clarity. The new comment "Remove a package catalog that is not present" is more straightforward and easier to understand.

Deleted the <ItemGroup> containing the <Natvis> element from the Microsoft.Management.Deployment.vcxproj.filters file. This change removes the inclusion of the wil.natvis file, which provided custom visualizations for native types in Visual Studio.

- Removed WINGET_CATALOG_CATCH_STORE macro from ExecutionContext.h. - Refactored GetPackageCatalogOperationStatus in Converters.h. - Updated PackageCatalogReference.cpp to use HandleException. - Refactored PackageManager.cpp for detailed exception handling. - Added new test cases for insecure URI and invalid type scenarios. - Updated PackageCatalogInterop.cs tests to use InvalidOptions status. - Added comments for admin validation checks in InProc/OutOfProc calls.

…tion - Introduced `TestClassTearDown` method with `[OneTimeTearDown]` attribute. - Added `AddUpdateRemovePackageCatalog` method for catalog management. - Added `RemovePackageCatalog` method for catalog removal. - Integrated `AddUpdateRemovePackageCatalog` into existing tests.

- Integrate the fix from "Fix for Source Argument Validation in SourceWorkflow for Default Source Type" (microsoft#4891). This commit incorporates part of the included fix for consistency. - Updated the method of obtaining the default source type in the winrt::Microsoft::Management::Deployment::implementation namespace within PackageManager.cpp. Replaced the use of ::AppInstaller::Repository::ISourceFactory::GetForType("")->TypeName() with ::AppInstaller::Repository::Source::GetDefaultSourceType(). This change simplifies the process and ensures future-proofing against any changes in the SourceFactory's default type.

src/Microsoft.Management.Deployment/PackageManager.cpp

Madhusudhan-MSFT · 2024-10-22T19:08:43Z

src/Microsoft.Management.Deployment/PackageCatalogReference.cpp

+    }
+
+    winrt::Windows::Foundation::IAsyncOperationWithProgress<winrt::Microsoft::Management::Deployment::RefreshPackageCatalogResult, double> PackageCatalogReference::RefreshPackageCatalogAsync()
+    {


Do we need to explicitly call ::AppInstaller::Logging::Telemetry().LogStartup(true); here?

Usually, a caller reaches this point after obtaining a PackageCatalogReference instance by calling one of the PackageManager.*PackageCatalog* methods, which in turn call LogStartupIfApplicable(...) to initialize the logs. Can we safely assume that the COM logging instance is already initialized before the caller attempts this operation? #Closed

Not at the moment. I don't think we have specific telemetry events for source operations.

JohnMcPMS · 2024-10-22T18:45:06Z

src/AppInstallerCommonCore/Public/AppInstallerProgress.h

@@ -126,6 +127,32 @@ namespace AppInstaller
        uint64_t m_globalMax = 0;
    };

+    using ProgressCallBack = std::function<void(uint64_t current, uint64_t maximum, ProgressType progressType)>;


Despite the case sensitive nature of C++, only confusion will result from reusing a name like this. It would be slightly less confusing if this were a member type of your new sink. But at that point I would suggest just having it be named CallbackDispatcherSink::Callback to prevent the reuse of the name.

JohnMcPMS · 2024-10-22T18:55:43Z

src/AppInstallerCommonCore/Public/AppInstallerProgress.h

+    using ProgressCallBack = std::function<void(uint64_t current, uint64_t maximum, ProgressType progressType)>;
+
+    // A sink that dispatches progress to a set of callbacks.
+    struct CallbackDispatcherSink : public IProgressSink


This doesn't seem sufficiently generic/safe to put here. If you want a multi-cast IProgressSink it should be more specifically that (and safer). If my suggestions below aren't meeting some requirements or are causing significant pain during implementation, either I will see why that might be during the rest of the review or we should discuss it more directly.

This is an IProgressSink, but it holds ProgressCallBacks (see name confusion comment), which has one function whose signature happens to match one of the functions of IProgressSink. Why not just have a std::vector<std::shared_ptr<IProgressSink>> m_progressCallbacks that you call out to?

Holding a synchronization primitive while calling out to code you don't explicitly control is dangerous. Imagine that one of the callbacks attempts to call AddCallback in response to a callback. The better pattern is to hold the mutex only to copy the set of callbacks, release it, then iterate through the copy while not holding the mutex.

Another alternative that would remove the need to have a mutex at all would be to have the callbacks set at construction only. This may or may not align with your current use case, but if it does it will be significantly more efficient at runtime than the constant copying required to be safe.

src/Microsoft.Management.Deployment.InProc/Microsoft.Management.Deployment.InProc.dll.manifest

src/Microsoft.Management.Deployment.OutOfProc/Factory.cpp

JohnMcPMS · 2024-10-22T19:06:14Z

src/Microsoft.Management.Deployment/AddPackageCatalogOptions.h

+
+#if !defined(INCLUDE_ONLY_INTERFACE_METHODS)
+    private:
+        std::wstring m_name = L"";


Suggested change

std::wstring m_name = L"";

hstring m_name;

[Applies to all string members] Storing these as std::wstring seems likely to just be a performance hit without any benefit.

The pattern of using std::wstring for private fields storing strings is consistent across various APIs *InputOptions runtimeclass implementation. However, the set and get methods use hstring.

I can change it to hstring, but it may look inconsistent with the existing code pattern.

JohnMcPMS · 2024-10-22T21:40:53Z

src/Microsoft.Management.Deployment/PackageCatalogReference.cpp

+                {
+                    UNREFERENCED_PARAMETER(type);
+                    UNREFERENCED_PARAMETER(maximum);
+                    report_progress(static_cast<double>(current));


Given this usage, I think what you really wanted to build was an IProgressSink that could be constructed from one or more lambdas. The options for that are:

Full generic support in the core implementation

Specific use case support in this project

I would suggest just going with a special implementation in this project. It should be able to handle the same way for Add.

Additionally, this implementation does not take into account the current state of progress reporting from the source functions. In particular:

The Begin and End progress functions are called by the CLI code. If you want consistent 0 and 100% progress reports to be sent, you will have to do that here.

PreIndexed currently performs two operations that report progress back-to-back. First the index is downloaded, then it is installed. The download is reported as bytes out of total bytes. The install is reported as a progress percent.

Rest doesn't report progress at all as neither add nor update actually do anything meaningful.

So what I think we want is:

To define what the progress double means in the IDL comment.

A method that creates an IProgressSink specific to the source type.

A sink implementation for PreIndexed that joins the download/install via something like a 70/30 split to produce a single progress measure.

src/Microsoft.Management.Deployment/PackageManager.cpp

src/Microsoft.Management.Deployment/ResetPackageCatalogResult.cpp

The implementation of PackageCatalogReference already includes the necessary logic to handle AcceptSourceAgreements. The `AcceptSourceAgreements` property has been removed from the `AddPackageCatalogOptions` class and its related files. This includes: - Removal of the property from `AddPackageCatalogOptions.h` and `AddPackageCatalogOptions.cpp`. - Removal of source agreement handling logic in `PackageManager.cpp`. - Removal of `SourceAgreementsNotAccepted` status from `AddPackageCatalogStatus` enum in `PackageManager.idl`. - Removal of the `AddPackageCatalogWithSourceAgreementNotAccepted` test case from `PackageCatalogInterop.cs`. - Removal of handling for `APPINSTALLER_CLI_ERROR_SOURCE_AGREEMENTS_NOT_ACCEPTED` error in `Converters.h` and `PackageManager.cpp`. - Updates to test cases in `GroupPolicyForInterop.cs` and `PackageCatalogInterop.cs` to remove the setting of `AcceptSourceAgreements` to `true`.

…tions - Added IsRunningAsAdminOrSystem() in Runtime.h and Runtime.cpp. - Added COM class entry for RemovePackageCatalogOptions in Microsoft.Management.Deployment.InProc.dll.manifest. - Corrected NameCLSIDPair syntax in Factory.cpp. - Added and declared new status mapping functions in Converters.cpp and Converters.h. - Encapsulated GetRefreshPackageCatalogResult in PackageCatalogReference.cpp. - Encapsulated package catalog operation functions in PackageManager.cpp. - Removed ResetPackageCatalogResult class and its implementation. - Updated error codes in PackageCatalogInterop.cs.

Corrected the indentation of the `clsid` attribute for the `RemovePackageCatalogOptions` comClass element in the `Microsoft.Management.Deployment.InProc.dll.manifest` file. The `clsid` value itself remains unchanged.

…eCatalogAsync to throw errors on invalid arguments, following API design guidelines. - Updated `AddPackageCatalogAsync` and `RemovePackageCatalogAsync` in `PackageManager.cpp` to perform argument validation at the beginning of the methods, ensuring early error detection and potential performance improvements. At this stage, we avoid constructing a result object for error codes and simply use a THROW statement to generate an exception for the caller. - Refactored `AddPackageCatalogWithInvalidOptions` and `RemovePackageCatalogWithInvalidOptions` in `PackageCatalogInterop.cs` to be synchronous and directly test for exceptions using `Assert.ThrowsAsync<ArgumentException>`.

Madhusudhan-MSFT added 4 commits September 15, 2024 22:05

Implement skeleton for package catalog management

874c523

- Update IDL for new runtime classes and methods. - Add files and classes for package catalog operations (add, remove, reset, update). - Update `PackageManager` with async methods for catalog operations. [Compiles successfully]

Add C# WinRT Projection for AddPackageCatalogOptions class

8dbdbb8

- Add entry to type dictionary in ClassesDefinition.cs. - Introduce CreateAddPackageCatalogOptions method in WinGetProjectionFactory.cs.

JohnMcPMS reviewed Sep 16, 2024

View reviewed changes

Madhusudhan-MSFT changed the title ~~win get source com api~~ WinGet Source com api Sep 17, 2024

Madhusudhan-MSFT changed the title ~~WinGet Source com api~~ WinGet Source COM Api Sep 17, 2024

Madhusudhan-MSFT and others added 11 commits September 16, 2024 22:55

Merge branch 'master' of https://github.com/Madhusudhan-MSFT/winget-cli…

aeb98ea

… into user/masudars/WinGet_Source_COM_API

Merge branch 'microsoft:master' into user/masudars/WinGet_Source_COM_API

aa6de76

This comment has been minimized.

Sign in to view

Madhusudhan-MSFT and others added 11 commits October 14, 2024 12:12

Fix typo in comment for AcceptSourceAgreements

ac4ae58

Corrected a typo in the comment for `AcceptSourceAgreements` from "PacakageCatalog" to "PackageCatalog". This change improves the readability and accuracy of the code documentation.

Reformat RemovePackageCatalogAsync method declaration

0bf0ad4

Reformatted the RemovePackageCatalogAsync method declaration in PackageManager.h for improved readability.

Merge branch 'microsoft:master' into user/masudars/WinGet_Source_COM_API

8a39538

Merge branch 'user/masudars/WinGet_Source_COM_API' of https://github.…

3853876

…com/Madhusudhan-MSFT/winget-cli into user/masudars/WinGet_Source_COM_API

This comment has been minimized.

Sign in to view

[SpellCheck_Error] Fix typo in RemovePackageCatalogAsync method comment

fead163

Corrected the term "Packagecatalog" to "Package Catalog" in the comment for the RemovePackageCatalogAsync method in the Microsoft.Management.Deployment namespace.

This comment has been minimized.

Sign in to view

Madhusudhan-MSFT added 2 commits October 17, 2024 23:48

Fix for forbidden-pattern error by spellchecking bot.

69462af

Updated the comment for the RemoveNonExistingPackageCatalog method to improve clarity. The new comment "Remove a package catalog that is not present" is more straightforward and easier to understand.

Madhusudhan-MSFT marked this pull request as ready for review October 18, 2024 16:42

Madhusudhan-MSFT requested a review from a team as a code owner October 18, 2024 16:42

Madhusudhan-MSFT requested review from JohnMcPMS and yao-msft October 18, 2024 23:42

Madhusudhan-MSFT and others added 4 commits October 20, 2024 22:16

Merge branch 'microsoft:master' into user/masudars/WinGet_Source_COM_API

e1902d9

JohnMcPMS added the Requires-AppxManifest-Update Indicates a Pull Request that will require an update to the internal AppxManifest file when pulled. label Oct 22, 2024

Madhusudhan-MSFT commented Oct 22, 2024

View reviewed changes

src/Microsoft.Management.Deployment/PackageManager.cpp Outdated Show resolved Hide resolved

Madhusudhan-MSFT commented Oct 22, 2024

View reviewed changes

JohnMcPMS reviewed Oct 22, 2024

View reviewed changes

Madhusudhan-MSFT added 4 commits October 22, 2024 20:54

Fix indentation in RemovePackageCatalogOptions clsid

f92b947

Corrected the indentation of the `clsid` attribute for the `RemovePackageCatalogOptions` comClass element in the `Microsoft.Management.Deployment.InProc.dll.manifest` file. The `clsid` value itself remains unchanged.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WinGet Source COM Api #4813

WinGet Source COM Api #4813

Madhusudhan-MSFT commented Sep 16, 2024 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Madhusudhan-MSFT Oct 22, 2024 •

edited

Loading

yao-msft Oct 22, 2024

JohnMcPMS Oct 22, 2024

JohnMcPMS Oct 22, 2024

JohnMcPMS Oct 22, 2024

Madhusudhan-MSFT Oct 23, 2024

JohnMcPMS Oct 22, 2024

WinGet Source COM Api #4813

Are you sure you want to change the base?

WinGet Source COM Api #4813

Conversation

Madhusudhan-MSFT commented Sep 16, 2024 • edited Loading

This update includes the following changes:

How Validated:

Microsoft Reviewers: Open in CodeFlow

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Madhusudhan-MSFT Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

yao-msft Oct 22, 2024

Choose a reason for hiding this comment

JohnMcPMS Oct 22, 2024

Choose a reason for hiding this comment

JohnMcPMS Oct 22, 2024

Choose a reason for hiding this comment

JohnMcPMS Oct 22, 2024

Choose a reason for hiding this comment

Madhusudhan-MSFT Oct 23, 2024

Choose a reason for hiding this comment

JohnMcPMS Oct 22, 2024

Choose a reason for hiding this comment

Madhusudhan-MSFT commented Sep 16, 2024 •

edited

Loading

Madhusudhan-MSFT Oct 22, 2024 •

edited

Loading