[BUG] Getting warning "coverage.cobertura.xml was not found" while publishing unit-test result

[biswajitsamal59]

Describe the bug
I am using below dotnet test command to run unit-test and collect code coverage. While publishing the unit-test result using PublishTestResults@2 task getting "coverage.cobertura.xml was not found" warning. When I checked the .trx file which contain unit-test result there is a entry for CollectorDataEntries which is causing this.

To Reproduce
Use below dotnet command
dotnet test TestApp.Tests.csproj --logger "trx;LogFileName=TestApp.trx" --collect:"XPlat Code Coverage" --settings "..\coverlet.runsettings"

coverlet.runsettings

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>cobertura</Format>
          <Exclude>[TestHelper*]*</Exclude>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Publish unit-test result using task: PublishTestResults@2

Expected behavior
Code coverage should not interfere with the unit-test result (.trx) file

Actual behavior
Unit-test result file has entry like below:

<CollectorDataEntries>
      <Collector agentName="Agent00002T" uri="datacollector://microsoft/CoverletCodeCoverage/1.0" collectorDisplayName="XPlat code coverage">
        <UriAttachments>
          <UriAttachment>
            <A href="Agent00002T\coverage.cobertura.xml"></A>
          </UriAttachment>
        </UriAttachments>
      </Collector>
</CollectorDataEntries>

Configuration (please complete the following information):
Please provide more information on your .NET configuration:
* Which coverlet package and version was used? 6.0.2
* Which version of .NET is the code running on? net6.0-windows x86
* What OS and version, and what distro if applicable? Windows 10
* What is the architecture (x64, x86, ARM, ARM64)? .Net x86
* Do you know whether it is specific to that configuration? No

[Bertk]

Please use Diag option and add the log file e.g.
--diag:"$(Build.SourcesDirectory)\artifacts\log\$(buildConfiguration)\test.diag.log;tracelevel=verbose"

We use this in coverlet CI for troubleshooting. We are not using .runsetting file.

Please share target framework information of unit test project.

Please run test without using dotnet test TestApp.Tests.csproj --collect:"XPlat Code Coverage" --settings "..\coverlet.runsettings" and check trx file content.

https://learn.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2022#loggerrunsettings-element

[biswajitsamal59]

Below are the configuration we use for the test project:

    <TargetFramework>net6.0-windows</TargetFramework>
    <RuntimeIdentifier>win-x86</RuntimeIdentifier>

<PackageReference Include="xunit" Version="2.7.0" />

If I am removing --logger "trx;LogFileName=TestApp.trx" then it's not producing any trx file which I need to publish. Also there are multiple projects in that solution we to change the LogFileName as well.

[Bertk]

Sorry, I did not add the .runsetting configuration for trx logger.

<LoggerRunSettings>
    <Loggers>
      <Logger friendlyName="trx" enabled="True">
        <Configuration>
          <LogFileName>TestApp.trx</LogFileName>
        </Configuration>
      </Logger>
    </Loggers>
  </LoggerRunSettings>

Alternatively you can also use Directory.Build.props file and define the trx logger.
see https://github.com/coverlet-coverage/coverlet/blob/7c8c6fae2715308a0116e2b40221f06cbf07e7bd/Directory.Build.props#L36C1-L49C19

I still think this is a configuration issue. Maybe it is better to avoid .runsettings file for the tests using dynamic test result names. Maybe you should also use coverlet.msbuild because there is a nice blue print for a build pipeline the coverlet git project.

[biswajitsamal59]

Yes I don't think this logger runsetting is different from the command I use.
dotnet test TestApp.Tests.csproj --logger "trx;LogFileName=TestApp.trx" --collect:"XPlat Code Coverage" --settings "..\coverlet.runsettings

If I am removing coverlet --collect and --settings parameters, the trx file is giving expected output. But after adding coverlet is modifying the .trx file.

Still I am surprised why coverlet is modifying the test results file.

[Bertk]

coverlet does not modify test results. Test results are created with VSTest.Console.exe and maybe something is missing in runsettings configuration file.

https://learn.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2022
https://github.com/microsoft/vstest/blob/main/docs/RFCs/0021-CodeCoverageForNetCore.md
microsoft/vstest#2378 (comment)

[biswajitsamal59]

Then why TestApp.trx file has below entry:

<CollectorDataEntries>
      <Collector agentName="Agent00002T" uri="datacollector://microsoft/CoverletCodeCoverage/1.0" collectorDisplayName="XPlat code coverage">
        <UriAttachments>
          <UriAttachment>
            <A href="Agent00002T\coverage.cobertura.xml"></A>
          </UriAttachment>
        </UriAttachments>
      </Collector>
</CollectorDataEntries>

I have shared the commands and runsetting file which are being used.

[Bertk]

coverlet does not modify test results. Test results are created with VSTest.Console.exe and maybe something is missing in runsettings configuration file.

Looks like the test results are not reflected in the trx file in your environment. I used a repo of another issue and got this results.

content of TestApp.trx:

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="e823f4d7-16c8-4dc3-a1a5-a23afdd0dcb1" name="bertk@LAPTOP-BERT 2024-10-03 09:06:51" runUser="LAPTOP-BERT\bertk" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2024-10-03T09:06:51.7254173+02:00" queuing="2024-10-03T09:06:51.7254178+02:00" start="2024-10-03T09:06:49.1711859+02:00" finish="2024-10-03T09:06:51.7303545+02:00" />
  <TestSettings name="default" id="938d9e56-4f11-4eb7-a4f0-46922a07f42f">
    <Deployment runDeploymentRoot="bertk_LAPTOP-BERT_2024-10-03_09_06_51" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="6ea6c19c-5b61-488f-86c6-e8e7c5632e5e" testId="8ee823ad-9cd6-033b-c5c3-1267f93a66f4" testName="GetObject_Generic_NotNull" computerName="LAPTOP-BERT" duration="00:00:00.0078380" startTime="2024-10-03T09:06:51.2370548+02:00" endTime="2024-10-03T09:06:51.2448483+02:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="6ea6c19c-5b61-488f-86c6-e8e7c5632e5e" />
    <UnitTestResult executionId="596d367a-3167-480c-8b82-36e525fd07a1" testId="9f039ce5-05b4-0335-0d1d-8c0743812357" testName="GetObject_Regular_NotNull" computerName="LAPTOP-BERT" duration="00:00:00.0001900" startTime="2024-10-03T09:06:51.2459357+02:00" endTime="2024-10-03T09:06:51.2461265+02:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="596d367a-3167-480c-8b82-36e525fd07a1" />
  </Results>
  <TestDefinitions>
    <UnitTest name="GetObject_Regular_NotNull" storage="c:\github\coverlet-issue-1691\coverletgenericissue.test\bin\debug\net8.0\coverletgenericissue.test.dll" id="9f039ce5-05b4-0335-0d1d-8c0743812357">
      <Execution id="596d367a-3167-480c-8b82-36e525fd07a1" />
      <TestMethod codeBase="C:\GitHub\coverlet-issue-1691\CoverletGenericIssue.Test\bin\Debug\net8.0\CoverletGenericIssue.Test.dll" adapterTypeName="executor://nunit3testexecutor/" className="CoverletGenericIssue.Test.Tests" name="GetObject_Regular_NotNull" />
    </UnitTest>
    <UnitTest name="GetObject_Generic_NotNull" storage="c:\github\coverlet-issue-1691\coverletgenericissue.test\bin\debug\net8.0\coverletgenericissue.test.dll" id="8ee823ad-9cd6-033b-c5c3-1267f93a66f4">
      <Execution id="6ea6c19c-5b61-488f-86c6-e8e7c5632e5e" />
      <TestMethod codeBase="C:\GitHub\coverlet-issue-1691\CoverletGenericIssue.Test\bin\Debug\net8.0\CoverletGenericIssue.Test.dll" adapterTypeName="executor://nunit3testexecutor/" className="CoverletGenericIssue.Test.Tests" name="GetObject_Generic_NotNull" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="8ee823ad-9cd6-033b-c5c3-1267f93a66f4" executionId="6ea6c19c-5b61-488f-86c6-e8e7c5632e5e" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="9f039ce5-05b4-0335-0d1d-8c0743812357" executionId="596d367a-3167-480c-8b82-36e525fd07a1" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="2" executed="2" passed="2" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <Output>
      <StdOut>NUnit Adapter 4.2.0.0: Test execution started&#xD;
Running all tests in C:\GitHub\coverlet-issue-1691\CoverletGenericIssue.Test\bin\Debug\net8.0\CoverletGenericIssue.Test.dll&#xD;
   NUnit3TestExecutor discovered 2 of 2 NUnit test cases using Current Discovery mode, Non-Explicit run&#xD;
NUnit Adapter 4.2.0.0: Test execution complete&#xD;
</StdOut>
    </Output>
    <CollectorDataEntries>
      <Collector agentName="LAPTOP-BERT" uri="datacollector://microsoft/CoverletCodeCoverage/1.0" collectorDisplayName="XPlat code coverage">
        <UriAttachments>
          <UriAttachment>
            <A href="LAPTOP-BERT\coverage.cobertura.xml"></A>
          </UriAttachment>
        </UriAttachments>
      </Collector>
    </CollectorDataEntries>
  </ResultSummary>
</TestRun>

Please check test execution for warnings and errors.

Tip

use diagnostic logging: --diag:"test.diag.log;tracelevel=verbose"
use build binary log file: -bl:test.binlog
MSBuild Binary and Structured Log Viewer

[biswajitsamal59]

Thank you @Bertk for putting effort to look into the issue.

It's generating three log files: TestWebApi.log, TestWebApi.datacollector.24-10-03_12-21-25_19621_1.log, TestWebApi.host.24-10-03_12-22-43_98717_12.log
TestWebApi.log file is not having any error in it. But there are some error on other two log files. I just masked the actual app name with Test. Are these sufficient find root cause if the issue or you need any other information.

TestWebApi.datacollector.24-10-03_12-21-25_19621_1.log

TpTrace Error: 0 : 8084, 1, 2024/10/03, 12:21:25.416, 11323495862, datacollector.dll, The type "Coverlet.Collector.DataCollection.CoverletCoverageCollector" defined in "coverlet.collector.dll" does not have ExtensionUri attribute.

TpTrace Error: 0 : 8084, 1, 2024/10/03, 12:21:25.419, 11323524966, datacollector.dll, The type "Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector" defined in "Microsoft.VisualStudio.TraceDataCollector.dll" does not have ExtensionUri attribute.

TpTrace Error: 0 : 8084, 1, 2024/10/03, 12:21:25.421, 11323543943, datacollector.dll, The type "Microsoft.TestPlatform.Extensions.BlameDataCollector.BlameCollector" defined in "Microsoft.TestPlatform.Extensions.BlameDataCollector.dll" does not have ExtensionUri attribute.

TpTrace Error: 0 : 8084, 1, 2024/10/03, 12:21:25.423, 11323568029, datacollector.dll, The type "Microsoft.TestPlatform.Extensions.EventLogCollector.EventLogDataCollector" defined in "Microsoft.TestPlatform.Extensions.EventLogCollector.dll" does not have ExtensionUri attribute.

TestWebApi.host.24-10-03_12-22-43_98717_12.log

TpTrace Verbose: 0 : 1632, 4, 2024/10/03, 12:22:44.293, 12112262789, testhost.dll, TestPluginCache: Discovered the extensions using extension path ''.
TpTrace Verbose: 0 : 1632, 4, 2024/10/03, 12:22:44.293, 12112263317, testhost.dll, TestPluginCache: Discoverers are ''.
TpTrace Verbose: 0 : 1632, 4, 2024/10/03, 12:22:44.293, 12112263765, testhost.dll, TestPluginCache: Executors are 'executor://xunit/VsTestRunner2/netcoreapp'.
TpTrace Verbose: 0 : 1632, 4, 2024/10/03, 12:22:44.293, 12112264325, testhost.dll, TestPluginCache: Setting providers are ''.
TpTrace Verbose: 0 : 1632, 4, 2024/10/03, 12:22:44.293, 12112264694, testhost.dll, TestPluginCache: Loggers are ''.
TpTrace Verbose: 0 : 1632, 4, 2024/10/03, 12:22:44.294, 12112271544, testhost.dll, TestExecutorService: Loaded the settings providers
TpTrace Information: 0 : 1632, 4, 2024/10/03, 12:22:44.294, 12112272138, testhost.dll, TestExecutorService: Loaded the extensions
TpTrace Error: 0 : 1632, 4, 2024/10/03, 12:22:44.318, 12112514335, testhost.dll, InProcDataCollectionExtensionManager: Error occured while Initializing the datacollectors : System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollectionExtensionManager.CreateDataCollector(DataCollectorSettings dataCollectorSettings, TypeInfo interfaceTypeInfo)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollectionExtensionManager.InitializeInProcDataCollectors(String runSettings)
TpTrace Information: 0 : 1632, 4, 2024/10/03, 12:22:44.337, 12112698444, testhost.dll, TestDiscoveryManager: Discovering tests from sources S:\a\1\s\TestApp\test\TestWebApi.Tests\bin\Release\net6.0-windows\win-x86\TestWebApi.Tests.dll

TestWebApi.Tests.UnitTests.AutoMapper.ValueResolvers.LicenseAppVersionValueResolverTests.ResolvesAppVersion_IfExists.
TpTrace Error: 0 : 1632, 16, 2024/10/03, 12:22:44.598, 12115310000, testhost.dll, TestCaseEventsHandler.RaiseTestCaseStart: Exception occurred while calling handler of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyOutOfProcDataCollectionManager for TestCaseStartEventArgs: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveRawMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.DataCollectionTestCaseEventSender.SendTestCaseStart(TestCaseStartEventArgs e)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyOutOfProcDataCollectionManager.TriggerTestCaseStart(Object sender, TestCaseStartEventArgs e)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at Microsoft.VisualStudio.TestPlatform.Utilities.MulticastDelegateUtilities.SafeInvoke(Delegate delegates, Object sender, EventArgs args, String traceDisplayName)

[Bertk]

Please execute the test project using following steps:

• dotnet build
• dotnet publish [path-to/TestApp.Tests.csproj]
• dotnet test ... --diag:"test.diag.log;tracelevel=verbose"

Also the test project should be updated (test should run everywhere):

    <TargetFramework>net6.0</TargetFramework>
    <PlatformTarget>AnyCPU</PlatformTarget>
    <!-- <RuntimeIdentifier>win-x86</RuntimeIdentifier> -->

Maybe you can share the csproj file for the test project or you compare it by yourself with the test project from the repo.