Test failure: When_there_are_subcommands_and_options_then_a_subcommand_must_be_provided

[natemcmaster]

Test Name:	System.CommandLine.Tests.ParsingValidationTests.When_there_are_subcommands_and_options_then_a_subcommand_must_be_provided
Test FullName:	System.CommandLine.Tests.ParsingValidationTests.When_there_are_subcommands_and_options_then_a_subcommand_must_be_provided
Test Source:	C:\dev\dotnet\System.CommandLine\src\System.CommandLine.Tests\ParsingValidationTests.cs : line 182
Test Outcome:	Failed
Test Duration:	0:00:00.095

Result StackTrace:	
at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message) in C:\projects\fluentassertions-vf06b\Src\FluentAssertions.Net45\Execution\XUnit2TestFramework.cs:line 32
   at FluentAssertions.Execution.AssertionScope.FailWith(String message, Object[] args) in C:\projects\fluentassertions-vf06b\Src\Core\Execution\AssertionScope.cs:line 197
   at FluentAssertions.Collections.SelfReferencingCollectionAssertions`2.ContainSingle(Expression`1 predicate, String because, Object[] becauseArgs) in C:\projects\fluentassertions-vf06b\Src\Core\Collections\SelfReferencingCollectionAssertions.cs:line 492
   at System.CommandLine.Tests.ParsingValidationTests.When_there_are_subcommands_and_options_then_a_subcommand_must_be_provided()
Result Message:	Expected collection to contain a single item matching ((e.Message == "Required command was not provided.") AndAlso (e.ParsedSymbol.Name == "inner")), but no such item was found.
Result StandardOutput:	
[ outer [ inner <arg> ] ]
the-message

This reproduces consistently in Debug, but not release. There appears to be some kind of race condition in the tests. The output "the-message" comes from another test, Default_validation_messages_can_be_replaced_in_order_to_add_localization_support.

cc @jonsequitur

[jonsequitur]

Oddly, it sometimes passes: https://ci2.dot.net/job/Private/job/dotnet_System.CommandLine/job/master/job/Debug_Windows_NT_prtest/22/.

Something similar-looking was happening as well in the original repo and the mystery wasn't solved, though I had a proposal that we could try. dotnet/CliCommandLineParser#93

[natemcmaster]

Apparently still failing :-/

https://ci2.dot.net/job/Private/job/dotnet_System.CommandLine/job/master/job/Release_Ubuntu16.04_prtest/38/testReport/junit/System.CommandLine.Tests/ParsingValidationTests/When_there_are_subcommands_and_options_then_a_subcommand_must_be_provided/

[jonsequitur]

Saw that. Trying an update to FluentAssertions.

[natemcmaster]

So, Jon and I found out on the way to Spokane that this is the race condition in test parallelization and ValidationMessages.Current. I think we should consider not having a static, settable property to configure validation message customization.

[jonsequitur]

I had thought I fixed this by referencing ValidationMessages.Current to work around the race condition but it apparently still exists:

https://ci2.dot.net/job/Private/job/dotnet_System.CommandLine/job/master/job/Release_Windows_NT_prtest/56/

So no shortcut here.. We should definitely change the localization story to not use a static.

[natemcmaster]

I drafted a change, but the localization is tough because much of the validation is currently done in static help methods. Will have to think about it more

[natemcmaster]

Should be resolved with #85