Add prototype for custom test reporter for minitest

[andyw8]

Addresses #3176

This adds a custom reporter for use with the upcoming changes planned for the Test Explorer UI. It emits the test progress as a series of events, for easy consumption by the VS Code extension.

I have a version for test/unit in progress separately.

No tests yet, but I will add once things become more concrete.

[graphite-app]

How to use the Graphite Merge Queue

Add the label graphite-merge to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

[andyw8]

There is some duplication between the various record_ methods, but I want to understand what's actually required before cleaning this up. We'll know better once integrated with the extension.

[st0012]

I think the methods in this class should just take in id (or full_name) and leave the assembling to each test framework's addons. For example, I may use test example/group's location as the rspec addon's test id first.

[vinistock]

What does before_test mean? If this means when we're starting to run it, should this be named start_test?

[vinistock]

Test outputs and messages may include line breaks, so we need to use JSON RPC to be able to parse this accurately from the client side.

Suggested change

	puts result.to_json
	json_message = result.to_json
	$stdout.write("Content-Length: #{json_message.bytesize}\r\n\r\n#{json_message}")

[andyw8]

When we call to to_json it will escape the line breaks.

Re. the client/server split: does this feature need to involve the server at all? If we're only targetting VS Code, then could the extension watch the output directly?

[vinistock]

What does before_test mean? If this means when we're starting to run it, should this be named start_test?

[vinistock]

This class has no state. Should these just be class methods instead? Also, I think it's clearer to call this a TestReporter.

[andyw8]

I've changed the name.

Also I am now passing in an IO instance to allow for easier testing and flexibility.

[vinistock]

What is type here? What will we use it for in the extension?

[andyw8]

I've seen two variants so far:

• A Minitest::Assertion failure
• A Minitest::UnexpectedError, e.g. if something in the test raises

I thought it may be useful to distingish these in the UI, similar to how the progress reporter shows E vs F.

[vinistock]

These record events seem to have access to the file path without having to resort to Module.const_source_location. Is there some other event, like start_suite that we can remember the file path ahead of time?

[andyw8]

There's a before_suite which takes a Minitest::Reporters::Suite, but that doesn't have file information:

https://github.com/minitest-reporters/minitest-reporters/blob/265ff4b40d5827e84d7e902b808fbee860b61221/lib/minitest/reporters/base_reporter.rb#L3C11-L3C16

We could propose adding this in minitest-reporters.

[st0012]

Just a headsup that even in minitest/test-unit it's possible to define test classes dynamically, which make their source location path a bit different, like:

class Foo
  class_eval <<~RUBY
    class FooTest < Test::Unit::TestCase
      def test_foo
        puts "========================================="
        puts Kernel.const_source_location("Foo::FooTest")
        puts "========================================="
        fail
      end
    end
  RUBY
end

=========================================
(eval at /Users/hung-wulo/src/github.com/Shopify/rdoc/test/rdoc/test_rdoc_comment.rb:483)
1
=========================================

[andyw8]

Pushed an update to handle that. Still need to think about what we do in the UI.