The test run did not record any output

[klaaspieter]

Description

Ruby LSP Information

Ruby LSP Information

VS Code Version

1.96.4

Ruby LSP Extension Version

0.8.18

Ruby LSP Server Version

0.23.6

Ruby LSP Add-ons

• Ruby LSP Rails

Ruby Version

3.3.0

Ruby Version Manager

none

Installed Extensions

Click to expand

• EditorConfig (0.16.4)
• bust-a-gem (0.1.8)
• color-highlight (2.8.0)
• debugpy (2024.14.0)
• endwise (1.5.1)
• prettier-vscode (11.0.0)
• python (2024.22.2)
• rails-db-schema (0.2.9)
• remote-containers (0.394.0)
• ruby-lsp (0.8.18)
• shellcheck (0.37.6)
• template-string-converter (0.6.1)
• tokyo-night (1.0.9)
• vscode-docker (1.29.4)
• vscode-dotnet-runtime (2.2.5)
• vscode-eslint (3.0.10)
• vscode-github-actions (0.27.0)
• vscode-lldb (1.11.2)
• vscode-mjml (2.0.0)
• vscode-pylance (2024.12.1)
• vscode-rdbg (0.2.2)
• vscode-rubocop (0.7.0)
• vscode-todo-highlight (1.0.5)
• vscode-versionlens (1.16.2)
• vsliveshare (1.0.5948)
• zeit-co-vscode-schemas (1.0.1)

Ruby LSP Settings

Click to expand

Workspace

{}

User

{
  "enabledFeatures": {
    "codeActions": true,
    "diagnostics": true,
    "documentHighlights": true,
    "documentLink": true,
    "documentSymbols": true,
    "foldingRanges": true,
    "formatting": true,
    "hover": true,
    "inlayHint": true,
    "onTypeFormatting": true,
    "selectionRanges": true,
    "semanticHighlighting": true,
    "completion": true,
    "codeLens": true,
    "definition": true,
    "workspaceSymbol": true,
    "signatureHelp": true,
    "typeHierarchy": true
  },
  "featuresConfiguration": {},
  "addonSettings": {},
  "rubyVersionManager": {
    "identifier": "auto"
  },
  "customRubyCommand": "",
  "formatter": "auto",
  "linters": null,
  "bundleGemfile": "",
  "testTimeout": 30,
  "branch": "",
  "pullDiagnosticsOn": "both",
  "useBundlerCompose": false,
  "bypassTypechecker": false,
  "rubyExecutablePath": "",
  "indexing": {},
  "erbSupport": true,
  "featureFlags": {}
}

### Reproduction steps

1. Run a minitest test in VSCode (perhaps also a problem with other test runners but I haven't tested any)

Code snippet or error message
The “TEST RESULTS” tab opens with “Ruby LSP” selected on the right side and “The test run did not record any output.” on the left. When I click on a specific test I do see the output. I always have to select a specific test to see its output. Is this intended behavior or is there something wrong with my setup?

[vinistock]

Thank you for the report. Currently, our implementation for the test explorer is a bit simplistic and we need to improve it.

For example, we have no way to tell test frameworks to return JSON structured results so that we can return that to the editor. Our only option is to try to parse the output printed to stdout manually.

Do you happen to be using any specific reporter for Minitest other than the default one? Or do you have a test example that reproduces the problem?

The correct solution, which we prototyped but haven't had the time to finish yet, is to have a special LSP reporter that we force tests to use and that returns us structured JSON responses, so that we can easily attach that to the test items in the extension rather than trying to parse arbitrary output.

[andyw8]

a special LSP reporter that we force tests to use and that returns us structured JSON responses, so that we can easily attach that to the test items in the extension rather than trying to parse arbitrary output.

An alternative may be to use the JUnit XML format which is already supported by several test frameworks: https://github.com/testmoapp/junitxml

[klaaspieter]

@vinistock thank you for the context. I tried reproducing it with a new Rails 8 app but its appears to work there. Do you have a way I can debug this in the problematic app?

[vinistock]

There's no easy way, but if you're willing to dig into the code, then you could try to figure out what's going on by:

1. Setting up Ruby LSP development using our CONTRIBUTING docs
2. Put a breakpoint here where we decide what output to append to each test run
3. Then try to understand why it's not being appended

You will need to start the extension in development mode (described in the contributing docs) and in the second VS Code window you would open the project that reproduces the issue and run tests to trigger the breakpoint.

[klaaspieter]

Thanks @vinistock I think I managed to find the issue.

[klaaspieter]

For my own sanity I think there are several situations that may need different solutions. I tested these scenarios in both the Rails app that I initially reported the problem for and a brand new Rails 8 app.

A single passing test

Test run output

Run options: --seed 24862

# Running:

.

Finished in 0.004230s, 236.4066 runs/s, 236.4066 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips    

Test case output

Run options: --seed 23134

# Running:

.

Finished in 0.003996s, 250.2503 runs/s, 250.2503 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

A single failing test

Test run output

*  The test run did not record any output.

This is solved by #3146. It's not appending double for me. Perhaps I'm misunderstanding the APIs and what I'm seeing but it looks like the call to run.failed is appending output to the test case but not the test run.

Test case output

Run options: --seed 16039

# Running:

F

Failure:
SchedulerTest#test_the_fake_news [test/jobs/scheduler_test.rb:9]:
Expected false to be truthy.

rails test test/jobs/scheduler_test.rb:8



Finished in 0.004678s, 213.7666 runs/s, 213.7666 assertions/s.
1 runs, 1 assertions, 1 failures, 0 errors, 0 skips

A file with a failing and passing test

Test case output

Run options: --seed 27726

# Running:

.

Finished in 0.004129s, 242.1894 runs/s, 242.1894 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

This only showing the passing test. I believe this is because even though we're running tests for the entire file we're executing separate commands for each test.

Passing test output

Run options: --seed 27726

# Running:

.

Finished in 0.004129s, 242.1894 runs/s, 242.1894 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

Failing test output

Run options: --seed 31575

# Running:

F

Failure:
SchedulerTest#test_the_fake_news [test/jobs/scheduler_test.rb:9]:
Expected false to be truthy.

rails test test/jobs/scheduler_test.rb:8



Finished in 0.004287s, 233.2634 runs/s, 233.2634 assertions/s.
1 runs, 1 assertions, 1 failures, 0 errors, 0 skips