Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve test output for failed stubs #2498

Merged
merged 1 commit into from
Jan 15, 2025
Merged

Improve test output for failed stubs #2498

merged 1 commit into from
Jan 15, 2025

Conversation

mislav
Copy link
Member

@mislav mislav commented Jan 15, 2025

The stub/unstub test helpers are unfortunately opaque as both are undocumented, and when stubbing fails for some reason, determining why is left as a guessing/debugging exercise. This was a time-sink for myself on many an occasion.

This improves the unstub <PROGRAM> helper output with the following features:

  • If the number of PROGRAM invocations was fewer than the number of stubbed (i.e. expected) arguments, it now prints expected X invocations, got Y to stderr;

  • If PROGRAM got invoked but unstub PROGRAM still failed, it was likely that some of the invocations didn't match the stubbed ones. However, the human reading the test output has no insight into which invocations actually happened. The unstub helper now prints:

    re-run test with <PROGRAM>_STUB_DEBUG=3 to log `<PROGRAM>' invocations
    

    This "debug" mode already existed, but is off by default, and could previously only be discovered by someone who read the stub implementation code. After following the new suggestion, actual invocations of PROGRAM and all its arguments will be printed to stderr during the test run, hopefully making it clearer for the human debugging the test.

@mislav mislav requested a review from eregon January 15, 2025 12:41
Copy link
Member

@eregon eregon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@eregon eregon merged commit fe8c6c2 into master Jan 15, 2025
6 checks passed
@eregon eregon deleted the stub-improvement branch January 15, 2025 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants