It's not intuitive that a particular fuzz test will get seeded differently depending on the number and order of tests being run

[ento]

What happened

I spotted a failing fuzz test in a CI build, and replicated the failure by running it locally with the same seed:

$ ./node_modules/.bin/elm-test --fuzz 100 --seed 330613302
...
Given ["",""]

    Just ""
    ╷
    │ Expect.equal
    ╵
    Nothing
...

Then I wanted to narrow down the tests being run to examine what's really going on, which made the failure disappear:

$ ./node_modules/.bin/elm-test --fuzz 100 --seed 330613302 \
tests/Flux/FluxableSpec.elm
...
TEST RUN PASSED

A good colleague of mine pointed out that because seeds are distributed across the entire tests being run, one should use only to isolate the test you want to run.

Thoughts

• As a user, it's unintuitive that specifying the seed will not "fix" the randomness of tests in place.
• While only works, I'd rather be able to run a single test file, because it takes ~15 seconds to run elm-test on our test suite even against a single only test on top of already compiled code.
• Is there ever a use case where we want the current behavior when the user has specified a seed? If not, will it be a good idea to not do any distribution at all if we're given a user-supplied seed?
• Maybe this really isn't an issue if Please contribute examples of fuzzing and shrinking bugs #190 is addressed and we have a good fuzzer in place or elm-test runs speedily.

[rtfeldman]

Great point!

If I understand correctly, I think this is CLI-specific, so rtfeldman/node-test-runner would probably be the better place for the issue to live.

I might title it something like:

Outcomes with the same --seed can change if file glob changes

I think it's solvable by using both the fully-qualified module name and the --seed as the starting point for seed distribution to the tests in that module. That should work out fine since fully-qualified module names have to be unique.

[mgold]

I think we made sure to distribute seeds before doing only pruning, but I guess we didn't account for the glob changing.

Closing. Please file the issue in the other issue like Richard suggests!

[jwoudenberg]

@rtfeldman I think that seed distribution logic is located in this repo, here: https://github.com/elm-community/elm-test/blob/master/src/Test/Runner.elm?

Just double checked node-test-runner and it looks like it's using that.

point for seed distribution to the tests in that module. That should work out fine since fully-qualified module names have to be unique.

Would the test description also work instead of the fully-qualified module name? That might be easier to access and be less dependant on the runner.

[rtfeldman]

Would the test description also work instead of the fully-qualified module name?

Great point!

I can't think of a reason this wouldn't work, and doing it this way means that all runners would get the same consistency for free. 🤔

[mgold]

Okay, so... reopening?