crystal tool unreachable support for specs

[devnote-dev]

The unreachable tool is great for applications because there's an entrypoint. This doesn't really work for normal shards, most wont have an entrypoint and you can't really target specs for coverage (without some hacks at least). I think there should be some support by the tool or by crystal spec for unreachable code coverage.

[straight-shoota]

you can't really target specs for coverage (without some hacks at least).

Could you elaborate what stands in the was of doing that? What kind of are you thinking about?

[devnote-dev]

Running the unreachable tool on a dummy file that manually invokes the spec runner seems to work (although it's showing some results that I don't think should be there). But the spec runner is internal API so it took me a while to figure out that you could even do this — it's not ideal for library maintainers.

[Blacksmoke16]

I been running it against the same file I needed for kcov. Which is basically:

require "../../src/components/clock/spec/**"

Kinda annoying, but works out of the box w/o any hacks/required changes to the compiler or anything. 🤷

[devnote-dev]

The tool is calling ASPEC.run_all hence why you're able to get coverage.

[Blacksmoke16]

That is irrelevant here. I'm sure this same approach would work for any shard. This is basically what crystal spec does. Running this file would run the spec suite even if you're using only stdlib spec features.

[devnote-dev]

Oh I see, when trying to run this from the command line it wasn't reaching anything so I assumed it was because the spec runner wasn't actually running. 😅

Having a file to require all specs for coverage isn't really an issue, but it would be nice if --include and --exclude supported glob patterns so that I can do -i spec/*.

[Blacksmoke16]

Oh I see, when trying to run this from the command line it wasn't reaching anything so I assumed it was because the spec runner wasn't actually running. 😅

It should if you run it directly. If you run it through unreachable, that doesn't do codegen so it wouldn't actually, but would still run against all your tests + real code.

[devnote-dev]

I'm running crystal tool unreachable spec/spec_helper.cr -i spec/* which lists everything as unreachable. I've also tried specifying all spec files with -i but it seems it doesn't accept multiple arguments.

[straight-shoota]

What happens if you run crystal run spec/spec_helper.cr? I bet it doesn't run anything.

Note the --include flag of unreachable does not affect which code to analyze. It only filters the output (thus mostly useful to override a less specific --exclude).

[Blacksmoke16]

Yea the spec_helper file doesn't actually require the *_spec.cr files, that's a compiler thing. Some more details on that: #1157 (comment).