Feature request

[XhmikosR]

Hello!

I'm looking for a good replacement of https://www.npmjs.com/package/broken-link-checker to use in Bootstrap and came across your module.

It does look pretty simple and does quite a good job. I was wondering if you could add a few options, haven't checked if the lib already exposes them so if it's a matter of exposing them to the CLI.

1. An option to skip external links. In our case this speeds things up a lot
2. An option to control concurrency
3. Maybe print the total time spent and/or total links that were checked even when an error occurs?
4. It seems the tool follows redirects. Can you show it in the output too?

This is a question, what's the depth when --recurse is used?

Thanks in advance!

[XhmikosR]

BTW It seems when a link is relative it wrongfully is detected as broken.

For example:

Scanning http://localhost:5180/docs/4.3/examples/navbar-static/
 [200] https://getbootstrap.com/docs/4.3/examples/navbar-static/
 [404] http://localhost:5180/navbar-top.css
 [200] http://localhost:5180/docs/4.3/examples/navbar-fixed/

navbar-top.css is linked in http://localhost:5180/docs/4.3/examples/navbar-static/ with <link href="navbar-top.css" rel="stylesheet">.

I made a PR twbs/bootstrap#28291 to test this.

EDIT: reported the relative file issue in #34

[JustinBeckwith]

Greetings, and thanks for having a look! I actually wrote this to replace our usage of broken-link-checker :) Just to make sure I'm headed down the right path, a few questions:

An option to skip external links. In our case this speeds things up a lot

So if you were to scan http://mysite, and your site had a hyperlink to www.google.com - are you saying you don't want the tool to check that? Like, ignore anything not on the origin domain?

An option to control concurrency

So as of today there's no need for concurrency control, because everything is 100% serial 🤣 I filed #35 to track parallelizing stuffs

Maybe print the total time spent and/or total links that were checked even when an error occurs?

Yeah, I wasn't sure what the right thing to do there was. Filed #36 to track.

It seems the tool follows redirects. Can you show it in the output too?

Good call! Tracking this in #37

[XhmikosR]

So if you were to scan http://mysite, and your site had a hyperlink to www.google.com - are you saying you don't want the tool to check that? Like, ignore anything not on the origin domain?

Yup, exactly. The reason is that this is a lot slower and some times breaks our builds temporarily but still.

If you check https://travis-ci.org/twbs/bootstrap/jobs/494434529 you will see it takes a lot of time to finish the scan, while with blc without external links, it only takes a few seconds :)

[JustinBeckwith]

Got it! So technically the --skip parameter accepts a regex, so I think this is possible today with no changes 😸 Can you give something like this a shot?

linkinator http://jbeckwith.com --skip '^(?!http://jbeckwith.com)'

[XhmikosR]

We need better docs for sure :)

That being said, this indeed works

linkinator _gh_pages --recurse --skip "^(?!http://localhost)"

Double quotes for Windows :)

Still, an option could be very handy.

[XhmikosR]

Another thing I'm thinking is the output. Maybe there should be an option to make it less verbose?

For example see https://travis-ci.org/twbs/bootstrap/jobs/494599205#L844

Personally I'd be happy to see something simpler and compact. The skipped links don't give me any value, assuming I trust linkinator --skip to do its job :)

Maybe it'll give you some ideas.

[JustinBeckwith]

Thanks for the awesome feedback. Just to keep this all straight, I created follow up issues to track the other feature requests.