Facebook link cache/preview suddenly not working.

[jackrt743]

Shlink version

4.0.0

PHP version

8.3.3

How do you serve Shlink

Self-hosted Apache

Database engine

MySQL

Database version

8.0.36

Summary

Hi all!

So - installed shlink over Christmas to replace rebrandly and it's been brillant!

Up until last weekend i've added my shortened links onto Facebook and they've 'translated' fine into the redirected links (text and image etc). Now it no longer wants to cache/preview the links correctly. It will not work with new short links, and older short links show a cached redirected link from a while back (I update them regularly). This seemed to coincide with a bit of drama I had with Lets Encrypt on the box which I had to redo all the certificates on the box including the one used for my shlink site.

I've been using the Facebook Sharing Debugger to try and make some sense out of this problem:

https://developers.facebook.com/tools/debug/

...and I notice that any of my shortlinks show the same error:

Bad Response Code
URL returned a bad HTTP response code.
Response Code	403
Response Code Reason	This response code could be due to a robots.txt block. Please allowlist facebookexternalhit on your sites robots.txt config to utilize Facebook scraping

The weird thing is though, my short links still work fine to use normally and even translate OK on Twitter fine like they used to on Facebook. I can also see the requests from the Facebook 'scrapers' in the Apache logs looking all good:

69.171.249.9 - - [16/Mar/2024:11:22:09 +0000] "GET /onward007 HTTP/1.1" 301 4149 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
69.171.249.116 - - [16/Mar/2024:11:22:09 +0000] "GET /onward007 HTTP/1.1" 301 4149 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"

I'm not exactly sure what happened - so was wondering if anyone else was experiencing the same or had any pointers? Could it be something to do with the Lets Encrypt certs, or is that a coincidence? I followed the guidance in Step 6 from this document for Lets Encrypt config: https://www.linuxbabe.com/ubuntu/install-shlink-url-shortener-ubuntu-20-04-server

I also upgraded to v4 from v3.7.1 and still the same issue.

Let me know if you need any more info!

Thanks, Jack

[RyanNutt]

I'm seeing the same thing after upgrading from an old version of shlink to 4.0.3, and then to 4.1.0 this morning. Same thing. I'm seeing the 301 log hits from facebookexternalhit. If I enter the link in Chrome, it works fine. It's just Facebook either in the debugger, posting new links, or old Facebook posts.

Also seeing the same if I try to create the post either directly through Facebook or through Buffer. Either way it doesn't seem to get what it needs.

Link was originally pointing to Gumroad, so I changed it to the same product on TpT to try to parse out whether it was a Gumroad issue. Same problem after changing. Tried with a new short link to example.com and same issue.

I went through the ./vendor/bin/shlink-installer upgrade process going from

• 2.10.3
• 3.0.3
• 3.1.2
• 3.2.1
• 3.3.2
• 3.4.0
• 3.5.4
• 3.6.4
• 3.7.4
• 4.0.3
• 4.1.0

Haven't had an issues with LetsEncrypt, so we don't have that in common.

Also happening on a server running 4.0.3 and 3.6.3.

[RyanNutt]

Hoping this is related: https://developers.facebook.com/support/bugs/1085796049211789/

Looks like the there is a robots.txt route in shlink that disallows everything, except short links where you've clicked crawlable. Those are added to robots.txt. Not sure why you wouldn't want your short links crawlable since they're just redirect.

Just ran a query to update all the links to crawlable and will see if that takes care of it. Seems like it should.

Now it's just a waiting game for Facebook to download and recache robots.txt and see if that clears it up.

Edit:

That seems to have done the trick. I put the URL for robots.txt into the Meta sharing debugger to get it crawled and then waited a bit and tried a short link and it worked like it should.

[acelaya]

I guess adding an option so Shlink to make every URL crawlable by default should not be something hard to implement.

I put together a draft. It was not too complex #2107

A few things are missing, but that would allow to make Shlink always return a robots.txt looking like this:

# For more information about the robots.txt standard, see:
# https://www.robotstxt.org/orig.html

User-agent: *
Disallow: /rest

[RyanNutt]

I was going to ask that question. If I'm okay with everything being crawlable, would blocking the /rest folder be enough? I haven't dug through the routes to see what all is there, but it seems easy enough to just copy and paste what you wrote into an actual robots.txt file in the public folder and skip routing for that file.

[acelaya]

If I'm okay with everything being crawlable, would blocking the /rest folder be enough?

Yes

it seems easy enough to just copy and paste what you wrote into an actual robots.txt file in the public folder and skip routing for that file.

You can temporarily do that, but remember any manual change will be gone when you update Shlink.

[RyanNutt]

Thanks again. I just went in and manually updated the crawlable field in the database so they're all crawlable.

And, since I haven't said it yet, thanks for this app. I'd been using another one that I wasn't all that happy with, and was thinking about writing my own when I came across shlink. Haven't looked back since :)

[jackrt743]

Thanks @RyanNutt and @acelaya , that looks to have done the trick :)