robots.txt future

[addison74]

Recently we approved a PR that came with a new version for robots.txt file. I analyzed the access logs of the servers I manage and found that this file was requested only 201 times last month. This is the highest value. It is obvious that it is no longer relevant and that every bot/crawler does what it wants inside the website.

First I would remove this file because it no longer offers anything. Secondly I would create more rules in .htaccess (if Apache webserver is used) for bots/crawlers so that not so much content is collected.

The robots.txt file does not help to reduce the bandwidth, on the contrary just check and you will find that it is ignored and so a bot can collect information whenever it wants.

[fballiano]

the robots.txt is cached by search engines and checked only sometimes, so if you didn't have one it will take sometime for them to pick it up, but also don't expect to have thousands of request daily, it won't happen.

It's true anyway that bad players will not respect it, but I wouldn't remove it, it's still useful for big search engines.