[Feature Request]: add robots.txt for crawlers

[Anandha-Vihari]

Is there an existing issue for this?

• I have searched the existing issues

Feature Description

add robots.txt for crawlers to index sitemap

Use Case

for crawlers and search engines

Benefits

No response

Priority

High

Record

• I agree to follow this project's Code of Conduct
• I'm a GSSOC contributor
• I want to work on this issue
• I'm willing to provide further clarification or assistance if needed.

[github-actions]

🙌 Thank you for bringing this issue to our attention! We appreciate your input and will investigate it as soon as possible !

[github-actions]

✅ This issue has been closed. Thank you for your contribution! If you have any further questions or issues, feel free to raise them, and we can discuss more!