Improve Web.CrawlWebsite with a Tool Response Prompt

[EricGustin]

Problem

When a crawl started asynchronously, the tool's response did not provide context on how the crawl's status could be checked, the crawl could be cancelled, or even that the data from the crawl could be retrieved.

PR Description

This PR provides 'llm instructions' in the Web.CrawlWebsite response. These instructions provide context to the language model. It can be thought of as a Tool Response Prompt.

This is the first tool to use llm instructions. We should think about how we can standardize and utilize 'llm_instructions' more in the future.

LLM Responses without Tool Response Prompt:

Example 1:

I successfully crawled the website you specified. If you need specific information or insights from this website, please let me know!

Example 2:

I've initiated a crawl of the specified website with your provided parameters. If you have specific data or content you need from this page, please let me know, and I'll do my best to assist you based on the information gathered.

LLM Responses with Tool Response Prompt:

Example1:

I've started the process of crawling the specified webpage. While it's currently underway, I have the capability to check the crawl's status,
retrieve the crawled data once available, or cancel the crawl if needed. Feel free to let me know what you would like me to do next!

Example2:

I've started crawling the website you specified. The process is currently in progress. Here are some things I can do next:

• Get Crawl Status: I can check the current status of the crawl job.
• Cancel the Crawl: If you decide not to proceed with this crawl, I can cancel it.
• Get Crawl Data: Once the crawl is complete, I can retrieve the data for you.

Please let me know if you'd like me to perform any of these actions!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

📢 Thoughts on this report? Let us know!