doc: setup CRAB documentation #17
Conversation
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (8)
docs/_static/CRAB_logo1.pngis excluded by!**/*.png,!**/*.pngdocs/_static/benchmark_config.pngis excluded by!**/*.png,!**/*.pngdocs/_static/crab_overview.pngis excluded by!**/*.png,!**/*.pngdocs/_static/favicon.pngis excluded by!**/*.png,!**/*.pngdocs/crab_benchmark_v0/assets/android_1.pngis excluded by!**/*.png,!**/*.pngdocs/crab_benchmark_v0/assets/android_2.pngis excluded by!**/*.png,!**/*.pngpoetry.lockis excluded by!**/*.lock,!**/*.lockpyproject.tomlis excluded by!**/*.toml
Files selected for processing (9)
- .github/workflows/documentation.yml (1 hunks)
- docs/conf.py (2 hunks)
- docs/crab.benchmarks.rst (2 hunks)
- docs/crab_benchmark_v0/environment_gcp_setup.md (1 hunks)
- docs/crab_benchmark_v0/environment_local_setup.md (1 hunks)
- docs/crab_benchmark_v0/get_started.md (1 hunks)
- docs/get_started/build_your_own_benchmark.md (1 hunks)
- docs/get_started/quickstart.md (1 hunks)
- docs/index.rst (1 hunks)
Files skipped from review due to trivial changes (5)
- docs/crab.benchmarks.rst
- docs/crab_benchmark_v0/environment_gcp_setup.md
- docs/crab_benchmark_v0/environment_local_setup.md
- docs/get_started/build_your_own_benchmark.md
- docs/get_started/quickstart.md
Additional context used
Ruff
docs/conf.py
60-60: f-string without any placeholders
Remove extraneous
fprefix(F541)
LanguageTool
docs/crab_benchmark_v0/get_started.md
[grammar] ~14-~14: In standard English, the noun “softwares” is uncountable. Consider using an alternative.
Context: ...lish a disk image contains all required softwares and configurations on google cloud, you...(SOFTWARES)
[grammar] ~14-~14: The name of this company should be capitalized.
Context: ...equired softwares and configurations on google cloud, you can use your own google acco...(GOOGLE_PRODUCTS)
[grammar] ~14-~14: “Google” is a proper noun and needs to be capitalized.
Context: ...s on google cloud, you can use your own google account to create a cloud computer thro...(A_GOOGLE)
Markdownlint
docs/crab_benchmark_v0/get_started.md
9-9: Expected: h3; Actual: h4
Heading levels should only increment by one level at a time(MD001, heading-increment)
Additional comments not posted (12)
.github/workflows/documentation.yml (4)
3-4: Good branch targeting for documentation deployment.Triggering the workflow on pushes to the
mainbranch ensures that the documentation is updated in sync with the primary branch of the project.
6-7: Ensure appropriate permissions for the workflow.The workflow has
writepermissions oncontents, which is necessary for deploying to GitHub Pages.
13-21: Setup and build steps are well-defined.Using a custom action for Python setup (
camel_install) is an interesting approach. Ensure this action properly configures the environment for Sphinx.Please verify that the
.github/actions/camel_installaction is configured correctly for Python 3.10 and necessary dependencies.
23-29: Deployment step configuration looks correct.Using
peaceiris/actions-gh-pages@v3with conditions to deploy only on main branch updates is a good practice. Theforce_orphanoption ensures a clean history on thegh-pagesbranch, which is typically desirable for documentation.docs/index.rst (2)
9-16: Enhanced documentation structure with newtoctreesections.Adding sections for "Get Started with CRAB" and linking to relevant guides improves user engagement and accessibility.
17-24: Well-organized section for benchmarks.The new
toctreefor "CRAB Benchmark-v0" is well-structured, providing clear paths to setup guides and benchmarks. Ensure all linked documents are present and correctly formatted.docs/conf.py (3)
32-36: Updated project metadata reflects new branding.Changing the project name to 'CRAB' and updating versioning details aligns with the new branding strategy.
44-45: Addition ofmyst_parserfor Markdown support.This extension allows for more flexible content creation using Markdown, which can enhance the documentation's accessibility and ease of maintenance.
56-64: Enhanced HTML output settings for better user experience.The switch to
sphinx_book_themeand the addition of new theme options like a repository button improve the visual appeal and functionality of the documentation site.Tools
Ruff
60-60: f-string without any placeholders
Remove extraneous
fprefix(F541)
docs/crab_benchmark_v0/get_started.md (3)
1-7: Introduction and Concept sections are well-written.These sections provide a clear and concise overview of the benchmark and its components.
18-21: Task section is comprehensive and well-structured.This section effectively communicates how tasks are managed and defined, providing useful links to further resources.
24-30: Clarify experiment setup instructions and correct markdown heading.The content is clear and provides a detailed step-by-step guide. However, the markdown heading level is inconsistent.
- Change the heading from h4 to h3 to maintain consistency.
Proposed change:
-#### Environment +### EnvironmentLikely invalid or redundant comment.
WalkthroughThe overall changes encompass the introduction of a GitHub Actions workflow for automated documentation build and deployment, updates to project metadata in configuration files, enhancements in README clarity, and structural improvements in various documentation files. Notably, several submodule documentations were removed, and new setup guides were added to streamline user navigation and support. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant GitHub Actions
participant Docs Build
participant Deployment
User->>GitHub Actions: Push to main
GitHub Actions->>Docs Build: Checkout code
Docs Build->>Docs Build: Set up Python environment
Docs Build->>Docs Build: Build documentation
Docs Build->>Deployment: Deploy to gh-pages
Deployment->>GitHub: Update gh-pages branch
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 4
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (7)
docs/_static/CRAB_logo1.pngis excluded by!**/*.pngdocs/_static/benchmark_config.pngis excluded by!**/*.pngdocs/_static/crab_overview.pngis excluded by!**/*.pngdocs/_static/favicon.pngis excluded by!**/*.pngdocs/crab_benchmark_v0/assets/android_1.pngis excluded by!**/*.pngdocs/crab_benchmark_v0/assets/android_2.pngis excluded by!**/*.pngpoetry.lockis excluded by!**/*.lock
Files selected for processing (11)
- .github/workflows/documentation.yml (1 hunks)
- crab-benchmark-v0/README.md (1 hunks)
- docs/conf.py (2 hunks)
- docs/crab.benchmarks.rst (2 hunks)
- docs/crab_benchmark_v0/environment_gcp_setup.md (1 hunks)
- docs/crab_benchmark_v0/environment_local_setup.md (1 hunks)
- docs/crab_benchmark_v0/get_started.md (1 hunks)
- docs/get_started/build_your_own_benchmark.md (1 hunks)
- docs/get_started/quickstart.md (1 hunks)
- docs/index.rst (1 hunks)
- pyproject.toml (1 hunks)
Files skipped from review due to trivial changes (1)
- docs/crab_benchmark_v0/environment_gcp_setup.md
Files skipped from review as they are similar to previous changes (6)
- .github/workflows/documentation.yml
- docs/conf.py
- docs/crab_benchmark_v0/environment_local_setup.md
- docs/get_started/build_your_own_benchmark.md
- docs/get_started/quickstart.md
- docs/index.rst
Additional context used
LanguageTool
crab-benchmark-v0/README.md
[grammar] ~16-~16: The name of this company should be capitalized.
Context: ...required software and configurations on google cloud, you can use your own google acco...(GOOGLE_PRODUCTS)
[grammar] ~16-~16: “Google” is a proper noun and needs to be capitalized.
Context: ...s on google cloud, you can use your own google account to create a cloud computer thro...(A_GOOGLE)
docs/crab_benchmark_v0/get_started.md
[grammar] ~14-~14: The name of this company should be capitalized.
Context: ...required software and configurations on google cloud, you can use your own google acco...(GOOGLE_PRODUCTS)
[grammar] ~14-~14: “Google” is a proper noun and needs to be capitalized.
Context: ...s on google cloud, you can use your own google account to create a cloud computer thro...(A_GOOGLE)
[uncategorized] ~20-~20: The grammatical number of this noun doesn’t look right. Consider replacing it.
Context: ...untu directories are single-environment task and those in cross directory are cross-...(AI_EN_LECTOR_REPLACEMENT_NOUN_NUMBER)
Markdownlint
docs/crab_benchmark_v0/get_started.md
9-9: Expected: h3; Actual: h4
Heading levels should only increment by one level at a time(MD001, heading-increment)
Additional comments not posted (4)
docs/crab.benchmarks.rst (1)
Line range hint
1-12: Review the removal of submodule documentation.The documentation for several submodules has been removed. Ensure this aligns with the project's documentation strategy and that these modules are either deprecated or documented elsewhere.
pyproject.toml (1)
114-116: Assess dependency changes in documentation tools.The shift from
sphinx-rtd-themetosphinx-book-themeand the loosening of the version constraint onsphinxsuggest a significant change in documentation styling and compatibility. Ensure these changes are tested to prevent issues with documentation generation.crab-benchmark-v0/README.md (1)
Line range hint
1-4: Approve grammatical corrections in README.The grammatical corrections improve the professionalism of the documentation. Ensure all changes are consistent throughout the document.
Tools
LanguageTool
[grammar] ~16-~16: The name of this company should be capitalized.
Context: ...required software and configurations on google cloud, you can use your own google acco...(GOOGLE_PRODUCTS)
[grammar] ~16-~16: “Google” is a proper noun and needs to be capitalized.
Context: ...s on google cloud, you can use your own google account to create a cloud computer thro...(A_GOOGLE)
docs/crab_benchmark_v0/get_started.md (1)
24-30: Content looks good for the Experiment section.The instructions are clear and provide a good overview of the experimental process.
| @@ -13,7 +13,7 @@ Our benchmark contains two important parts: **Environments** and **Tasks**. | |||
| Since our Ubuntu environment is built upon KVM, setting it up locally requires you an experienced Linux user to deal with many small and miscellaneous issues. Therefore, we provide two environment setup methods: | |||
|
|
|||
| * [Local setup](./docs/environment_local_setup.md) provides you a step-by-step guideline to build environments on a Linux Machine with **at least one monitor and 32G memory**, but it doesn't cover details like how to install KVM on your machine because they are various on different Linux distros. | |||
| * For those who want a quicker setup, we also provide a setup through [Google Clould Platform](./docs/environment_gcp_setup.md). Specifically, we publish a disk image contains all required softwares and configurations on google cloud, you can use your own google account to create a cloud computer through this disk image and use [google remote desktop](https://remotedesktop.google.com/access/) to connect to it. This method doesn't have any hardware limitations and when you set it up you can run the experiment immediately. As a tradeoff, the cloud computer that meets the minimum hardware requirement costs around $0.4 per hour (depend on the machine zone). | |||
| * For those who want a quicker setup, we also provide a setup through [Google Clould Platform](./docs/environment_gcp_setup.md). Specifically, we publish a disk image contains all required software and configurations on google cloud, you can use your own google account to create a cloud computer through this disk image and use [google remote desktop](https://remotedesktop.google.com/access/) to connect to it. This method doesn't have any hardware limitations and when you set it up you can run the experiment immediately. As a tradeoff, the cloud computer that meets the minimum hardware requirement costs around $0.4 per hour (depend on the machine zone). | |||
There was a problem hiding this comment.
Capitalize "Google" in the cloud setup description.
The term "Google" should be capitalized as it is a proper noun.
Apply this correction:
- on google cloud
+ on Google cloudCommittable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| * For those who want a quicker setup, we also provide a setup through [Google Clould Platform](./docs/environment_gcp_setup.md). Specifically, we publish a disk image contains all required software and configurations on google cloud, you can use your own google account to create a cloud computer through this disk image and use [google remote desktop](https://remotedesktop.google.com/access/) to connect to it. This method doesn't have any hardware limitations and when you set it up you can run the experiment immediately. As a tradeoff, the cloud computer that meets the minimum hardware requirement costs around $0.4 per hour (depend on the machine zone). | |
| * For those who want a quicker setup, we also provide a setup through [Google Clould Platform](./docs/environment_gcp_setup.md). Specifically, we publish a disk image contains all required software and configurations on Google cloud, you can use your own google account to create a cloud computer through this disk image and use [google remote desktop](https://remotedesktop.google.com/access/) to connect to it. This method doesn't have any hardware limitations and when you set it up you can run the experiment immediately. As a tradeoff, the cloud computer that meets the minimum hardware requirement costs around $0.4 per hour (depend on the machine zone). |
Tools
LanguageTool
[grammar] ~16-~16: The name of this company should be capitalized.
Context: ...required software and configurations on google cloud, you can use your own google acco...(GOOGLE_PRODUCTS)
[grammar] ~16-~16: “Google” is a proper noun and needs to be capitalized.
Context: ...s on google cloud, you can use your own google account to create a cloud computer thro...(A_GOOGLE)
|
|
||
| Our benchmark contains two important parts: **Environments** and **Tasks**. | ||
|
|
||
| #### Environment |
There was a problem hiding this comment.
Correct markdown heading level.
Heading levels should only increment by one level at a time to maintain proper structure and readability.
Change the heading from level 4 to level 3:
- #### Environment
+ ### EnvironmentCommittable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| #### Environment | |
| ### Environment |
Tools
Markdownlint
9-9: Expected: h3; Actual: h4
Heading levels should only increment by one level at a time(MD001, heading-increment)
| Since our Ubuntu environment is built upon KVM, setting it up locally requires you an experienced Linux user to deal with many small and miscellaneous issues. Therefore, we provide two environment setup methods: | ||
|
|
||
| * [Local setup](./environment_local_setup.md) provides you a step-by-step guideline to build environments on a Linux Machine with **at least one monitor and 32G memory**, but it doesn't cover details like how to install KVM on your machine because they are various on different Linux distros. | ||
| * For those who want a quicker setup, we also provide a setup through [Google Clould Platform](./environment_gcp_setup.md). Specifically, we publish a disk image contains all required software and configurations on google cloud, you can use your own google account to create a cloud computer through this disk image and use [google remote desktop](https://remotedesktop.google.com/access/) to connect to it. This method doesn't have any hardware limitations and when you set it up you can run the experiment immediately. As a tradeoff, the cloud computer that meets the minimum hardware requirement costs around $0.4 per hour (depend on the machine zone). |
There was a problem hiding this comment.
Correct capitalization errors.
Ensure proper nouns like "Google" are capitalized correctly to maintain professionalism and accuracy in the documentation.
Apply this diff to correct the capitalization:
- a disk image contains all required software and configurations on google cloud, you can use your own google account
+ a disk image contains all required software and configurations on Google Cloud, you can use your own Google accountCommittable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| * For those who want a quicker setup, we also provide a setup through [Google Clould Platform](./environment_gcp_setup.md). Specifically, we publish a disk image contains all required software and configurations on google cloud, you can use your own google account to create a cloud computer through this disk image and use [google remote desktop](https://remotedesktop.google.com/access/) to connect to it. This method doesn't have any hardware limitations and when you set it up you can run the experiment immediately. As a tradeoff, the cloud computer that meets the minimum hardware requirement costs around $0.4 per hour (depend on the machine zone). | |
| * For those who want a quicker setup, we also provide a setup through [Google Clould Platform](./environment_gcp_setup.md). Specifically, we publish a disk image contains all required software and configurations on Google Cloud, you can use your own Google account to create a cloud computer through this disk image and use [google remote desktop](https://remotedesktop.google.com/access/) to connect to it. This method doesn't have any hardware limitations and when you set it up you can run the experiment immediately. As a tradeoff, the cloud computer that meets the minimum hardware requirement costs around $0.4 per hour (depend on the machine zone). |
Tools
LanguageTool
[grammar] ~14-~14: The name of this company should be capitalized.
Context: ...required software and configurations on google cloud, you can use your own google acco...(GOOGLE_PRODUCTS)
[grammar] ~14-~14: “Google” is a proper noun and needs to be capitalized.
Context: ...s on google cloud, you can use your own google account to create a cloud computer thro...(A_GOOGLE)
|
|
||
| #### Task | ||
|
|
||
| We manage our task dataset using a CRAB-recommended method. Sub-tasks are defined through Pydantic models written in Python code, and composed tasks are defined in JSON format, typically combining several sub-tasks. The sub-tasks are defined in [android_subtasks](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/android_subtasks.py) and [ubuntu_subtasks](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/ubuntu_subtasks.py). The JSON files storing composed tasks are categorized into [android](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/android/), [ubuntu](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/ubuntu/), and [cross-platform](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/cross/). The tasks in android and ubuntu directories are single-environment task and those in cross directory are cross-environment tasks. Additionally, we create several tasks by hand instead of composing sub-tasks to provide semantically more meaningful tasks, which are found in [handmade tasks](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/handmade_tasks.py). |
There was a problem hiding this comment.
Correct grammatical number issue.
The description of tasks should use the plural form to match the context.
Apply this diff to correct the grammatical number:
- The tasks in android and ubuntu directories are single-environment task and those in cross directory are cross-environment tasks.
+ The tasks in android and ubuntu directories are single-environment tasks and those in cross directory are cross-environment tasks.Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| We manage our task dataset using a CRAB-recommended method. Sub-tasks are defined through Pydantic models written in Python code, and composed tasks are defined in JSON format, typically combining several sub-tasks. The sub-tasks are defined in [android_subtasks](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/android_subtasks.py) and [ubuntu_subtasks](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/ubuntu_subtasks.py). The JSON files storing composed tasks are categorized into [android](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/android/), [ubuntu](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/ubuntu/), and [cross-platform](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/cross/). The tasks in android and ubuntu directories are single-environment task and those in cross directory are cross-environment tasks. Additionally, we create several tasks by hand instead of composing sub-tasks to provide semantically more meaningful tasks, which are found in [handmade tasks](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/handmade_tasks.py). | |
| We manage our task dataset using a CRAB-recommended method. Sub-tasks are defined through Pydantic models written in Python code, and composed tasks are defined in JSON format, typically combining several sub-tasks. The sub-tasks are defined in [android_subtasks](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/android_subtasks.py) and [ubuntu_subtasks](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/ubuntu_subtasks.py). The JSON files storing composed tasks are categorized into [android](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/android/), [ubuntu](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/ubuntu/), and [cross-platform](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/cross/). The tasks in android and ubuntu directories are single-environment tasks and those in cross directory are cross-environment tasks. Additionally, we create several tasks by hand instead of composing sub-tasks to provide semantically more meaningful tasks, which are found in [handmade tasks](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0/dataset/handmade_tasks.py). |
Tools
LanguageTool
[uncategorized] ~20-~20: The grammatical number of this noun doesn’t look right. Consider replacing it.
Context: ...untu directories are single-environment task and those in cross directory are cross-...(AI_EN_LECTOR_REPLACEMENT_NOUN_NUMBER)
Summary by CodeRabbit
New Features
crab-benchmark-v0tool.Documentation Updates
Bug Fixes