[TOOL-3562] Portal: Add llms.txt and llms-full.txt generation script #6372

MananTank · 2025-02-28T14:57:35Z

PR-Codex overview

This PR primarily focuses on enhancing the layout components and the extraction process for search data, including the addition of new files for LLM content and adjustments to the indexing strategy.

Detailed summary

Added public/llms.txt and public/llms-full.txt to .gitignore.
Removed data-noindex attribute from various layout components.
Added noLLM prop to DocLayout.
Modified extractSearchData to return LLM content.
Updated extractContent function to handle new LLM data.
Enhanced handling of data-noindex and data-no-llm attributes in content extraction.
Refactored ArticleCard and Code components to support new features.
Updated dependencies in package.json and pnpm-lock.yaml.

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

linear · 2025-02-28T14:57:38Z

TOOL-3562 Add llms.txt and llms-full.txt for Portal

changeset-bot · 2025-02-28T14:57:39Z

⚠️ No Changeset found

Latest commit: 8887e46

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

vercel · 2025-02-28T14:57:43Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs-v2	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 28, 2025 7:20pm
login	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 28, 2025 7:20pm
thirdweb_playground	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 28, 2025 7:20pm
thirdweb-www	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 28, 2025 7:20pm
wallet-ui	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 28, 2025 7:20pm

MananTank · 2025-02-28T14:57:51Z

[TOOL-3562] Portal: Add llms.txt and llms-full.txt generation script #6372 👈 (View in Graphite)
main

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

merge-queue - adds this PR to the back of the merge queue
hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

apps/portal/src/app/api/search/extraction/index.ts

codecov · 2025-02-28T15:01:20Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 54.60%. Comparing base (56301bc) to head (8887e46).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6372      +/-   ##
==========================================
- Coverage   54.60%   54.60%   -0.01%     
==========================================
  Files         874      874              
  Lines       54525    54525              
  Branches     3682     3680       -2     
==========================================
- Hits        29774    29772       -2     
- Misses      24656    24658       +2     
  Partials       95       95

Flag	Coverage Δ
packages	`54.60% <ø> (-0.01%)`	⬇️

see 1 file with indirect coverage changes

apps/portal/package.json

github-actions · 2025-02-28T15:03:11Z

size-limit report 📦

Path	Size	Loading time (3g)	Running time (snapdragon)	Total time
thirdweb (esm)	46.87 KB (0%)	938 ms (0%)	210 ms (+198.31% 🔺)	1.2 s
thirdweb (cjs)	127.15 KB (0%)	2.6 s (0%)	286 ms (+83.98% 🔺)	2.9 s
thirdweb (minimal + tree-shaking)	5.6 KB (0%)	112 ms (0%)	127 ms (+2308.79% 🔺)	238 ms
thirdweb/chains (tree-shaking)	506 B (0%)	10 ms (0%)	45 ms (+2745.5% 🔺)	55 ms
thirdweb/react (minimal + tree-shaking)	19.38 KB (0%)	388 ms (0%)	137 ms (+761.97% 🔺)	525 ms

apps/portal/src/app/api/search/extraction/index.ts

graphite-app · 2025-02-28T19:13:41Z

Merge activity

Feb 28, 2:13 PM EST: A user added this pull request to the Graphite merge queue.
Feb 28, 2:21 PM EST: A user merged this pull request with the Graphite merge queue.

…6372)  ## PR-Codex overview This PR focuses on enhancing the layout components and search data extraction in the application. It introduces new files for LLM content and modifies existing components to include additional properties for better indexing and rendering. ### Detailed summary - Added `public/llms.txt` and `public/llms-full.txt` files. - Removed `data-noindex` from `<div>` in multiple layout files. - Added `noLLM` property to `DocLayout` in various components. - Updated `extractSearchData` to return LLM content. - Modified `extractContent` to handle new LLM data. - Adjusted `ArticleCard` and `Code` components to include `data-noindex`. - Updated dependencies in `package.json` and `pnpm-lock.yaml`. > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}`

github-actions bot added the Portal Involves changes to the Portal (docs) codebase. label Feb 28, 2025

github-actions bot assigned MananTank Feb 28, 2025

MananTank marked this pull request as ready for review February 28, 2025 14:57

MananTank requested review from jnsdls, joaquim-verges and a team as code owners February 28, 2025 14:57

graphite-app bot reviewed Feb 28, 2025

View reviewed changes

apps/portal/src/app/api/search/extraction/index.ts Show resolved Hide resolved

vercel bot deployed to Preview – wallet-ui February 28, 2025 14:58 View deployment

vercel bot deployed to Preview – thirdweb_playground February 28, 2025 14:59 View deployment

vercel bot deployed to Preview – login February 28, 2025 14:59 View deployment

graphite-app bot reviewed Feb 28, 2025

View reviewed changes

apps/portal/package.json Outdated Show resolved Hide resolved

vercel bot deployed to Preview – thirdweb-www February 28, 2025 15:04 View deployment

vercel bot deployed to Preview – docs-v2 February 28, 2025 15:05 View deployment

MananTank force-pushed the 02-28-_tool-3562_portal_add_llms.txt_and_llms-full.txt_generation_script branch from 035af9b to f18bdcc Compare February 28, 2025 15:06

vercel bot deployed to Preview – wallet-ui February 28, 2025 15:09 View deployment

vercel bot deployed to Preview – login February 28, 2025 15:09 View deployment

graphite-app bot reviewed Feb 28, 2025

View reviewed changes

apps/portal/src/app/api/search/extraction/index.ts Outdated Show resolved Hide resolved

apps/portal/src/app/api/search/extraction/index.ts Outdated Show resolved Hide resolved

vercel bot deployed to Preview – thirdweb_playground February 28, 2025 15:10 View deployment

MananTank force-pushed the 02-28-_tool-3562_portal_add_llms.txt_and_llms-full.txt_generation_script branch 4 times, most recently from 1f5a10b to f8417b2 Compare February 28, 2025 15:44

jnsdls approved these changes Feb 28, 2025

View reviewed changes

MananTank force-pushed the 02-28-_tool-3562_portal_add_llms.txt_and_llms-full.txt_generation_script branch from f8417b2 to 02fadf7 Compare February 28, 2025 18:55

graphite-app bot reviewed Feb 28, 2025

View reviewed changes

apps/portal/src/app/api/search/extraction/index.ts Outdated Show resolved Hide resolved

graphite-app bot reviewed Feb 28, 2025

View reviewed changes

apps/portal/src/app/api/search/extraction/index.ts Outdated Show resolved Hide resolved

vercel bot deployed to Preview – login February 28, 2025 18:58 View deployment

vercel bot deployed to Preview – thirdweb_playground February 28, 2025 18:59 View deployment

vercel bot deployed to Preview – wallet-ui February 28, 2025 19:00 View deployment

vercel bot deployed to Preview – thirdweb-www February 28, 2025 19:05 View deployment

MananTank force-pushed the 02-28-_tool-3562_portal_add_llms.txt_and_llms-full.txt_generation_script branch from 02fadf7 to ebbc521 Compare February 28, 2025 19:05

vercel bot deployed to Preview – wallet-ui February 28, 2025 19:07 View deployment

vercel bot deployed to Preview – thirdweb_playground February 28, 2025 19:07 View deployment

vercel bot deployed to Preview – login February 28, 2025 19:08 View deployment

vercel bot deployed to Preview – thirdweb-www February 28, 2025 19:12 View deployment

vercel bot deployed to Preview – docs-v2 February 28, 2025 19:13 View deployment

graphite-app bot force-pushed the 02-28-_tool-3562_portal_add_llms.txt_and_llms-full.txt_generation_script branch from ebbc521 to 8887e46 Compare February 28, 2025 19:14

vercel bot deployed to Preview – wallet-ui February 28, 2025 19:15 View deployment

vercel bot deployed to Preview – thirdweb_playground February 28, 2025 19:15 View deployment

vercel bot deployed to Preview – docs-v2 February 28, 2025 19:15 View deployment

vercel bot deployed to Preview – login February 28, 2025 19:17 View deployment

vercel bot deployed to Preview – thirdweb-www February 28, 2025 19:20 View deployment

graphite-app bot merged commit 8887e46 into main Feb 28, 2025
32 checks passed

graphite-app bot deleted the 02-28-_tool-3562_portal_add_llms.txt_and_llms-full.txt_generation_script branch February 28, 2025 19:21

vercel bot deployed to Production – login February 28, 2025 19:24 View deployment

vercel bot deployed to Production – wallet-ui February 28, 2025 19:24 View deployment

vercel bot deployed to Production – thirdweb_playground February 28, 2025 19:26 View deployment

vercel bot deployed to Production – thirdweb-www February 28, 2025 19:29 View deployment

vercel bot deployed to Production – docs-v2 February 28, 2025 19:30 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TOOL-3562] Portal: Add llms.txt and llms-full.txt generation script #6372

[TOOL-3562] Portal: Add llms.txt and llms-full.txt generation script #6372

MananTank commented Feb 28, 2025 •

edited by pr-codex bot

Loading

linear bot commented Feb 28, 2025

changeset-bot bot commented Feb 28, 2025 •

edited

Loading

vercel bot commented Feb 28, 2025 •

edited

Loading

MananTank commented Feb 28, 2025

codecov bot commented Feb 28, 2025 •

edited

Loading

github-actions bot commented Feb 28, 2025 •

edited

Loading

graphite-app bot commented Feb 28, 2025 •

edited

Loading

[TOOL-3562] Portal: Add llms.txt and llms-full.txt generation script #6372

[TOOL-3562] Portal: Add llms.txt and llms-full.txt generation script #6372

Conversation

MananTank commented Feb 28, 2025 • edited by pr-codex bot Loading

PR-Codex overview

Detailed summary

linear bot commented Feb 28, 2025

changeset-bot bot commented Feb 28, 2025 • edited Loading

⚠️ No Changeset found

vercel bot commented Feb 28, 2025 • edited Loading

MananTank commented Feb 28, 2025

How to use the Graphite Merge Queue

codecov bot commented Feb 28, 2025 • edited Loading

Codecov Report

github-actions bot commented Feb 28, 2025 • edited Loading

size-limit report 📦

graphite-app bot commented Feb 28, 2025 • edited Loading

Merge activity

MananTank commented Feb 28, 2025 •

edited by pr-codex bot

Loading

changeset-bot bot commented Feb 28, 2025 •

edited

Loading

vercel bot commented Feb 28, 2025 •

edited

Loading

codecov bot commented Feb 28, 2025 •

edited

Loading

github-actions bot commented Feb 28, 2025 •

edited

Loading

graphite-app bot commented Feb 28, 2025 •

edited

Loading