[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

joyeecheung · 2024-10-01T16:42:48Z

This backports the following PRs that are related to the unflagging of --experimental-require-module:

Note that I don't think this backport PR is strictly required - the PRs seem to land cleanly on v22.x except the unflagging commit has a small conflict in the YAML changelog description. This PR is mostly opened to look for regressions in v22 with CITGM.

Refs: #52697

nodejs-github-bot · 2024-10-01T16:42:54Z

Review requested:

@nodejs/loaders

joyeecheung · 2024-10-01T16:46:41Z

CITGM for v22.x-staging: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3481/
CITGM for this PR: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3482/

joyeecheung · 2024-10-02T13:52:39Z

Some regressions found:

yarn is expecting ERR_REQUIRE_ESM in its tests, probably to check that they didn't do anything to alter Node.js module loading https://github.com/yarnpkg/berry/blob/33f249b6cc06db1803f03eb5f17306a23d11115c/packages/acceptance-tests/pkg-tests-specs/sources/pnp-esm.test.ts#L617
Gulp tests are breaking because they don't expect the experimental warning coming out of Node.js https://github.com/gulpjs/gulp/blob/2fa4981a910d7bdedb758bd09868620c9bb21d54/test/index.test.js#L90 (I am not sure how they load the .mjs configs, though it seems they are also doing a try-catch of require(config) somewhere)
2 also happened to Jest.
In local testing, I also noticed that the debug package (used by npm, etc.) has a try { require(esm) } catch { /* fallback */ } pattern here, which only gets improvements out of require(esm) support after the unflagging, but then the experimental warning would also break the output.

I think for backporting to v22.x we should probably do some changes:

a. Provides process.features.require_module for feature detection
b. Do not emit the experimental warning if it's coming out of node_modules

b would not fix the test regressions 2-4, the packages still need to update their tests which is to be expected. But at least it would not break the CLIs for end users.

joyeecheung · 2024-10-02T14:58:59Z

As for esm, from what I can tell in standard-things/esm#883 it seems to have been broken in different ways since many major releases of Node.js ago. The regression that predated unflagging of require(esm) I found in #55085 (comment) also seem to come from some TDZ issue in the way the package is written (it relies on access to Node.js internals and seem to expect certain internals to be invoked in certain order). It doesn't seem to be very cost-effective to do anything about it on v22.

richardlau · 2024-10-02T15:27:11Z

I think for backporting to v22.x we should probably do some changes:

a. Provides process.features.require_module for feature detection
b. Do not emit the experimental warning if it's coming out of node_modules

b would not fix the test regressions 2-4, the packages still need to update their tests which is to be expected. But at least it would not break the CLIs for end users.

These SGTM

joyeecheung · 2024-10-02T16:33:21Z

Opened #55241 for feature detection.

joyeecheung · 2024-10-04T10:39:20Z

Added a commit from #55243 and a commit for v22.x specifically (and maybe v20.x in the future) to disable the warning when require() comes from node_modules.

targos · 2024-10-04T11:58:50Z

I just updated v22.x-staging so this now has conflicts because of duplicated commits I guess.

joyeecheung · 2024-10-07T15:49:07Z

Rebased to drop the duplicate commits. Also cherry-picked #55243 and #55241 from the main branch. I think for backport to v22.x it is ready, note that the last commit (silence the experimental warning when it comes from node_modules) is only targeting v22.x, as proposed in #55217 (comment) and agreed in the last TSC meeting.

nodejs-github-bot · 2024-10-07T15:56:13Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62972/

aduh95 · 2024-10-08T10:47:39Z

Can you rebase please?

mcollina

lgtm

nicolo-ribaudo · 2024-11-07T17:17:13Z

Backporting the unflagging without backporting #55502 and #55520 at the same time would break everybody using a Babel version that is more than a few weeks old and released in 2024.

This unflags --experimental-require-module so require(esm) can be used without the flag. For now, when require() actually encounters an ESM, it will still emit an experimental warning. To opt out of the feature, --no-experimental-require-module can be used. There are some tests specifically testing ERR_REQUIRE_ESM. Some of them are repurposed to test --no-experimental-require-module. Some of them are modified to just expect loading require(esm) to work, when it's appropriate. PR-URL: nodejs#55085 Backport-PR-URL: nodejs#55217 Refs: nodejs#52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Filip Skokan <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: Richard Lau <[email protected]>

This is faster and more consistent with other places using the regular expression to detect node_modules. PR-URL: nodejs#55243 Backport-PR-URL: nodejs#55217 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Refs: nodejs#52697

Some packages have been using try-catch to load require(esm) in environments that are available. In 23, where require(esm) is unflagged, we emit an experimental warning when require() is used to load ESM. To backport require(esm) to older LTS releases, however, this could break existing CLI output. To reduce the disruption for LTS, on older release lines, this commit is applied to skip the warning if require(esm) comes from node_modules. This warning will be eventually removed when require(esm) becomes stable. PR-URL: nodejs#55217 Refs: nodejs#52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

joyeecheung · 2024-11-07T17:31:05Z

@nicolo-ribaudo Thanks for pointing that out. I think we should wait for those patches to qualify to unflag require(esm). If it misses the next semver-minor train, then we can wait for the next one.

This unflags --experimental-require-module so require(esm) can be used without the flag. For now, when require() actually encounters an ESM, it will still emit an experimental warning. To opt out of the feature, --no-experimental-require-module can be used. There are some tests specifically testing ERR_REQUIRE_ESM. Some of them are repurposed to test --no-experimental-require-module. Some of them are modified to just expect loading require(esm) to work, when it's appropriate. PR-URL: nodejs#55085 Backport-PR-URL: nodejs#55217 Refs: nodejs#52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Filip Skokan <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: nodejs#54563 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

This is faster and more consistent with other places using the regular expression to detect node_modules. PR-URL: nodejs#55243 Backport-PR-URL: nodejs#55217 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Refs: nodejs#52697

Some packages have been using try-catch to load require(esm) in environments that are available. In 23, where require(esm) is unflagged, we emit an experimental warning when require() is used to load ESM. To backport require(esm) to older LTS releases, however, this could break existing CLI output. To reduce the disruption for LTS, on older release lines, this commit is applied to skip the warning if require(esm) comes from node_modules. This warning will be eventually removed when require(esm) becomes stable. PR-URL: nodejs#55217 Refs: nodejs#52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

joyeecheung · 2024-11-12T01:23:58Z

Updated the branch a bit and included the following commits:

module: unflag --experimental-require-module
module: support 'module.exports' interop export in require(esm)
doc: update `require(ESM)` history and stability status
module: use kNodeModulesRE to detect node_modules
module: check --experimental-require-module separately from detection
src: implement IsInsideNodeModules() in C++
module: include module information in require(esm) warning
module: do not warn when require(esm) comes from node_modules
module: allow ESM that failed to be required to be re-imported
module: trim off internal stack frames for require(esm) warnings

PR-URL: nodejs#55199 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

This is faster and more consistent with other places using the regular expression to detect node_modules. PR-URL: nodejs#55243 Backport-PR-URL: nodejs#55217 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Refs: nodejs#52697

Previously we assumed if `--experimental-detect-module` is true, then `--experimental-require-module` is true, which isn't the case, as the two can be enabled/disabled separately. This patch fixes the checks so `--no-experimental-require-module` is still effective when `--experimental-detect-module` is enabled. Drive-by: make the assertion messages more informative and remove obsolete TODO about allowing TLA in entrypoints handled by require(esm). PR-URL: nodejs#55250 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]>

This previously compiles a script and run it in a new context to avoid global pollution, which is more complex than necessary and can be too slow for it to be reused in other cases. The new implementation just checks the frames in C++ which is safe from global pollution, faster and simpler. The previous implementation also had a bug when the call site is in a ESM, because ESM have URLs as their script names, which don't start with '/' or '\' and will be skipped. The new implementation removes the skipping to fix it for ESM. PR-URL: nodejs#55286 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>

When emitting the experimental warning for `require(esm)`, include information about the parent module and the module being require()-d to help users locate and update them. PR-URL: nodejs#55397 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>

Some packages have been using try-catch to load require(esm) in environments that are available. In 23, where require(esm) is unflagged, we emit an experimental warning when require() is used to load ESM. To backport require(esm) to older LTS releases, however, this could break existing CLI output. To reduce the disruption for LTS, on older release lines, this commit is applied to skip the warning if require(esm) comes from node_modules. This warning will be eventually removed when require(esm) becomes stable. PR-URL: nodejs#55217 Refs: nodejs#52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

When a ESM module cannot be loaded by require due to the presence of TLA, its module status would be stopped at kInstantiated. In this case, when it's imported again, we should allow it to be evaluated asynchronously, as it's also a common pattern for users to retry with dynamic import when require fails. PR-URL: nodejs#55502 Fixes: nodejs#55500 Refs: nodejs#52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Chemi Atlow <[email protected]>

Trim off irrelevant internal stack frames for require(esm) warnings so it's easier to locate where the call comes from when --trace-warnings is used. PR-URL: nodejs#55496 Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Paolo Insogna <[email protected]>

github-actions · 2024-11-12T01:41:16Z

Failed to start CI

   ⚠  Something was pushed to the Pull Request branch since the last approving review.
   ✘  Refusing to run CI on potentially unsafe PR

https://github.com/nodejs/node/actions/runs/11789132997

nodejs-github-bot · 2024-11-15T07:31:39Z

CI: https://ci.nodejs.org/job/node-test-pull-request/63551/

nodejs-github-bot · 2024-11-15T11:49:25Z

CI: https://ci.nodejs.org/job/node-test-pull-request/63552/

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. v22.x v22.x Issues that can be reproduced on v22.x or PRs targeting the v22.x-staging branch. labels Oct 1, 2024

RedYetiDev added module Issues and PRs related to the module subsystem. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. labels Oct 1, 2024

RedYetiDev added the needs-citgm PRs that need a CITGM CI run. label Oct 1, 2024

tats-u mentioned this pull request Oct 4, 2024

Enable consuming of ES modules in extensions microsoft/vscode#130367

Open

targos mentioned this pull request Oct 5, 2024

doc: fix typos #55066

Merged

This was referenced Oct 5, 2024

src: implement IsInsideNodeModules() in C++ #55286

Merged

Tracking issue: require(esm) #52697

Open

joyeecheung force-pushed the backport-require-esm-2 branch from dda6fc3 to 9c282c3 Compare October 7, 2024 15:45

joyeecheung marked this pull request as ready for review October 7, 2024 15:47

joyeecheung force-pushed the backport-require-esm-2 branch from 9c282c3 to 3d0afe0 Compare October 7, 2024 15:52

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Oct 7, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Oct 7, 2024

joyeecheung force-pushed the backport-require-esm-2 branch 2 times, most recently from 5ed6462 to 898396a Compare October 8, 2024 19:00

mcollina approved these changes Oct 9, 2024

View reviewed changes

joyeecheung force-pushed the backport-require-esm-2 branch from 898396a to 5ea60dc Compare October 9, 2024 17:08

joyeecheung force-pushed the backport-require-esm-2 branch from 71c2d02 to 0bf18f3 Compare November 7, 2024 17:26

arthur-fontaine mentioned this pull request Nov 9, 2024

Refactoring: study replacement of import-sync by native require(esm) arthur-fontaine/agrume#129

Open

joyeecheung and others added 2 commits November 12, 2024 08:53

module: support 'module.exports' interop export in require(esm)

4f33767

PR-URL: nodejs#54563 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

joyeecheung mentioned this pull request Nov 12, 2024

module: support 'module.exports' interop export name in require(esm) #54563

Closed

joyeecheung force-pushed the backport-require-esm-2 branch from 0bf18f3 to 274b3bb Compare November 12, 2024 01:21

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Nov 12, 2024

aduh95 and others added 8 commits November 12, 2024 09:31

doc: update require(ESM) history and stability status

4799d9a

PR-URL: nodejs#55199 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

joyeecheung force-pushed the backport-require-esm-2 branch from 274b3bb to f7bdcee Compare November 12, 2024 01:31

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Nov 12, 2024

github-actions bot added the request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. label Nov 12, 2024

panva removed the request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. label Nov 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

joyeecheung commented Oct 1, 2024 •

edited

Loading

nodejs-github-bot commented Oct 1, 2024

joyeecheung commented Oct 1, 2024

joyeecheung commented Oct 2, 2024 •

edited

Loading

joyeecheung commented Oct 2, 2024 •

edited

Loading

richardlau commented Oct 2, 2024

joyeecheung commented Oct 2, 2024

joyeecheung commented Oct 4, 2024

targos commented Oct 4, 2024

joyeecheung commented Oct 7, 2024

nodejs-github-bot commented Oct 7, 2024

aduh95 commented Oct 8, 2024

mcollina left a comment

nicolo-ribaudo commented Nov 7, 2024 •

edited

Loading

joyeecheung commented Nov 7, 2024

joyeecheung commented Nov 12, 2024

github-actions bot commented Nov 12, 2024

nodejs-github-bot commented Nov 15, 2024

nodejs-github-bot commented Nov 15, 2024

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

Are you sure you want to change the base?

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

Conversation

joyeecheung commented Oct 1, 2024 • edited Loading

nodejs-github-bot commented Oct 1, 2024

joyeecheung commented Oct 1, 2024

joyeecheung commented Oct 2, 2024 • edited Loading

joyeecheung commented Oct 2, 2024 • edited Loading

richardlau commented Oct 2, 2024

joyeecheung commented Oct 2, 2024

joyeecheung commented Oct 4, 2024

targos commented Oct 4, 2024

joyeecheung commented Oct 7, 2024

nodejs-github-bot commented Oct 7, 2024

aduh95 commented Oct 8, 2024

mcollina left a comment

Choose a reason for hiding this comment

nicolo-ribaudo commented Nov 7, 2024 • edited Loading

joyeecheung commented Nov 7, 2024

joyeecheung commented Nov 12, 2024

github-actions bot commented Nov 12, 2024

nodejs-github-bot commented Nov 15, 2024

nodejs-github-bot commented Nov 15, 2024

joyeecheung commented Oct 1, 2024 •

edited

Loading

joyeecheung commented Oct 2, 2024 •

edited

Loading

joyeecheung commented Oct 2, 2024 •

edited

Loading

nicolo-ribaudo commented Nov 7, 2024 •

edited

Loading