ESM in Gatsby files

[LekoArts]

Welcome to the overarching discussion related to ECMAScript modules (ESM) in Gatsby files!

Here is where you can get the latest info on how to try out the feature while it's in development and share your feedback.

Motivation

The ECMAScript module format is the official TC39 standard for packaging JavaScript. For many years a different module format, CommonJS (CJS) was the de facto standard in Node.js. Today, Gatsby supports CJS in gatsby-config and gatsby-node but not ESM.

Support for ESM in gatsby-config and gatsby-node files is a highly requested feature in the community, and ecosystems that Gatsby interacts with like mdx and unified have switched to ESM, dropping CJS in their latest major releases.

We intend to enable support for both CommonJS and ESM in Gatsby files.

Direction

There are two main goals we need to meet:

• Avoid breaking changes for existing usage
• Match the official Node and TypeScript conventions

In order to do both, our direction is to adopt the .mjs and .mts file extensions for files written in ESM.

This allows us to not change the existing behavior in Gatsby (.js and .ts extensions are CJS) for current users, stay compliant with how Node and TypeScript officially handle ESM, and provide a smooth transition to a future where the default format of Gatsby files is ESM.

Current status

• Support for gatsby-config.mjs and gatsby-node.mjs is available in gatsby@next and starting with [email protected]
• Support for gatsby-config.mts and gatsby-node.mts is underway, check back again here in the future for more updates

Try it out

To try out ESM in your Gatsby files:

1. Install the next/latest version of Gatsby with npm install gatsby@latest --legacy-peer-deps
2. Change the extension of your gatsby-config.js and gatsby-node.js files to gatsby-config.mjs and gatsby-node.mjs
3. Refer to the Node.js documentation for how to change CJS features to ESM features (e.g. require becomes import)

Limitations

• Top-level await is not supported

Where can I ask questions and/or give feedback?

• Here in this discussion!
• Join the Gatsby Discord

[sandren]

I think this long anticipated feature finally arriving introduces an even bigger opportunity for Gatsby. 🙂

I'm probably one of Gatsby's biggest fans, but the honest truth is that when I open an unfamiliar Gatsby codebase for the first time, I feel a bit of apprehension. I just have absolutely no idea what I'm getting myself into. The gap between the developer experience of Gatsby 2 and Gatsby 5 is so incredibly wide that they feel like completely different frameworks. So until I open the package.json and scan through it to find the gatsby version, I'm nervous that I might be walking into an ancient codebase with a less than stellar developer experience.

But there's an easy way to fix that! 😊

Instead of introducing a clunky new file extension to support ESM, I propose that Gatsby gives the file structure a bit of a refresh for the Gatsby 2022 Glow Up and adopts the file naming conventions of the modern JavaScript ecosystem:

gatsby.browser.js
gatsby.config.js
gatsby.node.js
gatsby.server.js

Make this opt-in for ESM support in Gatsby 5 and a requirement for Gatsby 6. Afterwards it will be abundantly clear if a codebase is using "legacy" Gatsby (the static site generator) or modern Gatsby (the full-stack React framework).

I know it's a bit silly to have such strong opinions about file names and extensions, but now that Gatsby has caught up to or even surpassed its competitors in many ways, I'm confident that small details like these can make a larger long-term impact on Gatsby's future than even shipping new features. The story of Gatsby's evolution into a full-stack React framework has already been written. Now it's time to get the word out!

P.S. Please rename "Valhalla Content Hub" to "Gatsby Content Mesh". I think people are finally ready for the ✨ 𝓬𝓸𝓷𝓽𝓮𝓷𝓽 𝓶𝓮𝓼𝓱 ✨! 😅

[rebeljoi]

I completely agree with this!

[raae]

100%

[kxxt]

Thanks for supporting ESM, which I have always been waiting for.

I have upgraded my website to gatsby v5 and use the ESM feature. (kxxt/kxxt-website#104)

gatsby develop and gatsby build works fine locally except some warnings that can be ignored:

{ lang: 'dot', meta: null }
⠧ Building development bundle
<w> [webpack.cache.PackFileCacheStrategy] Caching failed for pack: Error: Can't resolve '/mnt/data/Repos/Mine/kxxt-website/gatsby-node.js' in '/mnt/data/Repos/Mine/kxxt-website'
<w> while resolving '/mnt/data/Repos/Mine/kxxt-website/gatsby-node.js' in /mnt/data/Repos/Mine/kxxt-website as file

But it failed to build on vercel because of a GraphQL error. I don't know if it's related to ESM but this is the first time I encounter an error like this.

Logs from vercel

published is a property defined in MDX files' frontmatter for controlling whether the post should be published.

[21:38:33.935] Cloning github.com/kxxt/kxxt-website (Branch: feat/esm, Commit: 49cd833)
[21:38:33.942] Skipping build cache since Vercel CLI used --force option
[21:38:35.733] Cloning completed: 1.798s
[21:38:36.987] Running "vercel build"
[21:38:37.627] Vercel CLI 28.8.0
[21:38:39.202] Injecting Gatsby.js analytics plugin "gatsby-plugin-vercel" to `/vercel/path0/gatsby-config.js`
[21:38:39.204] Adding "gatsby-plugin-vercel" to `package.json` "dependencies"
[21:38:39.214] Running "install" command: `SHARP_IGNORE_GLOBAL_LIBVIPS=true yarn install --network-concurrency 1`...
[21:38:39.665] yarn install v1.22.17
[21:38:39.813] [1/4] Resolving packages...
[21:38:41.121] [2/4] Fetching packages...
[21:40:36.689] info No lockfile found.
[21:40:36.704] [1/5] Validating package.json...
[21:40:36.707] [2/5] Resolving packages...
[21:40:38.435] warning jest > jest-cli > jest-config > jest-environment-jsdom > jsdom > [email protected]: Use your platform's native performance.now() and performance.timeOrigin.
[21:40:39.109] [3/5] Fetching packages...
[21:40:42.055] [4/5] Linking dependencies...
[21:40:45.198] [5/5] Building fresh packages...
[21:40:45.241] success Saved lockfile.
[21:40:45.247] $ npm run build
[21:40:45.666] 
[21:40:45.666] > [email protected] build
[21:40:45.666] > tsc
[21:40:45.667] 
[21:41:10.786] [3/4] Linking dependencies...
[21:41:10.796] warning "gatsby > [email protected]" has incorrect peer dependency "[email protected]".
[21:41:10.810] warning " > [email protected]" has incorrect peer dependency "gatsby@^4.0.0".
[21:41:10.811] warning " > [email protected]" has incorrect peer dependency "react@^16 || ^17".
[21:41:10.811] warning " > [email protected]" has incorrect peer dependency "react-dom@^16 || ^17".
[21:41:10.890] warning Workspaces can only be enabled in private projects.
[21:41:36.723] [4/4] Building fresh packages...
[21:41:53.200] success Saved lockfile.
[21:41:53.208] $ husky install
[21:41:53.309] husky - Git hooks installed
[21:41:53.315] Done in 193.66s.
[21:41:57.142] success compile gatsby files - 1.014s
[21:41:57.143] warning The file 'gatsby-config' has both .js and .mjs variants, please use one or the other. Using .js by default.
[21:41:57.164] success load gatsby config - 0.021s
[21:41:57.443] success load plugins - 0.278s
[21:41:57.444] success onPreInit - 0.001s
[21:41:57.546] success initialize cache - 0.101s
[21:41:57.679] success copy gatsby files - 0.132s
[21:41:57.972] success Compiling Gatsby Functions - 0.281s
[21:41:57.978] success onPreBootstrap - 0.297s
[21:41:58.026] success createSchemaCustomization - 0.006s
[21:41:58.092] success Checking for changed pages - 0.000s
[21:41:58.093] success source and transform nodes - 0.066s
[21:41:58.163] info Writing GraphQL type definitions to /vercel/path0/.cache/schema.gql
[21:41:58.314] success building schema - 0.221s
[21:41:58.395] error There was an error in your GraphQL query:
[21:41:58.396] 
[21:41:58.396] Field "published" is not defined by type "FrontmatterFilterInput".
[21:41:58.427] 
[21:41:58.428] 
[21:41:58.428]   GraphQLError: Field "published" is not defined by type "FrontmatterFilterInput  ".
[21:41:58.428]   
[21:41:58.428]   - graphql-runner.ts:129 GraphQLRunner.validate
[21:41:58.429]     [path0]/[gatsby]/src/query/graphql-runner.ts:129:14
[21:41:58.429]   
[21:41:58.429]   - graphql-runner.ts:218 GraphQLRunner.query
[21:41:58.429]     [path0]/[gatsby]/src/query/graphql-runner.ts:218:49
[21:41:58.429]   
[21:41:58.429]   - create-graphql-runner.ts:57 apply
[21:41:58.430]     [path0]/[gatsby]/src/bootstrap/create-graphql-runner.ts:57:8
[21:41:58.430]   
[21:41:58.430]   - create-pages.ts:36 wrappedGraphQL
[21:41:58.430]     [path0]/[gatsby]/src/services/create-pages.ts:36:47
[21:41:58.430]   
[21:41:58.430]   - gatsby-node.mjs:27 Module.createPages
[21:41:58.431]     file:///vercel/path0/gatsby-node.mjs:27:24
[21:41:58.431]   
[21:41:58.431]   - api-runner-node.js:487 runAPI
[21:41:58.431]     [path0]/[gatsby]/src/utils/api-runner-node.js:487:22
[21:41:58.431]   
[21:41:58.431] 
[21:41:58.433] error There was an error in your GraphQL query:
[21:41:58.434] 
[21:41:58.434] Cannot query field "body" on type "Mdx".
[21:41:58.434] 
[21:41:58.435] If you don't expect "body" to exist on the type "Mdx" it is most likely a typo. However, if you expect "body" to exist there are a couple of solutions to common problems:
[21:41:58.435] 
[21:41:58.436] - If you added a new data source and/or changed something inside gatsby-node/gatsby-config, please try a restart of your development server.
[21:41:58.437] - You want to optionally use your field "body" and right now it is not used anywhere.
[21:41:58.437] 
[21:41:58.437] It is recommended to explicitly type your GraphQL schema if you want to use optional fields.
[21:41:58.438] error There was an error in your GraphQL query:
[21:41:58.438] 
[21:41:58.438] Cannot query field "fields" on type "Mdx".
[21:41:58.439] 
[21:41:58.439] If you don't expect "fields" to exist on the type "Mdx" it is most likely a typo. However, if you expect "fields" to exist there are a couple of solutions to common problems:
[21:41:58.439] 
[21:41:58.440] - If you added a new data source and/or changed something inside gatsby-node/gatsby-config, please try a restart of your development server.
[21:41:58.440] - You want to optionally use your field "fields" and right now it is not used anywhere.
[21:41:58.440] 
[21:41:58.440] It is recommended to explicitly type your GraphQL schema if you want to use optional fields.
[21:41:58.441] error There was an error in your GraphQL query:
[21:41:58.441] 
[21:41:58.441] Cannot query field "published" on type "Frontmatter".
[21:41:58.441] 
[21:41:58.442] If you don't expect "published" to exist on the type "Frontmatter" it is most likely a typo. However, if you expect "published" to exist there are a couple of solutions to common problems:
[21:41:58.443] 
[21:41:58.443] - If you added a new data source and/or changed something inside gatsby-node/gatsby-config, please try a restart of your development server.
[21:41:58.443] - You want to optionally use your field "published" and right now it is not used anywhere.
[21:41:58.443] 
[21:41:58.444] It is recommended to explicitly type your GraphQL schema if you want to use optional fields.
[21:41:58.444] error There was an error in your GraphQL query:
[21:41:58.444] 
[21:41:58.444] Field "tags" is not defined by type "FrontmatterFieldSelector".
[21:41:58.457] 
[21:41:58.458] 
[21:41:58.459]   GraphQLError: Field "tags" is not defined by type "FrontmatterFieldSelector".
[21:41:58.459]   
[21:41:58.460]   - graphql-runner.ts:129 GraphQLRunner.validate
[21:41:58.461]     [path0]/[gatsby]/src/query/graphql-runner.ts:129:14
[21:41:58.461]   
[21:41:58.461]   - graphql-runner.ts:218 GraphQLRunner.query
[21:41:58.461]     [path0]/[gatsby]/src/query/graphql-runner.ts:218:49
[21:41:58.462]   
[21:41:58.462]   - create-graphql-runner.ts:57 apply
[21:41:58.462]     [path0]/[gatsby]/src/bootstrap/create-graphql-runner.ts:57:8
[21:41:58.463]   
[21:41:58.463]   - create-pages.ts:36 wrappedGraphQL
[21:41:58.463]     [path0]/[gatsby]/src/services/create-pages.ts:36:47
[21:41:58.463]   
[21:41:58.464]   - gatsby-node.mjs:27 Module.createPages
[21:41:58.464]     file:///vercel/path0/gatsby-node.mjs:27:24
[21:41:58.464]   
[21:41:58.465]   - api-runner-node.js:487 runAPI
[21:41:58.465]     [path0]/[gatsby]/src/utils/api-runner-node.js:487:22
[21:41:58.465]   
[21:41:58.465] 
[21:41:58.466] not finished createPages - 0.145s
[21:41:59.796] Error: Command "gatsby build" exited with 1

[kxxt]

It's related to createSchemaCustomization. There are some graphql types that I forget to update ever since my website was created. But it doesn't cause any problems on my local machine.

After I removed createSchemaCustomization, vercel still failed to build:

error There was an error in your GraphQL query:
Cannot query field "allMdx" on type "Query".
If you don't expect "allMdx" to exist on the type "Query" it is most likely a typo. However, if you expect "allMdx" to exist there are a couple of solutions to common problems:
- If you added a new data source and/or changed something inside gatsby-node/gatsby-config, please try a restart of your development server.
- You want to optionally use your field "allMdx" and right now it is not used anywhere.
It is recommended to explicitly type your GraphQL schema if you want to use optional fields.
not finished createPages - 0.075s

[kxxt]

Do not spend time looking into this. My site can be successfully deployed to netlify. Maybe there are some weird bugs when building on vercel.

[kxxt]

I just found out why vercel failed to build.

I enabled vercel analytics plugin when exploring the features that vercel provides.

That feature injects gatsby-plugin-vercel into gatsby-config.js before installing dependencies and building the site.

Injecting Gatsby.js analytics plugin "gatsby-plugin-vercel" to `/vercel/path0/gatsby-config.js`
Adding "gatsby-plugin-vercel" to `package.json` "dependencies"
...
warning The file 'gatsby-config' has both .js and .mjs variants, please use one or the other. Using .js by default.

So now there are two config files: gatsby-config.mjs and gatsby-config.js, which causes a build failure because the .js version is used and it only contains the config of gatsby-plugin-vercel.

[LekoArts]

@j0sh77 has reached out to Vercel about this to get it fixed :)

[tyhopp]

The Vercel team fixed this in vercel/vercel#9183. Not sure if it's in production already, there was a stable publish a few commits later in vercel/vercel@721cd3a.

[LekoArts]

If you're using VS Code, the files should have the correct icon when using .mjs once merged :)
vscode-icons/vscode-icons#3204

[Ethan-Arrowood]

Is this being backported to Gatsby v4 or is it only available in v5 on wards?

[LekoArts]

No, it will only be available in Gatsby 5 and later

[tujoworker]

I really hope «local» repo Gatsby plugins are supported as well? 🤞🤞🤞

[LekoArts]

Yeah, local plugins are supported: https://github.com/gatsbyjs/gatsby/tree/master/integration-tests/esm-in-gatsby-files/__tests__/fixtures/plugins/a-local-plugin

[alvinometric]

I can confirm that local, ESM plugins work 🥳

[poudelprakash]

Getting this error after I switched to gatsby-node.mjs from gatsby-node.js, how can I make this warning go?

⠴ Building development bundle
[webpack.cache.PackFileCacheStrategy] Caching failed for pack: Error: Can't resolve '/Users/prakash/workspaces/2021/truemark/truemark_blog/gatsby-node.js' in '/Users/prakash/workspaces/2021/truemark/truemark_blog'
while resolving '/Users/prakash/workspaces/2021/truemark/truemark_blog/gatsby-node.js' in /Users/prakash/workspaces/2021/truemark/truemark_blog as file
⠀

[amcc]

how do i use the above - can you give me a heads up on what to do? I have the same error above after using mjs

I have gatsby 5.7.0

A quick search revealed this line in node_modules/gatsby/dist/utils/webpack.config.js line 719:
return (_plugin$resolvedCompi = plugin.resolvedCompiledGatsbyNode) !== null && _plugin$resolvedCompi !== void 0 ? _plugin$resolvedCompi : path.join(plugin.resolve, `gatsby-node.js`);

[nkuehn]

report back if that is the fix :)

Yes, it is. Thank you.

[LekoArts]

@nkuehn If you want, send in a PR for that? If not I'll probably get to it tomorrow.

[nkuehn]

No hurry. The impact is very low at least in our case and we can live with the warning.

[LekoArts]

A PR is up: #37750

[LekoArts]

I've added a new page to our docs now to explain ESM and Gatsby: https://www.gatsbyjs.com/docs/how-to/custom-configuration/es-modules/

[arsdehnel]

Thanks for those docs, they were helpful to get things started. What wasn't clear to me (and isn't clear from this discussion unless I missed a comment somewhere) is that while you can have .mjs files within Gatsby (v5.3.0+) you cannot run gatsby in a project that has type: module in the package.json.

I have a project that had some pre-existing data fetching/publishing scripts that were done in ESM with the type: module in the package.json. I needed to add a microsite to pull some of that data together into something easily digestible and went for Gatsby. I can get it to work by removing that type: module and changing all files and import references to .mjs but that's pretty annoying and I only have a couple dozen files. Seems like somewhere along the line here Gatsby needs to switch internally to either .cjs file extensions or go through the arduous process of switching to proper ESM itself.

I get that neither of these are easy so really all I think would be good to do now is note that in the docs somewhere. Just so people don't read "you can use ESM with Gatsby" and expect the type:module to work like I did and burn several hours trying to figure out what extension you messed up or issue you missed in seeing how to make this work.

[rogermparent]

I know it's a bit outside of this discussion's scope, but would .mts file handling (or at least planned and/or possible implementations) resolve TypeScript being unsupported in workspaces? While it can be worked around, I do find this issue a significant and hard-to-diagnose hurdle to an otherwise smooth plugin development experience.

[techfg]

Any updates on timing of .mts support?

[joerobot]

Running into this issue a lot in dev mode: #31599 (comment)

Where files in .cache/ aren't transformed to ESM and therefore error. This issue is intermittent, which suggests there should be a way for it to stop happening or there's a bug somewhere else.