Parallel Sourcing

[KyleAMathews]

Original PR

Gatsby runs APIs sequentially. That's probably what we want as running many APIs is CPU bound. But sourcing is almost entirely bounded by the speed of the external APIs so we save quite a bit of time for sites with multiple source plugins by sourcing in parallel.

This change speeds up cold sourcing for gatsbyjs.com (which has 4 external sources) from 78s -> 45s or a 42% improvement.

One worry with changing this is it could increase memory pressure from more nodes being created and transformed at the same time. This is a separate issue that needs addressed but until is, sites might see increased memory usage from adding this flag.

There's a chance someone is depending on the serial behavior of API running so we'd need to wait for Gatsby v3 to enable it for everyone.

To enable this in your site, modify your gatsby-config.js to add the following:

module.exports = {
  flags: {
    PARALLEL_SOURCING: true
  },
  plugins: [], // Your plugins go here
}

[jerelmiller]

Hey there 👋

I (and my team at New Relic) are super interested in this feature, but had a couple questions before we dig in too deep. We are in the process of migrating our docs site over to open source and using Gatsby in the process!

Due to the size of the new site (thousands of markdown and MDX files + many other auto generated pages), we find that our boot-up time for development is a bit slow. We've developed quite a few custom plugins that handle various requirements of the site. While we haven't taken too much time (yet) to diagnose why our boot-up time is so long, I suspect its likely due to one or more of our custom plugins.

The new parallel sourcing feature sounds super intriguing. We haven't tried it out yet, but I'd be curious what kind of limitations there are (if any) for these. Does this feature require that all of the source plugins operate independently from data created from other plugins? And does this feature mainly benefit async plugins? Or is this feature designed to help across all plugins?

For reference, here are 2 examples from our site that rely on data created from other plugins:

• We auto generate pages from other pages. One of the requirements of our site is that we have a page available at any URL path segment (for example, if we have pages at /docs/a/b and /docs/c/d, we need to auto create pages for /docs/a, and /docs/b. This requires that the data that lists all pages in the site is available in order for that plugin to do its work. (If you're curious at the full details, you can check out this comment)
• We source our nav for each page in its own plugin. This plugin makes heavy use of context.nodeModelto pull in YAML data source from gatsby-transformer-yaml.

I would love to hear more about this! We will try this feature out soon. Patrick had asked me to get the discussion going!

[KyleAMathews]

👋

Looking at your links above, it doesn't seem like this flag would affect you. What it changes is pretty small. If you have two plugins which both implement the sourceNodes API, with the flag enabled, it'll call both of them at the same time vs. serially (which is the current method).

Why this can speed up your site is sometimes source plugins can block Gatsby as they're making slow network requests. Running source plugins in parallel then means that all the slow network requests can be happening at the same time.

But in sites like yours where all the data seems local, the problem is more just the pure computational expense of sorting through MDX/md files. MDX in particular is quite expensive. We do hope to speed up these sorts of operations in the future by spreading CPU intensive work across multiple node.js processes but that hasn't happened yet.

Are you on the latest Gatsby? You might have seen Query on Demand get released — this helps speed up dev server startup time as it delays most of the markdown work until you actually visit the page. The MDX plugin still does a lot of work upfront so QoD is less effective but you should notice some effect https://twitter.com/GatsbyJS/status/1338929948975898624

[jerelmiller]

Thanks for getting back!

That is good to know and makes total sense. We definitely do a lot of computationally expensive work.

We do have the new features enabled! Query on demand and lazy images helped a TON with our boot-up time, so thanks so much for those features! We are still sitting about ~1min+ for boot-up from a cold start. I'd love to get that down if we can, but it may just be that we have too much going on at the moment to really help that.

I'm hoping with the release of MDX v2, there is some help in terms performance. I know the underlying remark core just got a HUGE update, something we are hoping to try out soon (we are on an alpha version of v2 right before the big update).

We'll keep watch for new features. Thanks for all the great work thus far!

[cedricdelpoux]

I have a lot of source plugin and when I use PARALLEL_SOURCING: true flag I always get a Request timed out error

[herecydev]

This has been working absolutely fine for me for several weeks now 🚀

[krismorf]

This was working fine when I was only using:

gatsby-source-filesystem
gatsby-source-google-spreadsheets
gatsby-source-wordpress

but started getting errors when I've added gatsby-source-youtube-v3

Sometimes I get error Client network socket disconnected before secure TLS connection was established, other times I get Socket hang up or Request timed out

Disabling PARALLEL_SOURCING fixes this.

Gatsby Info

  System:
    OS: Linux 5.10 Ubuntu 20.04.2 LTS (Focal Fossa)
    CPU: (8) x64 Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
    Shell: 5.8 - /usr/bin/zsh
  Binaries:
    Node: 14.17.3 - ~/.nvm/versions/node/v14.17.3/bin/node
    Yarn: 1.22.10 - ~/.nvm/versions/node/v14.17.3/bin/yarn
    npm: 6.14.13 - ~/.nvm/versions/node/v14.17.3/bin/npm
  npmPackages:
    gatsby: ^3.9.1 => 3.9.1 
    gatsby-plugin-eslint: ^3.0.0 => 3.0.0 
    gatsby-plugin-feed: ^3.9.0 => 3.9.0 
    gatsby-plugin-gatsby-cloud: ^2.9.1 => 2.9.1 
    gatsby-plugin-google-tagmanager: ^3.9.0 => 3.9.0 
    gatsby-plugin-image: ^1.9.0 => 1.9.0 
    gatsby-plugin-manifest: ^3.9.0 => 3.9.0 
    gatsby-plugin-netlify: ^3.9.0 => 3.9.0 
    gatsby-plugin-react-helmet: ^4.9.0 => 4.9.0 
    gatsby-plugin-react-helmet-canonical-urls: ^1.4.0 => 1.4.0 
    gatsby-plugin-react-i18next: ^1.1.1 => 1.1.1 
    gatsby-plugin-sharp: ^3.9.0 => 3.9.0 
    gatsby-plugin-sitemap: ^4.5.0 => 4.5.0 
    gatsby-plugin-theme-ui: ^0.10.1 => 0.10.1 
    gatsby-source-filesystem: ^3.9.0 => 3.9.0 
    gatsby-source-google-spreadsheets: 3.0.8 => 3.0.8 
    gatsby-source-wordpress: ^5.9.2 => 5.9.2 
    gatsby-source-youtube-v3: ^3.0.1 => 3.0.1 
    gatsby-transformer-sharp: ^3.9.0 => 3.9.0 

(Side note: I'm on Windows 11 Insider with WSL2, sometimes when changing the port gatsby develop -p 3000 fixes the problem but inconsistenly. Also increased gatsby-source-wordpress => options.schema.timeout)

[vai0]

we experienced the same, but with google sheets api. Google doesn't like when you're hitting them too frequently. the fix for us was to add an artificial sleep between each google request. 1 second for us was sufficient in getting past that timeout