Bug: Undefined errors while attempting to create new index

[ashutoshsaboo]

Hi Reese @reese ,

Thank you so much for creating this plugin! 👏 I was trying to use this in my Gatsby site, and was unfortunately running into few internal errors from the plugin.

This is my configuration for the gatsby plugin. I have ensured and validated from Gatsby's GraphQL explorer that the below fields exist correctly for nodes of type annotations:

    {
      resolve: "gatsby-plugin-stork",
      options: {
        indexes: [
          {
            resolvers: {
              annotations: {
                url: node => node.fields.book_slug_path,
                title: node => node.extracted_google_books_metadata.book_title,
                contents: node => node.extracted_google_books_metadata.book_description,
              }
            },
            filename: 'annotations.st'
          }
        ],
        theme: "dark",
      }
    }

And for some reason, i keep on running into this exception below - which seems related to this line:

 ERROR #11321  PLUGIN

"gatsby-plugin-stork" threw an error while running the onPostBootstrap lifecycle:

Cannot set property 'title' of undefined

  60 |                             var key = _ref6[0],
  61 |                                 resolveFunc = _ref6[1];
> 62 |                             acc[key] = resolveFunc(node);
     |                                      ^
  63 |                           }, {});
  64 |                           files.push(resolvedValues);
  65 |                         }); // Serialize and write to TOML

File: node_modules/gatsby-plugin-stork/gatsby-node.js:62:38



  TypeError: Cannot set property 'title' of undefined

I tried to oversimplify the config above to this subsequently, i.e, just hardcoded strings - and even then I'm running into the same exception as above. Surely if there was some issue with my nodes configuration above and if that might have been resulting in the exception above, then it should have succeeded here:

    {
      resolve: "gatsby-plugin-stork",
      options: {
        indexes: [
          {
            resolvers: {
              annotations: {
                url: node => "test1",
                title: node => "test2",
                contents: node => "test3",
              }
            },
            filename: 'annotations.st'
          }
        ],
        theme: "dark",
      }
    },

Would really appreciate if you could help with this? If you need any other blob of local nodes or other related configs from my end for looking into this, please let me know.

Thanks! :)

[reese]

Thanks for the report! Yep, this was an issue from my last refactor. I put up a bug fix that I think should resolve this in a patch version, so if you try using version 0.3.1, that should fix this issue. Let me know if you run into any issues!

[ashutoshsaboo]

Hmm thanks Reese, that seems to have solved the first part of the issue. Now I seem to be getting:

Writing temporary TOML to /var/folders/d_/z5zsztwn7wscks__9klng7msnbwqz7/T/tmp-52018-Ny8Rtg0THxEH-.toml
⠙ onPostBootstrap

 ERROR

Could not generate index for generated TOML file:


 ERROR

[input]
base_directory = "/Users/ashsaboo/repos/highlights/node_modules/gatsby-plugin-stork"

  [[input.files]]
  url = "/book/atomic-habits"
  title = "Atomic Habits"
  contents = "James Clear presents strategies to form good habits, break bad ones, and master the tiny behaviors that help lead to an improved life."

  [[input.files]]
  url = "/book/reinforcement-learning"
  title = "Reinforcement Learning"
  contents = "The significantly expanded and updated new edition of a widely used text on reinforcement learning, one of the most active research areas in artificial intelligence. Reinforcement learning, one of
 the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives while interacting with a complex,
uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the field's key ideas and algorithms. This second edition has been significantly expanded
and updated, presenting new topics and updating coverage of other topics. Like the first edition, this second edition focuses on core online learning algorithms, with the more mathematical material set off in
shaded boxes. Part I covers as much of reinforcement learning as possible without going beyond the tabular case for which exact solutions can be found. Many algorithms presented in this part are new to the
second edition, including UCB, Expected Sarsa, and Double Learning. Part II extends these ideas to function approximation, with new sections on such topics as artificial neural networks and the Fourier basis,
and offers expanded treatment of off-policy learning and policy-gradient methods. Part III has new chapters on reinforcement learning's relationships to psychology and neuroscience, as well as an updated
case-studies chapter including AlphaGo and AlphaGo Zero, Atari game playing, and IBM Watson's wagering strategy. The final chapter discusses the future societal impacts of reinforcement learning."

[output]
filename = "public/annotations.st"



 ERROR #11321  PLUGIN

"gatsby-plugin-stork" threw an error while running the onPostBootstrap lifecycle:

Command failed: stork --build /var/folders/d_/z5zsztwn7wscks__9klng7msnbwqz7/T/tmp-52018-Ny8Rtg0THxEH-.toml
Could not generate index: No files could be indexed


  158 | function buildStorkIndex(tempFileName, tomlString) {
  159 |   try {
> 160 |     execSync("stork --build " + tempFileName);
      |     ^
  161 |   } catch (e) {
  162 |     console.error("Could not generate index for generated TOML file:");
  163 |     console.error(tomlString);

File: node_modules/gatsby-plugin-stork/gatsby-node.js:160:5



  Error: Command failed: stork --build /var/folders/d_/z5zsztwn7wscks__9klng7msnbwqz7/T/tmp-52018-Ny8Rtg0THxEH-.toml
  Could not generate index: No files could be indexed

  - gatsby-node.js:160 buildStorkIndex
    [highlights]/[gatsby-plugin-stork]/gatsby-node.js:160:5

  - gatsby-node.js:108 _callee$
    [highlights]/[gatsby-plugin-stork]/gatsby-node.js:108:25

The intermediate toml file seems to be correct, not sure then why this seems to be happening with the plugin? Would appreciate any help out here!

Also, Is there any project of yours/any other public project that currently uses this plugin - could you link to it if so, for reference configs?

[reese]

That's an error actually coming from stork itself, although I'm not exactly sure what "valid files" is referring to. My only guess here is that since you're passing contents instead of the actual file, stork might not know the file type (since there's no extension). You might want to try having a filetype resolver (see the config docs for more info on that) and pass it whatever file type you're using (maybe PlainText from the looks of it).

(And as a side note for myself, I should add something about filetype to the README and probably require it if you only pass contents.)

Unfortunately the project I'm using this for still isn't public right now, and I'm not totally sure who else is using this extension since I mostly just made it for myself 😅 Of course my project also uses a pretty standard Gatsby setup, so it just uses the default configuration for this project.

[ashutoshsaboo]

These are the configs i'm using currently, very similar to the default one's that you linked to (only that i added filetype just now):

    {
      resolve: "gatsby-plugin-stork",
      options: {
        indexes: [
          {
            resolvers: {
              annotations: {
                url: node => node.fields.book_slug_path,
                title: node => node.extracted_google_books_metadata.book_title,
                contents: node => node.extracted_google_books_metadata.book_description,
                filetype: 'PlainText'
              }
            },
            filename: 'annotations.st'
          }
        ],
        theme: "dark",
      }
    },

With filetype, i get this:

warn Warning: there are unknown plugin options for "gatsby-plugin-stork": indexes.0.resolvers.annotations.filetype
Please open an issue at ghub.io/gatsby-plugin-stork if you believe this option is valid.
 ERROR #11321  PLUGIN

"gatsby-plugin-stork" threw an error while running the onPostBootstrap lifecycle:

resolveFunc is not a function

  62 |                             var key = _ref6[0],
  63 |                                 resolveFunc = _ref6[1];
> 64 |                             resolvedValues[key] = resolveFunc(node);
     |                                                   ^
  65 |                           });
  66 |                           files.push(resolvedValues);
  67 |                         }); // Serialize and write to TOML

File: node_modules/gatsby-plugin-stork/gatsby-node.js:64:51



  TypeError: resolveFunc is not a function

  - gatsby-node.js:64
    [highlights]/[gatsby-plugin-stork]/gatsby-node.js:64:51

  - Array.forEach

  - gatsby-node.js:61
    [highlights]/[gatsby-plugin-stork]/gatsby-node.js:61:52

  - Array.forEach

  - gatsby-node.js:57 _callee$
    [highlights]/[gatsby-plugin-stork]/gatsby-node.js:57:31

  - new Promise

  - gatsby-node.js:121
    [highlights]/[gatsby-plugin-stork]/gatsby-node.js:121:30

  - Array.map

  - gatsby-node.js:47 _callee2$
    [highlights]/[gatsby-plugin-stork]/gatsby-node.js:47:40

Is this plugin passing all props that stork accepts - https://stork-search.net/docs/config-ref - especially filetype for now for specificity? If it's not too much, can you add it?

I'm actually also wondering why should stork need filetype in the first place when we are passing contents altogether (jameslittle230/stork#163), since with path it makes sense to require filetype for it to determine parsing strategy etc for relevant file extensions. But with passing contents (Btw I am passing contents because for my case all these text contents are present in nodes rather than Mdx blog files etc), that first step is already done for it by me as the client, and all it should do is tokenize it and index it ideally.

[reese]

For the error from the plugin, that's just because your resolver needs to be a function: filetype: () => "PlainText". I think it does make sense that you would need to pass a filetype, because otherwise Stork doesn't know what parser is required. contents isn't a parsed format -- it could be a raw HTML string, or plain markdown, so Stork needs to know what type of contents they are. It can't tokenize something that it can't parse tokens for.

As for config, this plugin doesn't currently support them all yet. I have a ticket (#2) to support them eventually, but if you have any in particular that you need, you can comment on that issue and I can try to prioritize them. PRs are also welcome if I don't get to it soon enough.

[reese]

I'm happy to keep helping out in this thread, but I'm going to close out the issue here since the original issue is now resolved.

[ashutoshsaboo]

Bravo @reese , that seemed to work -- thank you so much! :) Do you think this little detail of all params that are currently supported by the plugin, and what type/schema their values should be of (possibly with an example) should be there in the README? That might help for future users of the plugin!

Other than that, I understand this might be a bit orthogonal from what the issue was opened for. I don't want to re-open this, but would appreciate if you could help with this! @reese
Is there any way currently in the plugin to pass multiple resolvers (for each different node types), and let Stork have their url/title/contents pairs tokenized and concatenated into an eventual index? I used this below hinting from the docs, but it didn't seem to work and it only picks up the first resolver for creating the index, thus ignoring all others below it. Can this config be tweaked in any way to make this intended functionality^ work? :

    {
      resolve: "gatsby-plugin-stork",
      options: {
        indexes: [
          {
            resolvers: {
              books: {
                url: node => node.fields.book_slug_path,
                title: node => node.extracted_google_books_metadata.book_title,
                contents: node => Object.values(JSON.stringify(node.extracted_google_books_metadata)).join('\n') + '\n' + node.book_original_metadata.join('\n'),
                filetype: () => "PlainText"
              },
              Annotation: {
                url: node => node.annotation_slug_path,
                title: node => `${node.extracted_google_books_metadata.book_title}-${node.annotation_highlight}`.replace(/^(.{20}[^\\s]*).*/, "$1"),
                contents: node => (node.annotation_note !== undefined ? node.annotation_note + '\n' : '') + node.annotation_highlight + '\n' + node.section_name,
                filetype: () => "PlainText"
              }
            },
          }
        ],
        theme: "basic",
      }
    },

[reese]

That functionality you're describing should be how the plugin already works. It just looks up the resolvers by the node.internal.type, so it should be able to handle both resolvers for multiple types. In previous comments you had the type as annotations instead of Annotation, is that a possible reason for it not picking up your second set of resolvers? Note that the type name there is case-sensitive.

[ashutoshsaboo]

The node names are correct I validated them again. But for some weird reason the plugin was continuing to build only on a single node type even when i ran gatsby develop/gatsby build again (even though other plugins were updating just fine). I just ran gatsby clean and re-built using gatsby develop - it then seemed to work fine with the correct behaviour - taking both node types into account.

Thank you for the help, this seems to work fine now! :)

One small thing though that I noticed @reese : when using pathPrefix the slug page links usually are auto-appended by Gatsby. But for building the index, only way is that you need to provide the prefix manually. The only caveat with this is that for develop path-prefixes are not supported - so on either one of develop/build search results would point to the link either just with or without prefix. Maybe the plugin, can use gatsby's withPrefix as pre-processing (would be a generic solution) before passing it to stork to build the index? Or does the plugin already supports for this via some configurable param?

[reese]

There isn't a config for using withPrefix at the moment, although I think that might be a nice config addition. If you want to make a separate Feature Request issue to track that, I can look into adding it at some point.

[ashutoshsaboo]

There you go opened a new feature request - #13 ! Thanks!

[ashutoshsaboo]

Btw just checking in on have you tried running this plugin with stork in a CI build? I'm trying to use github actions, see workflow below, and i've tried add installed stork to path as well, still which -s stork seems to be failing for the plugin to run. :

# This workflow will do a clean install of node dependencies, build the source code and run tests across different versions of node
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-nodejs-with-github-actions

name: Build and Deploy

on:
  push:
    branches: [ master ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v1

      - name: Setup python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8

      - name: Execute pre-processing script
        run: |
          export PATH=./stork:$PATH
          pip install -r requirements.txt

      - name: Publish to Github pages
        uses: enriikke/gatsby-gh-pages-action@v2
        with:
          deploy-branch: gh-pages
          gatsby-args: --prefix-paths

This is the error I get:

error It looks like the Stork executable is not installed. For some instructions on how to install Stork, see the documentation: https://stork-search.net/docs/install
error "gatsby-plugin-stork" threw an error while running the onPostBootstrap lifecycle:

Command failed: which -s stork

  144 |   // Check if Stork is present
  145 |   try {
> 146 |     execSync("which -s stork"); // `-s` omits output and just returns a 0 or 1 exit code
      |     ^
  147 |   } catch (e) {
  148 |     console.error("It looks like the Stork executable is not installed. For some instructions on how to install Stork, see the documentation: https://stork-search.net/docs/install");
  149 |     throw e;


  Error:Command failed: which -s stork

Any ideas on how can the plugin start detecting stork from a specific path in the CI build? @reese
In terms of github actions: stork binary on ubuntu linux is installed on PWD -> ./stork => so can this be configured somehow for the plugin to use stork from this specific path?

[reese]

This plugin currently doesn't have CI support -- see #9 for the work needed for that. Feel free to comment there if you have a specific features for that if you'd like.

[ashutoshsaboo]

Ahhh! Would really hope if this can be supported sooner than other features lined up from gh issues @reese ! CI builds would be kind of a huge need for almost about everyone who uses the plugin, as you also mention in the linked issue.

[reese]

That is currently the highest-priority issue, although for full transparency I probably won't have time to do the work for it until at least next week (although I'm happy to review PRs 🙂). In the short term, given that it it looks like you already have stork somewhere in the tree, you could always do something like alias stork=./stork or something like that until there's a more robust solution supported by the plugin.

[ashutoshsaboo]

aliases don't seem to work with github actions for some weird reason :/ apparently this is the only closest thing i found - https://github.community/t/create-and-use-an-alias-inside-github-action/147616 @reese