Cannot read property 'charAt' of undefined

[gordonwoodhull]

Hi @75lb!

A user reported that our dc.js docs build broke: dc-js/dc.js#1142

The docs were building fine for months, but as often happens, if you a fresh install, then some npm dependency somewhere updates and whamo.

I know the problem is deeper than grunt-jsdoc-to-markdown, but I'm not sure where to file it. It appears to be deeper than jsdoc-to-markdown, and I tried to diagnose it further using npm-shrinkwrap.json, but backing out jsdoc-to-markdown and jsdoc-parse didn't help, so I'm asking for help in case you might have seen this.

It's reproduceable by checking out dc.js and doing npm install && grunt docs. I'll gladly close this if you can point me to a better way to diagnose this!

It is very likely that we have invalid jsdoc somewhere in our extensive corpus.

[75lb]

well, the first thing to do is tell me exactly where the Fatal error: Cannot read property 'charAt' of undefined is coming from!

Post a stack trace please.

[gordonwoodhull]

Yes, sorry for neglecting that, here it is.

Running "jsdoc2md:dist" (jsdoc2md) task
>> writing web/docs/api-latest.md
Fatal error: Cannot read property 'charAt' of undefined
TypeError: Cannot read property 'charAt' of undefined
    at instantiate (/Users/gordonwoodhull/src/dc.js/node_modules/grunt-jsdoc-to-markdown/node_modules/jsdoc-to-markdown/node_modules/dmd/node_modules/ddata/lib/ddata.js:830:19)
    at Object.parentName (/Users/gordonwoodhull/src/dc.js/node_modules/grunt-jsdoc-to-markdown/node_modules/jsdoc-to-markdown/node_modules/dmd/node_modules/ddata/lib/ddata.js:842:11)
    at Object.sig (/Users/gordonwoodhull/src/dc.js/node_modules/grunt-jsdoc-to-markdown/node_modules/jsdoc-to-markdown/node_modules/dmd/node_modules/ddata/lib/ddata.js:479:30)
    at Object.eval (eval at createFunctionContext (/Users/gordonwoodhull/src/dc.js/node_modules/grunt-jsdoc-to-markdown/node_modules/jsdoc-to-markdown/node_modules/dmd/node_modules/stream-handlebars/node_modules/handlebars/dist/cjs/handlebars/compiler/javascript-compiler.js:221:23), <anonymous>:5:266)
    at Object.prog [as fn] (/Users/gordonwoodhull/src/dc.js/node_modules/grunt-jsdoc-to-markdown/node_modules/jsdoc-to-markdown/node_modules/dmd/node_modules/stream-handlebars/node_modules/handlebars/dist/cjs/handlebars/runtime.js:193:15)
    at Object.<anonymous> (/Users/gordonwoodhull/src/dc.js/node_modules/grunt-jsdoc-to-markdown/node_modules/jsdoc-to-markdown/node_modules/dmd/node_modules/stream-handlebars/node_modules/handlebars/dist/cjs/handlebars/base.js:205:22)
    at Object.eval (eval at createFunctionContext (/Users/gordonwoodhull/src/dc.js/node_modules/grunt-jsdoc-to-markdown/node_modules/jsdoc-to-markdown/node_modules/dmd/node_modules/stream-handlebars/node_modules/handlebars/dist/cjs/handlebars/compiler/javascript-compiler.js:221:23), <anonymous>:7:32)
    at ret (/Users/gordonwoodhull/src/dc.js/node_modules/grunt-jsdoc-to-markdown/node_modules/jsdoc-to-markdown/node_modules/dmd/node_modules/stream-handlebars/node_modules/handlebars/dist/cjs/handlebars/runtime.js:159:30)
    at ret (/Users/gordonwoodhull/src/dc.js/node_modules/grunt-jsdoc-to-markdown/node_modules/jsdoc-to-markdown/node_modules/dmd/node_modules/stream-handlebars/node_modules/handlebars/dist/cjs/handlebars/compiler/compiler.js:488:21)
    at Object.invokePartial (/Users/gordonwoodhull/src/dc.js/node_modules/grunt-jsdoc-to-markdown/node_modules/jsdoc-to-markdown/node_modules/dmd/node_modules/stream-handlebars/node_modules/handlebars/dist/cjs/handlebars/runtime.js:218:12)

[gordonwoodhull]

(And if I'd figured out how to get the stack trace earlier, I could probably have narrowed in on the right package to back out. Live and learn.)

[75lb]

i have reproduced it but it's a strange one as the dc.js docs render fine if processed directly with jsdoc2md dc.js.. it's only via grunt it fails.. i noticed your grunt plugin version is tied to v1.1.1, but it still fails when updated to the latest v1.2.1..

i'm pretty sure this is an npm/dependency-resolution issue, rather than dc.js docs or jsdoc2md code.. will see.

[75lb]

it's something to do with npm..

if i reduce your dev-deps to this, so only these two deps are installed:

  "devDependencies": {
    "grunt": "~0.4.5",
    "grunt-jsdoc-to-markdown": "^1.2.1"
  },

and your gruntfile to this

module.exports = function (grunt) {
    'use strict';

    grunt.loadNpmTasks('grunt-jsdoc-to-markdown')

    grunt.initConfig({
      jsdoc2md: {
          dist: {
              src: 'dc.js',
              dest: 'web/docs/api-latest.md'
          }
      }
    });
};

and run grunt jsdoc2md then it works.. in other words, the jsdoc2md plugin works fine in isolation.. i suspect the issue is caused by an npm bug.. i'll leave you guys to have a closer look..

[gordonwoodhull]

Ugh, that's horrible. Thanks for doing the detective work!

I'm not sure if this is the kind of thing I can troubleshoot. I'm used to weird side-effects in C++ but can't see why that would happen in JavaScript.

[75lb]

yeah, it's not suppose to happen - it's an npm bug.. i'll probably have a deeper look tonight..

[75lb]

i've traced the problem to this issue, in a different module (not one that my tools use).. the problem is the offending module overwrites the global Array.prototype.find method, breaking it for everyone else..

montagejs/collections#139

i'll try to find out which other module is using this broken one..

[75lb]

i found the offending plugin which pulls in the broken "collections" module - grunt-gh-pages.

remove it (npm rm grunt-gh-pages) and grunt jsdoc2md will work. Or try upgrading it to the latest version (yours is old).

[75lb]

your grunt stack is getting old, you need to keep your dependencies up to date..

[gordonwoodhull]

Ugh. Yes, clearly we need to. Or just shrink-wrap the whole thing so that our build doesn't break without us touching it.

Thank you so much for tracking this down. I would not have thought that it was someone screwing with Array.prototype. I also thought there was some isolation between grunt tasks. I guess I am naive.

[75lb]

you don't need to shrinkwrap, you are already (unwittingly) doing it - most of your dep versions are fixed.. e.g. the semver expression "~0.10.1" is fixed at version 0.10.1, no room for movement.. run npm help 7 semver and read about semver expressions.

besides, shrinkwrapping is not the answer - all you do is lock the bugs in.. we have found a bug in your version of grunt-gh-pages, which according to your package.json is fixed at a certain version (over a year old).. you need to upgrade that package, or remove it - not shrinkwrap it..

[gordonwoodhull]

I am aware of that. I was trying to say that there are two choices - either roll with all the changes in the npm ecosystem, or nail everything to the floor (including transitive dependencies).

We are doing something in between, and apparently in this case a loose constraint in jsdoc-to-markdown allowed dmd to upgrade and that exposed a bug in a package it doesn't even know about.

I agree that the more responsible thing to do is to loosen our dependencies and allow other packages to upgrade. But this would mean we would break even more often, and honestly I don't have time for that, so I was just whining that I should nail everything to the floor.

But I'm just whining and of course I'll just get some sleep and upgrade the dependency instead. 😊

A million thanks!

[75lb]

apologies, on closer inspection there is room for movement in most of your devDependency semver expressions, so shrinkwrapping would make a difference..

yes, you could nail modules down until you get round to reviewing your build stack..

you're welcome, let me know if there are any more issues.