Module not found: Error: Cannot resolve module 'fs'...

[samzhao]

Hi,

I was trying to use Yadda with Karma and Jasmine. When I use webpack to build and run the tests, I get these errors:

There's no usage of CasperJS or PhantomJS or Mocha, and the test is exactly the same as the jasmine example found in here.

And here's my karma.conf.js

module.exports = function(config) {
  config.set({
    browsers: ['Chrome'],
    colors: true,
    files: [
      'client/tests/all.js'
    ],
    frameworks: ['jasmine'],
    plugins: [
      'karma-chrome-launcher',
      'karma-script-launcher',
      'karma-junit-reporter',
      'karma-spec-reporter',
      'karma-jasmine',
      'karma-webpack',
    ],

    preprocessors: {
      'client/tests/all.js': ['webpack']
    },
    reporters: ['spec', 'junit'],
    junitReporter: {
      outputDir: 'report'
    },
    // singleRun: true,
    webpack: require('./karma.webpack.config.js'),
    webpackMiddleware: {
      noInfo: true,
      colors: true,
      hash: false,
      timings: true,
      errorDetails: true
    }
  })
};

karma.webpack.config.js

var webpack = require('webpack');
var path = require('path');

module.exports = {
    cache: true,
    debug: false,
    watch: true,
    module: {
        loaders: [{
            test: /\.js$/,
            include: /client\/tests\/unit/,
            loader: 'babel-loader?optional[]=runtime&stage=1'
        }]
    },
    resolve: {
        root: path.resolve('./client'),
        extensions: ['', '.js']
    },
    plugins: [
        new webpack.NoErrorsPlugin()
    ]
}

And because webpack requires the files to have a context, so client/tests/all.js looks like this:

var testsContext = require.context("./unit", true, /\.test\.js$/);
testsContext.keys().forEach(testsContext);
module.exports = testsContext;

It tries to find *.test.js inside of the ./unit directory, which is the entry point for Yadda:

bottles.test.js

var Yadda = require('yadda')
Yadda.plugins.jasmine.StepLevelPlugin.init()

new Yadda.FeatureFileSearch('features').each(function(file) {
    featureFile(file, function(feature) {
        var library = require('./steps/bottles-library')
        var yadda = Yadda.createInstance(library)

        scenarios(feature.scenarios, function(scenario) {
            steps(scenario.steps, function(step, done) {
                yadda.run(step, done)
            })
        })
    })
})

After googling about the fs not found problem, I learned that I can get rid of the error messages by setting

node: {
  fs: "empty"
}

But that causes another problem because I think Yadda's FileSearch.js is trying to use fs.existsSync which will not be found because fs becomes undefined.

Sorry if this is a really special case, but it would be awesome if you guys could help find out the cause of the problem. Thanks very much!

[samzhao]

Upon further investigation, I found that webpack doesn't seem to provide a shim for fs - browserify uses an empty module as the "shim". I think I should be looking for a way to load the features without using Yadda.FeatureFileSearch.

Let me know if I'm on the right track.

[cressie176]

Sounds like you are on the right track - if there's no fs module, difficult to see how file search will work, however it should be possible obtain the feature files in an alternative way. Some options may be...

1. Host them in a web server and obtain them via an ajax call
2. Include them in the karma package - I haven't used karma so not sure if it would be possible to read text files after they've been packaged. I guess it would be possible if you rewrote the features as json (yuck), it's also possible you could convert the features to json as part of a prepack step.

[samzhao]

Thank you so much for the quick response!

I think I can just let karma ignore those .feature files and instead use webpack to require those files. I'll then tell webpack to not parse any of the .feature files. After that I can get the feature files as plain texts. And to make them usable, I might need to write my own FeatureFileParser to not use fs which should be simple enough. Does that sound like a viable plan?

EDIT (just as I'm writing this message I discovered something):

It appears FileSearch which is required by FeatureFileSearch is the only thing I need to "shim" after writing my own FeatureFileParser because it uses fs.existsSync.

This whole custom rewriting thing is bearable if it works, but please please let me know if there's an easier way assuming that I can already acquire the feature files in plain text.

Thanks!

[cressie176]

Assuming you can obtain the files as plain text I would try...

• Create a new object to find, read and iterate over the features in your webpack (maybe call it Yadda.WebPackFeatures). The only function it needs to expose is each, but instead of iterating over feature file paths it should iterate over feature strings, i.e.

Feature: some feature

Scenario: some scenario
Given X
Then Y

• Use WebPackFeatures in your test instead of FeatureFileSeach by changing this line to something like

new Yadda.WebPackFeatures().each(function(file) {

• Specify FeatureParser as the file parser instead of allowing the plugin to default to FeatureFileParser by changing this line to

Yadda.plugins.jasmine.StepLevelPlugin.init({parser: new Yadda.parsers.FeatureParser());

• I would really appreciate a PR with an example project allow me to close Add first class karma support #58. It's worth looking at this issues as there is a brief discussion of alternative approaches.

[samzhao]

Thank you so much for the detailed answer!!

I realized that I misunderstood the webpack context thing in that it doesn't give me the text file, but only gives me a path to the bundled file like this /_karma_webpack_//[hash].feature. Instead of using webpack, I should have used karma to include the feature files. Anyway, I'll create a sample project and submit a PR once I figure it out.

Thanks again :)

[cressie176]

I'll create a sample project and submit a PR once I figure it out.

That would be awesome! Thanks

[samzhao]

@cressie176 hey, I created a sample project, and I managed to start karma without any errors 😃 however, I don't think the tests are run...

When I run karma, it will launch the browser in which I can click "Debug" and check the console for any info or errors, but all I see is "Skipped 0 tests". Could you take a look at it and help figure out if the tests are indeed run?

Here's the repo https://github.com/samzhao/yadda-karma-webpack

Thanks very much!

EDIT:

I just read through @zpratt 's repo https://github.com/zpratt/yadda-karma-example, and it's quite obvious that he knows the library better than I do.

I noticed that he also acquired the feature files via AJAX just like I did in the sample project, but he set the async flag to false. I tried it, and it worked! However, Chrome complains that Synchronous AJAX is deprecated, so I'll look into how to do it asynchronously.

[cressie176]

Good stuff. I'll also take a look (wont be this weekend though)

[samzhao]

Thanks! Please let me know when you do because even though I got it working now (with $.ajax({ async: false })), but I started to get other problems like "next is not a function" inside of the library definition. I would really love to be able to do the ajax stuff async though. Judging from my experiments, fetching feature files asynchronously causes Yadda to not "see" them for the first time until after I save the file again to trigger a Karma auto test run.

[samzhao]

@cressie176 more updates:

I am finally able to run tests now after applying 2 fixes to my code:

1. Setting async to false for $.ajax
2. Removing the done callback in yadda.run(scenario.steps, done) so I no longer get the "next is not a function" error

[madmikeross]

Any update on this? Is there a workaround that doesn't require giving up async?

[cressie176]

Nothing from my side. I haven't used karma and am not finding much time to investigate

[samzhao]

@semblant is there any critical feature you are missing by disabling async for fetching the feature files? In my case, I think I either had to disable async or write some custom karma or yadda plugin, and the former option is the easiest the quickest. Besides, I didn't miss much by doing so.

[zpratt]

@samzhao and @semblant, when I set up my project, I sunk a ton of time trying to figure how to not use sync ajax. I was not able to find a way around that and it's the primary motivation for me creating the project.

[zpratt]

Is there any interest in a PR to merge my project as an example to show how to use this with karma if using sync ajax to load the feature files is acceptable?

[cressie176]

Was thinking about this. My problem is I don't have enough karma knowledge to maintain the example if it goes wrong (e.g. karma gets updated with a different version). The absolute idea would be a spin off project, similar to cucumber-boilerplate that would get people up and running with sensible defaults.

Alternatively (and much less effort), would be to link to your existing example (and others) from README.md

[madmikeross]

@samzhao I am not sure why I took issue with sync ajax. I went ahead and disabled async and got it working, and that's really what matters.

I'm not sure that I am forfeiting any feature by disabling the async. I am using yadda with the mocha plugin, so it feels a bit handicapped in spirit to not have async tests, but honestly, I'm just happy to have it working.

[samzhao]

@semblant I'm pretty sure it won't affect your tests at all given that

1. you keep the ajax config under control, meaning if you did $.ajaxSetup({ async: false }) globally, make sure to set it back if you ever need to use ajax in the code you are testing; better yet you can just set it only for instances that need it like I did here.
2. this is only to synchronously get your features files. Once all the files are done, and Yadda finishes parsing the files, none of your code or tests should be affected if you did point 1 right.

[ghost]

It is possible to get the feature file list from karma, so there is not need to use a REST like interface. I don't remember how to do that, but I remember I did it maybe a year ago. It requires a karma start file or a reporter or something like that. It is different from the karma config file.

There is another way to start karma if you don't want use the karma-cli lib:

var karma = require("karma");
new karma.Server({
    configFile: 'karma.conf.js',
    singleRun: true
}, function (exitCode) {

}).start();

Maybe this could be used instead of a reporter.

update:

I found this mysterious start file. It is used by require.js https://karma-runner.github.io/0.8/plus/RequireJS.html, and it looks something like this:

var tests = [];
for (var file in window.__karma__.files) {
  if (window.__karma__.files.hasOwnProperty(file)) {
    if (/Spec\.js$/.test(file)) {
      tests.push(file);
    }
  }
}

requirejs.config({
    // Karma serves files from '/base'
    baseUrl: '/base/src',
    paths: {
        'jquery': '../lib/jquery',
        'underscore': '../lib/underscore',
    },
    shim: {
        'underscore': {
            exports: '_'
        }
    },
    // ask Require.js to load these files (all our tests)
    deps: tests,
    // start test run, once Require.js is done
    callback: window.__karma__.start
});

By iterating through the files it is possible to create a list of .feature files. I am not sure where this window.__karma__ comes from. I'll ask karma developers about this, maybe they have a recommended solution.

update:
I am pretty sure that ajax and jquery is not the proper way to solve this. I asked a question on the karma mailing list, maybe they can recommend something better.

Another way could be to optimize this is parsing gherkin with nodejs, serialize the results and use it in the browser to run the tests. So we would have only a single .json file instead of a bunch of .feature files. I am not sure whether this is possible with yadda.

If we stick with multiple gets, maybe it would be better to use https://github.com/mafintosh/level-filesystem instead of jquery, which is the browserified implementation of fs. I am not sure how exactly it works.

[samzhao]

@Inf3rno hey, thanks for the information!

I agree with you that getting the feature files using AJAX probably isn't the proper way, and it isn't elegant.

I think the problem was we couldn't directly get the step files because they were being loaded by Webpack (honestly I don't remember why I needed to do that...). That was why we needed the AJAX requests in the first place — to work around the cannot resolve fs issue with Webpack; otherwise we could just get the feature files directly using fs, and completely decouple our app files and webpack.config from the tests.

So if you don't need to share any of your app code with your tests, and you don't need to use Webpack loaders on your test files, then you don't need to go through any of these hoops.

I think the most common reason to use Webpack is to use ES6 in step files, in which case this example code can help.

[ghost]

@samzhao I am using browserify. I think I experience the same cannot resolve fs problem, at least FeatureFileSearch does not appear to work properly.

[ghost]

I checked yadda. By browserify the problem that we don't have shim for browser fs https://github.com/acuminous/yadda/blob/master/lib/shims/index.js#L46 and fs from browserify does not support sync methods: https://github.com/mafintosh/level-filesystem In theory it would work by an async file search, so we would not need to use jquery. I'll try to implement something like this. I don't know if this helps by webpack users.

[ghost]

@samzhao I modified the lib this way: #58 (comment) Try it out, I think it works with webpack as well. It uses sync XHR and karma file list to implement a browser fs shim yadda can use. I tried out with browserify, it worked. Maybe others develop it further and integrate it into the lib.

[samzhao]

wow, awesome job! It looks like it's gonna work.
Thanks, @Inf3rno! I'll try it out with webpack when I find time :)

[ghost]

@samzhao I don't fully understand yet how to integrate properly with yadda and the browserified version of the node path module. I'll might rewrite it later, so it is experimental now. If webpack does not emulate this path module somehow, it won't work, since I use require("path") in the fs shim and yadda depends on it as well.

[ghost]

@samzhao I created a fork with some enhancements on the code. You can try it out by putting this in the package.json: "yadda": "git://github.com/inf3rno/yadda.git",. Keep me noted whether it works with webpack. Have a nice day!

[cressie176]

Think this should be OK to close thanks to @Inf3rno's efforts with karma shims. Agree?

[ghost]

@cressie176 I don't think that webpack is supported yet, but this is a duplicate of #58 , so it might be closed in favor of that issue.