Webpack4

[staeke]

Hello @jedwards1211 and @luisherranz . I thought I'd make a pull request now. Have been working on a couple of things for a while and wanted a little feedback. Included:

• webpack 4 support
• using autoupdate and instructions for production builds
• more advanced project exclude
• new take on parsing and including of modules to be more compatible with Meteor builds

It's quite meaty...happy to discuss further

[jedwards1211]

@staeke I see you've made a bunch of commits since opening this PR, is it ready now?
I haven't had much time to devote to this package lately because I haven't been working much on our old project that uses it, but if you still need to do some work on this, ping me when it's ready.

[staeke]

Thanks for getting back @jedwards1211 . Yes, I've tested it now quite thouroughly. But...I haven't tested css packages (like bootstrap). I want to do that before completion.

[staeke]

Hello again @jedwards1211

I've now tested with bootstrap and made a demo application, and updated the README files again. I think this is ready

[CaptainN]

I have been trying to get this to work in a NativeScript app (through Svelte Native). I keep hitting various parsing or config errors. I wish I could point to a specific error - but I'm not sure if it's something in the package, or if it's my configuration. Here's what I'm getting:

ERROR in ../node_modules/meteor-imports-webpack-plugin/meteor-imports.js 43:104
Module parse failed: Bad character escape sequence (43:104)
You may need an appropriate loader to handle this file type.
| require("C:\Users\Kevin\repos\test-svelte-mobile\web\.meteor\local\build\programs\web.browser\packages\sha.js");
| require("C:\Users\Kevin\repos\test-svelte-mobile\web\.meteor\local\build\programs\web.browser\packages\srp.js");
> require("C:\Users\Kevin\repos\test-svelte-mobile\web\.meteor\local\build\programs\web.browser\packages\underscore.js");
| require("C:\Users\Kevin\repos\test-svelte-mobile\web\.meteor\local\build\programs\web.browser\packages\accounts-password.js");
| require("C:\Users\Kevin\repos\test-svelte-mobile\web\.meteor\local\build\programs\web.browser\packages\audit-argument-checks.js");
 @ ./app.ts 51:0-24

My configuration:

const { join, relative, resolve, sep } = require('path')

const webpack = require('webpack')
const nsWebpack = require('nativescript-dev-webpack')
const nativescriptTarget = require('nativescript-dev-webpack/nativescript-target')
const {
  getNoEmitOnErrorFromTSConfig
} = require('nativescript-dev-webpack/utils/tsconfig-utils')
const CleanWebpackPlugin = require('clean-webpack-plugin')
const CopyWebpackPlugin = require('copy-webpack-plugin')
const ForkTsCheckerWebpackPlugin = require('fork-ts-checker-webpack-plugin')
const { BundleAnalyzerPlugin } = require('webpack-bundle-analyzer')
const {
  NativeScriptWorkerPlugin
} = require('nativescript-worker-loader/NativeScriptWorkerPlugin')
const TerserPlugin = require('terser-webpack-plugin')
const hashSalt = Date.now().toString()
const svelteNativePreprocessor = require('svelte-native-preprocessor')
const MeteorImportsPlugin = require('meteor-imports-webpack-plugin')

module.exports = env => {
  // Add your custom Activities, Services and other Android app components here.
  const appComponents = [
    'tns-core-modules/ui/frame',
    'tns-core-modules/ui/frame/activity'
  ]

  const platform = env && ((env.android && 'android') || (env.ios && 'ios'))
  if (!platform) {
    throw new Error('You need to provide a target platform!')
  }

  const platforms = ['ios', 'android']
  const projectRoot = __dirname

  // Default destination inside platforms/<platform>/...
  const dist = resolve(
    projectRoot,
    nsWebpack.getAppPath(platform, projectRoot)
  )

  const {
    // The 'appPath' and 'appResourcesPath' values are fetched from
    // the nsconfig.json configuration file.
    appPath = 'app',
    appResourcesPath = 'app/App_Resources',

    // You can provide the following flags when running 'tns run android|ios'
    snapshot, // --env.snapshot
    production, // --env.production
    uglify, // --env.uglify
    report, // --env.report
    sourceMap, // --env.sourceMap
    hiddenSourceMap, // --env.hiddenSourceMap
    hmr, // --env.hmr,
    unitTesting, // --env.unitTesting,
    verbose, // --env.verbose
    snapshotInDocker, // --env.snapshotInDocker
    skipSnapshotTools, // --env.skipSnapshotTools
    compileSnapshot // --env.compileSnapshot
  } = env

  const useLibs = compileSnapshot
  const isAnySourceMapEnabled = !!sourceMap || !!hiddenSourceMap
  const externals = nsWebpack.getConvertedExternals(env.externals)

  const appFullPath = resolve(projectRoot, appPath)
  const appResourcesFullPath = resolve(projectRoot, appResourcesPath)

  const entryModule = nsWebpack.getEntryModule(appFullPath, platform)
  const entryPath = `.${sep}${entryModule}.ts`
  const entries = { bundle: entryPath }

  const tsConfigPath = resolve(projectRoot, 'tsconfig.tns.json')

  const areCoreModulesExternal =
    Array.isArray(env.externals) &&
    env.externals.some(e => e.indexOf('tns-core-modules') > -1)
  if (platform === 'ios' && !areCoreModulesExternal) {
    entries['tns_modules/tns-core-modules/inspector_modules'] =
      'inspector_modules'
  }

  const sourceMapFilename = nsWebpack.getSourceMapFilename(
    hiddenSourceMap,
    __dirname,
    dist
  )

  const itemsToClean = [`${dist}/**/*`]
  if (platform === 'android') {
    itemsToClean.push(
      `${join(
        projectRoot,
        'platforms',
        'android',
        'app',
        'src',
        'main',
        'assets',
        'snapshots'
      )}`
    )
    itemsToClean.push(
      `${join(
        projectRoot,
        'platforms',
        'android',
        'app',
        'build',
        'configurations',
        'nativescript-android-snapshot'
      )}`
    )
  }

  const noEmitOnErrorFromTSConfig = getNoEmitOnErrorFromTSConfig(tsConfigPath)

  nsWebpack.processAppComponents(appComponents, platform)
  const config = {
    mode: production ? 'production' : 'development',
    context: appFullPath,
    externals,
    watchOptions: {
      ignored: [
        appResourcesFullPath,
        // Don't watch hidden files
        '**/.*'
      ]
    },
    target: nativescriptTarget,
    entry: entries,
    output: {
      pathinfo: false,
      path: dist,
      sourceMapFilename,
      libraryTarget: 'commonjs2',
      filename: '[name].js',
      globalObject: 'global',
      hashSalt
    },
    resolve: {
      extensions: ['.ts', '.mjs', '.js', '.svelte', '.scss', '.css'],
      // Resolve {N} system modules from tns-core-modules
      modules: [
        resolve(__dirname, 'node_modules/tns-core-modules'),
        resolve(__dirname, 'node_modules'),
        'node_modules/tns-core-modules',
        'node_modules'
      ],
      alias: {
        '~': appFullPath
      },
      // resolve symlinks to symlinked modules
      symlinks: true
    },
    resolveLoader: {
      // don't resolve symlinks to symlinked loaders
      symlinks: false
    },
    node: {
      // Disable node shims that conflict with NativeScript
      http: false,
      timers: false,
      setImmediate: false,
      fs: 'empty',
      __dirname: false
    },
    devtool: hiddenSourceMap
      ? 'hidden-source-map'
      : sourceMap
        ? 'inline-source-map'
        : 'none',
    optimization: {
      runtimeChunk: 'single',
      noEmitOnErrors: noEmitOnErrorFromTSConfig,
      splitChunks: {
        cacheGroups: {
          vendor: {
            name: 'vendor',
            chunks: 'all',
            test: (module, chunks) => {
              const moduleName = module.nameForCondition
                ? module.nameForCondition()
                : ''
              return (
                /[\\/]node_modules[\\/]/.test(moduleName) ||
                appComponents.some(comp => comp === moduleName)
              )
            },
            enforce: true
          }
        }
      },
      minimize: !!uglify,
      minimizer: [
        new TerserPlugin({
          parallel: true,
          cache: true,
          sourceMap: isAnySourceMapEnabled,
          terserOptions: {
            output: {
              comments: false,
              semicolons: !isAnySourceMapEnabled
            },
            compress: {
              // The Android SBG has problems parsing the output
              // when these options are enabled
              collapse_vars: platform !== 'android',
              sequences: platform !== 'android'
            }
          }
        })
      ]
    },
    module: {
      rules: [
        {
          include: join(appFullPath, entryPath),
          use: [
            // Require all Android app components
            platform === 'android' && {
              loader: 'nativescript-dev-webpack/android-app-components-loader',
              options: { modules: appComponents }
            },

            {
              loader: 'nativescript-dev-webpack/bundle-config-loader',
              options: {
                loadCss: !snapshot, // load the application css if in debug mode
                unitTesting,
                appFullPath,
                projectRoot,
                ignoredFiles: nsWebpack.getUserDefinedEntries(entries, platform)
              }
            }
          ].filter(loader => !!loader)
        },

        {
          test: /\.(ts|css|scss|html|xml)$/,
          use: 'nativescript-dev-webpack/hmr/hot-loader'
        },

        {
          test: /\.(html|xml)$/,
          use: 'nativescript-dev-webpack/xml-namespace-loader'
        },

        {
          test: /\.css$/,
          use: 'nativescript-dev-webpack/css2json-loader'
        },

        {
          test: /\.scss$/,
          use: ['nativescript-dev-webpack/css2json-loader', 'sass-loader']
        },

        {
          test: /\.mjs$/,
          type: 'javascript/auto'
        },
        {
          test: /\.ts$/,
          use: {
            loader: 'ts-loader',
            options: {
              configFile: tsConfigPath,
              // https://github.com/TypeStrong/ts-loader/blob/ea2fcf925ec158d0a536d1e766adfec6567f5fb4/README.md#faster-builds
              // https://github.com/TypeStrong/ts-loader/blob/ea2fcf925ec158d0a536d1e766adfec6567f5fb4/README.md#hot-module-replacement
              transpileOnly: true,
              allowTsInNodeModules: true,
              compilerOptions: {
                sourceMap: isAnySourceMapEnabled,
                declaration: false
              }
            }
          }
        },
        {
          test: /\.svelte$/,
          exclude: /node_modules/,
          use: [
            {
              loader: 'svelte-loader',
              options: {
                preprocess: svelteNativePreprocessor(),
                hotReload: true,
                hotOptions: {
                  native: true
                }
              }
            }
          ]
        }
      ]
    },
    plugins: [
      // Define useful constants like TNS_WEBPACK
      new webpack.DefinePlugin({
        'global.TNS_WEBPACK': 'true',
        process: 'global.process'
      }),
      // Remove all files from the out dir.
      new CleanWebpackPlugin(itemsToClean, { verbose: !!verbose }),
      // Copy assets to out dir. Add your own globs as needed.
      new CopyWebpackPlugin(
        [
          { from: { glob: 'fonts/**' } },
          { from: { glob: '**/*.jpg' } },
          { from: { glob: '**/*.png' } }
        ],
        { ignore: [`${relative(appPath, appResourcesFullPath)}/**`] }
      ),
      new nsWebpack.GenerateNativeScriptEntryPointsPlugin('bundle'),
      // For instructions on how to set up workers with webpack
      // check out https://github.com/nativescript/worker-loader
      new NativeScriptWorkerPlugin(),
      new nsWebpack.PlatformFSPlugin({
        platform,
        platforms
      }),
      // Does IPC communication with the {N} CLI to notify events when running in watch mode.
      new nsWebpack.WatchStateLoggerPlugin(),
      // https://github.com/TypeStrong/ts-loader/blob/ea2fcf925ec158d0a536d1e766adfec6567f5fb4/README.md#faster-builds
      // https://github.com/TypeStrong/ts-loader/blob/ea2fcf925ec158d0a536d1e766adfec6567f5fb4/README.md#hot-module-replacement
      new ForkTsCheckerWebpackPlugin({
        tsconfig: tsConfigPath,
        async: false,
        useTypescriptIncrementalApi: true,
        checkSyntacticErrors: true,
        memoryLimit: 4096
      }),
      new MeteorImportsPlugin({
        ROOT_URL: 'http://localhost:3000/',
        PUBLIC_SETTINGS: {},
        meteorFolder: 'web',
        meteorProgramsFolder: '../web/.meteor/local/build/programs',
        legacy: true,
        meteorEnv: { NODE_ENV: 'development' },
        exclude: {
          // 'global-imports': true,
          reload: true,
          ecmascript: true,
          'es5-shim': true,
          underscore: true,
          'hwillson:stub-collections': true,
          autoupdate: { mode: 'development' },
          'ecmascript-runtime-client': '{Map,Symbol,Set}'
        },
        excludeGlobals: ['ecmascript-runtime-client', '_']
      })
    ]
  }

  if (report) {
    // Generate report files for bundles content
    config.plugins.push(
      new BundleAnalyzerPlugin({
        analyzerMode: 'static',
        openAnalyzer: false,
        generateStatsFile: true,
        reportFilename: resolve(projectRoot, 'report', 'report.html'),
        statsFilename: resolve(projectRoot, 'report', 'stats.json')
      })
    )
  }

  if (snapshot) {
    config.plugins.push(
      new nsWebpack.NativeScriptSnapshotPlugin({
        chunk: 'vendor',
        requireModules: ['tns-core-modules/bundle-entry-points'],
        projectRoot,
        webpackConfig: config,
        snapshotInDocker,
        skipSnapshotTools,
        useLibs
      })
    )
  }

  if (hmr) {
    config.plugins.push(new webpack.HotModuleReplacementPlugin())
  }

  return config
}

I'll keep playing with it.

[CaptainN]

I also have questions - why does it pull in a solid bundle, instead of individual packages? Is it because of a lack of a dependency graph? I wonder if we could add something to Meteor to get whatever else we'd need to allow us to import components piecemeal. Meteor has code to output a dep tree to the console using meteor list --tree, maybe that could be added to the program.json file, or a second tree.json file. What do you think?

[CaptainN]

I wonder if an alternative way to get the dep graph for an individual meteor package would be to set up a Package Proxy shim, and then load the individual module. In the Proxy, we could capture which Package nodes are accessed, and build the graph. I'd need to work up a proof of concept, but it seems at an abstract level that it could work.

The compiled meteor modules all have a list of imports at the top:

var Meteor = Package.meteor.Meteor;
var global = Package.meteor.global;
var meteorEnv = Package.meteor.meteorEnv;
var Tracker = Package.tracker.Tracker;
var Deps = Package.tracker.Deps;
var Random = Package.random.Random;
var Hook = Package['callback-hook'].Hook;
var ReactiveVar = Package['reactive-var'].ReactiveVar;
var DDP = Package['ddp-client'].DDP;
var Mongo = Package.mongo.Mongo;
var meteorInstall = Package.modules.meteorInstall;
var Promise = Package.promise.Promise;

So if we load that module in a sandbox, and shim Package with a proxy, we can get a list of the packages associated with the current imported package. (We can shim Package.modules.meteorInstall with a noop to keep it from leaking.)

// super sloppy, untested psuedo code
function getDeps() {
  const deps = []
  // icky - just set a global Package var - probably there's a better way
  Package = new Proxy({}, {
    get (key) {
      deps.push(key)
      // return a deadend proxy, or something more robust to prevent subnode errors
      return new Proxy({}, {get () {} })
    }
  })
  Package.modules.meteorInstall = () => {}
  // load the module
  return deps
}

[jedwards1211]

@CaptainN mainly because Meteor's old package system relied adding packages as properties to a global Package variable, e.g. import { Meteor } from 'meteor/meteor' gets turned into Package.meteor.Meteor, and we haven't gone to any trouble to try to work with the dep graph just because of the amount of effort it would take to get right...

I can't tell what loader is being applied to node_modules/meteor-imports-webpack-plugin/meteor-imports.js but I don't see a .js loader in your webpack rules. I think you'll have to use babel-loader to handle the .js files in this package, surprisingly. I thought there would be a webpack loader for js that doesn't rely on babel, but the only other one I know of is script-loader and I don't think it converts require statements.

[jedwards1211]

@CaptainN it would be nice to reduce the bundle size, but at the time this plugin was written, even Meteor itself would load all the packages on startup in the browser. Do you know if Meteor has made any improvements to that?

[jedwards1211]

@CaptainN shimming Package with a proxy would be awkward because you would have to actually execute the code to find out what depends on what (and executing some of the packages in node might not even work since they're intended for the browser). It would be better to parse and analyze the AST to see what properties of Package are being accessed.

[CaptainN]

It looks like meteor makes a separate file for each package now:

I'm inferring quite a bit here though.

I'll look into that loader issue - this webpack config is mostly from a template - I'm surprised there's not already a handler for js files...

[jedwards1211]

@CaptainN that's always been the case that they're separate files, but AFAIK it always includes script tags for all of them, not just what you need to use on a given page.

[CaptainN]

Well, it's a bit more complicated with dynamic-imports (I think since Meteor 1.5), though for webpack that's probably not relevant. I think the piece I'm missing is why it's necessary to include everything in a webpack build. I guess if you are using it as an alternative packager for a web project, then it makes sense to include everything. Since I'm looking to use it in more of an alternative - webpack for NativeScript (or for React Native, or Cordova even), I'm more looking to cherry pick packages (hopefully from a git submodule with my full web app, to keep everything in sync).

[jedwards1211]

Ah, the reason we include stuff in the webpack build is for projects that serve up their own HTML instead of the HTML generated by Meteor.

I do have babel-plugin-meteor-imports which transpiles the import statements but doesn't bundle any meteor code. Babel does support typescript now, so that's an option for you if you're willing to switch from ts-loader to babel-loader. Or maybe it would be possible to pass the output of ts-loader through babel-loader?

[CaptainN]

I tried it both ways - importing the js with ts-laoder using "allowJs", and with a "babel-loader" config I quickly ported from another project. Either way I get this error:

ERROR in ../node_modules/meteor-imports-webpack-plugin/meteor-imports.js
eposle nest-svelte-mobileweb.meteorlocauildprogramsweb.browserpackagesecmascript.js' in 'C:\Users\Kevin\repos\test-svelte-mobile\node_modules\meteor-imports-webpack-plugin'
 @ ../node_modules/meteor-imports-webpack-plugin/meteor-imports.js 7:0-118
 @ ./app.ts

Looking through the code, I think maybe some of that assumes *NIX pathing. I'm going to see if I make a patch for it using require.resolve or something. (The whole family is home for Thanksgiving - they are very distracting, making for slow going lol)

[jedwards1211]

Slashes aside, it's weird that repos\test-svelte-mobile somehow got turned into eposle nest-svelte-mobile?

[CaptainN]

I assumed the output got mangled - it's Windows after all..

[CaptainN]

This was straight forward. The path was getting output to a javascript string. On Windows, that means the backslashes need to be escaped. I simply replaced that one concatenation line with a JSON.stringify wrapper, and it solved the problem.

    // Require all packages
    for (let pkg of packages) {
      if (pkg.source)
        output += 'Package._define("' + pkg.name + '", ' + pkg.source + ');\n';
      else
        output += 'require(' + JSON.stringify(path.join(meteorBuild, pkg.path)) + ');\n';
    }

[jedwards1211]

Okay cool, I'm confused though, where did you put that code?

[CaptainN]

I put it in a PR staeke#1

[gustawdaniel]

Can we merge this pull request? It seems to be connected with problem from issue #26