Skip to content

Commit

Permalink
Merge branch 'GPII-3308'
Browse files Browse the repository at this point in the history
* GPII-3308:
  GPII-3308:  Added "node_modules" test fixture content blocked by my global gitignore.
  GPII-3308: Fixed typo.
  GPII-3308: Updated based on PR feedback.
  GPII-3308: Standardised and added ability to control 'invalid pattern' rules.
  GPII-3308: Initial implementation.
  • Loading branch information
amb26 committed Sep 19, 2018
2 parents 8d687c5 + 4aae5fb commit ec11577
Show file tree
Hide file tree
Showing 23 changed files with 1,091 additions and 1 deletion.
6 changes: 6 additions & 0 deletions .eslintrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"extends": "eslint-config-fluid",
"env": {
"node": true
}
}
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
node_modules
package-lock.json
coverage
reports
.vagrant
6 changes: 6 additions & 0 deletions .nycrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"reporter": ["html", "text-summary"],
"report-dir": "reports",
"temp-directory": "coverage",
"include": ["src/**/*.js"]
}
25 changes: 25 additions & 0 deletions .vagrant.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# QI Configuration file, see: https://github.com/amatas/vagrant-gpii-ci
---

env:
vms:
windows10:
cpu: 2 # number of cpus
memory: 2048 # amount of RAM memory (in Mb)
clone: true # use the linked_clone Vagrant feature
autostart: true
box: inclusivedesign/windows10-eval-x64-Apps

stages: # Stages to perform when 'ci test' command is invoked
- setup # Install our system-level dependencies, etc.
- test # Run the actual tests

setup_job:
stage: setup
script:
- choco install nodejs-lts -y

test_job:
stage: test # name of the stage
script: # One line per command to execute
- "do.ps1 -c 'v: && npm install && npm test'"
32 changes: 32 additions & 0 deletions Gruntfile.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
/* eslint-env node */
"use strict";
var fluid = require("infusion");
var gpii = fluid.registerNamespace("gpii");
require("./");

module.exports = function (grunt) {
var globbedSources = {
md: ["./*.md"],
js: ["./*.js", "./src/**/*.js", "./tests/**/*.js", "./tests/*.js", "./*.js"],
json: ["./*.json", "!./package-lock.json"],
json5: [],
other: ["./.*"]
};

// We manually resolve our globs to raw paths to ensure that our code is used rather than
// the copy of ourselves we inherit from gpii-grunt-lint-all. In regular usage, you should
// simply pass the globs themselves.
var fullPathSources = fluid.transform(globbedSources, function (globbedPaths) {
return gpii.glob.findFiles("%gpii-glob", globbedPaths, [], {dot: true});
});

grunt.initConfig({
lintAll: {
sources: fullPathSources,
expandPaths: false // This package is really the only one that should use this option.
}
});

grunt.loadNpmTasks("gpii-grunt-lint-all");
grunt.registerTask("lint", "Perform all standard lint checks.", ["lint-all"]);
};
160 changes: 159 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,160 @@
# gpii-glob
GPII Glob is for....

This library provides a means of determining a list of relevant files from a given location based on one or more
"globbed" patterns. Its API consists of a single static function (see below).

This package was written as a drop-in replacement for grunt's globbing, which can be quite inefficient when working with
packages that have a lot of dependencies or other content that you wish to exclude. In short, grunt's strategy appears
to be to perform a full recursive scan of the directory before deciding whether to include or exclude files. By
contrast, this package only recursively scans content that potentially matches at least one "include". In real-world
testing, this strategy seems to result in linting runs that take around a tenth of the time.

To achieve this, this package disallows overly broad patterns like '**/*.js' and some of the more advanced features of
the underlying library, such as regular expressions. See below for full details.

## `gpii.glob.findFiles(rootPath, includes, [excludes], [minimatchOptions])`

* `rootPath`: A full or package-relative directory to scan for matching files.
* `includes`: An `Array` of glob patterns that should be included in the results.
* `excludes`: An optional `Array` of glob patterns that should be excluded from the results.
* `minimatchOptions`: An optional `Object` detailing configuration options to be passed to [minimatch](https://github.com/isaacs/minimatch#options).
* Returns: An `Array` of full paths to files that match the supplied glob patterns.

## "glob" Patterns

A "glob" pattern is a string that describes the path to one or more files. It may contain single-asterisk wildcards
such as `*.js`. Single asterisks are only matched within a single path segment. So, for example, `./test/*-node*.js`
matches `./test/all-node-tests.js`, but not `./test/js/another-node-test.js`.

"Glob" patterns may also use double-asterisks to indicate that any number of subdirectories may appear between one part
of a pattern and the next. So, for example, `./src/**/*.js` matches `./src/index.js` as well as
`./src/js/other.js`.

The underlying concept of a glob is powerful, but can lead to inefficient lookup strategies. For the sake of
performance, the following are not allowed in patterns used with this library:

1. Patterns starting with `./**` or `**`, which might require traversing all subdirectories before excludes can be
applied.
2. Patterns that attempt to break out of the starting directory, i.e. that start with `../`.
3. Patterns that use regular expressions to represent one or more parts of the path.
4. Patterns that use the windows backslash separator in any part of the path.

Patterns can be negated by prepending an exclamation point. This mechanism allows you to define a more general rule and
then identify one or more exceptions to that rule. See below for examples.

## Usage Examples

Let's say you have a package called "my-package" whose structure looks roughly as diagrammed in this list:

* (repository root)
* README.md
* index.js
* package.json
* .gitignore
* .eslintrc.json
* src
* lib
* forked-deps.js
* js
* index.js
* tests
* all-tests.js
* js
* test1.js
* test2.js

Let's start by demonstrating includes. Content can only be brought into scope by a regular (non-negated) include.

```javascript
"use strict";
var fluid = require("infusion");
var gpii = fluid.registerNamespace("gpii");

require("gpii-glob");

// Let's assume that `fluid.module.resolvePath("%my-package")` resolves to `/source/my-package` for the purposes of
// these examples.
fluid.require("%my-package");

gpii.glob.findFiles("%my-package", [], [], {});
// Returns: An empty array, as there are no includes.

gpii.glob.findFiles("%my-package", ["./src/**/*.js"], [], {});
// Returns: ["/source/my-package/src/js/index.js", "/source/my-package/src/lib/forked-deps.js"]
```

Please note, in order to use the package-relative notation as show above, you must register your package using
[`fluid.module.register`](https://docs.fluidproject.org/infusion/development/NodeAPI.html#fluidmoduleregistername-basedir-modulerequire)
and either `require` or [`fluid.require`](https://docs.fluidproject.org/infusion/development/NodeAPI.html#fluidrequiremodulename-foreignrequire-namespace)
your package.

Negated includes and excludes take precedence over includes, i.e. they remove material from the results:

```javascript
"use strict";
var fluid = require("infusion");
var gpii = fluid.registerNamespace("gpii");

require("gpii-glob");

// Let's assume that `fluid.module.resolvePath("%my-package")` resolves to `/source/my-package` for the purposes of
// these examples.
fluid.require("%my-package");

gpii.glob.findFiles("%my-package", ["./src/**/*.js", "!./src/lib/**/*.js"], [], {});
// Returns: ["/source/my-package/src/js/index.js"]

// A negated include is basically the same as an exclude.
gpii.glob.findFiles("%my-package", ["./src/**/*.js"], ["./src/lib/**/*.js"], {});
// Also returns: ["/source/my-package/src/js/index.js"]
```

A negated exclude takes precedence over both negated includes and regular excludes.

```javascript
"use strict";
var fluid = require("infusion");
var gpii = fluid.registerNamespace("gpii");

require("gpii-glob");

// Let's assume that `fluid.module.resolvePath("%my-package")` resolves to `/source/my-package` for the purposes of
// these examples.
fluid.require("%my-package");

// A negated exclude takes precedence over both negated includes and regular excludes.
gpii.glob.findFiles("%my-package", ["./tests/**/*.js"], ["./tests/js/**/*.js", "!./tests/js/test1.js"], {});
// Returns: [
// "/source/my-package/tests/all-tests.js",
// "/source/my-package/tests/js/test1.js",
// ]
```

The file `test1.js` is brought back into the list of matching files by the negated exclude, `test2.js` remains excluded.

By default, filename wildcards (such as `*.json`) do not explicitly match "dot files". By passing the relevant option
to the underlying "minimatch" library, you can change this behaviour as shown here.

```javascript
"use strict";
var fluid = require("infusion");
var gpii = fluid.registerNamespace("gpii");

require("gpii-glob");

// Let's assume that `fluid.module.resolvePath("%my-package")` resolves to `/source/my-package` for the purposes of
// these examples.
fluid.require("%my-package");

// A filename wildcard search with the default minimatch options.
gpii.glob.findFiles("%my-package", ["./*.json"], [], {});
// Returns: ["/source/my-package/package.json"]

// A filename wildcard search with custom minimatch options.
gpii.glob.findFiles("%my-package", ["./*.json"], [], { dot: true });
// Returns: ["/source/my-package/.eslintrc.json", "/source/my-package/package.json"]
```

For a full list of minimatch options, see [their documentation](https://github.com/isaacs/minimatch#options). Please
note, minimatch options only control which files match. This package uses its own means of evaluating whether a
directory *might* contain matching content, and minimatch options will not affect this.
6 changes: 6 additions & 0 deletions index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
/* eslint-env node */
"use strict";
var fluid = require("infusion");
fluid.module.register("gpii-glob", __dirname, require);

require("./src/js/glob.js");
28 changes: 28 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"name": "gpii-glob",
"version": "1.0.0",
"description": "A library to standardise resolving \"globbed\" file patterns within GPII projects.",
"main": "index.js",
"scripts": {
"pretest": "node node_modules/rimraf/bin.js coverage/* reports/*",
"test": "node node_modules/nyc/bin/nyc.js node tests/all-tests.js"
},
"repository": "https://github.com/GPII/gpii-glob.git",
"author": "Tony Atkins <[email protected]>",
"license": "BSD-3-Clause",
"homepage": "https://github.com/GPII/gpii-glob#readme",
"dependencies": {
"infusion": "3.0.0-dev.20180801T212157Z.09bf3d438",
"minimatch": "3.0.4"
},
"devDependencies": {
"eslint": "5.1.0",
"eslint-config-fluid": "1.3.0",
"gpii-grunt-lint-all": "1.0.1-dev.20180706T153657Z.4cbbd61",
"grunt": "1.0.3",
"mkdirp": "0.5.1",
"node-jqunit": "1.1.8",
"nyc": "13.0.1",
"rimraf": "2.6.2"
}
}
Loading

0 comments on commit ec11577

Please sign in to comment.