Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pa11y reporters do not output to console when used with Pa11y CI #174

Open
aarongoldenthal opened this issue Feb 22, 2022 · 4 comments
Open

Comments

@aarongoldenthal
Copy link
Contributor

When pa11y loads reporters, it wraps the reporter functions so that any output from any reporter function is written to console (here). That logic is missing from pa11y-ci (it should probably be here). So, when pa11y reporters are used with pa11y-ci, any reporters that returns a value expecting it to externally be written to console have their output lost. This includes the cli, csv, html, and tsv reporters. The json reporter is not impacted since it directly writes its output.

The dedicated pa11y-ci reporters handle their own output so are not effected.

@aarongoldenthal
Copy link
Contributor Author

After thinking about this a little, if pa11y-ci were updated to simply write the value returned from a reporter to stdout, as pa11y does (in most cases), then it would do the same for all analyzed files. The results could be piped to a file, but the result wouldn't be valid for the given format (excluding cli for the moment, and technically tsv and csv would but would have headers repeated for every file).

An alternative proposal - create a new reporter that's essentially a shim for using any pa11y reporter:

  • Assume one reporter, requiring configuration to specify the pa11y reporter
  • This reporter would take the output from the specified pa11y reporter results function, create a unique file name based on the URL, and save the results to the file (which would then be valid for the given format)
  • The reporter configuration could also include a destination directory
  • My original thought was exclude the cli reporter, although it could be used, but would probably want to strip out the ANSI escape codes
  • For errors, the output from the reporter error function would be written to stderr

For the pa11y reporters:

  • The csv, html, and tsv reporters would work with the above implementation
  • The json reporter writes to stdout, but could be changed to simply return the results, which would then work with the implementation above. Pa11y already writes that output to stdout, so would be functionally transparent to the user.
  • The cli reporter could be updated to write to stdout itself (as the json reporter does now). This would be transparent for pa11y, but would allow the reporter to be used directly with pa11y-ci with the expected output. In this case it is valid and logical output on its own.

@langalex
Copy link

It's not much but I took the JSON reporter and changet it to wrap the pa11y HTML reporter:

'use strict';

const fs = require('fs');
const { resolve, isAbsolute, dirname } = require('path');
const pa11yHtmlReporter = require('pa11y/lib/reporters/html');

function writeReport(fileName, data) {
  try {
    const dirPath = dirname(fileName);
    if (!(fs.existsSync(dirPath) && fs.lstatSync(dirPath).isDirectory())) {
      fs.mkdirSync(dirPath, { recursive: true });
    }
    fs.writeFileSync(fileName, data);
  } catch (error) {
    console.error(`Unable to write ${fileName}`);
    console.error(error);
  }
}

function resolveFile(fileName) {
  if (typeof fileName !== 'string') {
    return null;
  }
  return isAbsolute(fileName) ? fileName : resolve(process.cwd(), fileName);
}

module.exports = function htmlReporter(options = {}) {
  const fileName = resolveFile(options.fileName);
  return {
    results(report) {
      return pa11yHtmlReporter.results(report).then((html) => {
        // If reporter options specify an output file, write to file.
        // Otherwise, write to console.
        if (fileName) {
          writeReport(fileName, html);
        } else {
          console.log(html);
        }
      });
    },
  };
};

Config:

{
  "defaults": {
    "reporters": [
      "cli",
      ["./pa11y-ci-html-reporter.js", { "fileName": "./pa11y-cli-report.html" }]
    ]
  }
}

It now writes its output either to a file or console.

@aarongoldenthal
Copy link
Contributor Author

@langalex if you're just looking for an HTML reporter for pa11y-ci, you can try https://www.npmjs.com/package/pa11y-ci-reporter-html, which produces a report for each page as well as a summary (for full disclosure, it's my npm package)

@langalex
Copy link

Thanks. I ended up creating my own Markdown report so I could attach it to a PR check summary here on GitHub.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants