-
-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tinybench runs the test cases in parallel which is bad for tests that access resources like files #71
Comments
Sure, @sirenkovladd might be interested? |
Also relevant in context of vitest vitest-dev/vitest#5412 |
Tasks are already running sequentially. https://github.com/tinylibs/tinybench/blob/main/test/sequential.test.ts So let me know what is the specific issue! As far as I know, vitest does not use bench.run from tinybench and it implements its own way. |
@BioPhoton Do you have any reproduciton? |
I have the following suite: // eslint-ignore-next-line import/no-named-as-default-member
import fastGlob from 'fast-glob';
import { glob } from 'glob';
import { globby } from 'globby';
import { join } from 'node:path';
import yargs from 'yargs';
const cli = yargs(process.argv).options({
pattern: {
type: 'array',
string: true,
default: [
join(process.cwd(), 'node_modules/**/*.md'),
],
},
outputDir: {
type: 'string',
},
logs: {
type: 'boolean',
default: true,
},
});
// eslint-disable-next-line n/no-sync
const { pattern, logs } = cli.parseSync();
if (logs) {
// eslint-disable-next-line no-console
console.log('You can adjust the test with the following arguments:');
// eslint-disable-next-line no-console
console.log(
`pattern glob pattern of test --pattern=${pattern.toString()}`,
);
}
const fastGlobName = 'fast-glob';
const globName = 'glob';
const globbyName = 'globby';
// ==================
const suiteConfig = {
suiteName: 'glob-matching',
cases: [
[fastGlobName, callAndValidate(fastGlob.async, pattern, fastGlobName)],
[globName, callAndValidate(glob, pattern, globName)],
[globbyName, callAndValidate(globby, pattern, globbyName)],
]
};
export default suiteConfig;
// ==============================================================
const logged: Record<string, boolean> = {};
function callAndValidate<T = string | string[]>(
fn: (patterns: T) => Promise<unknown[]>,
globPatterns: T,
fnName: string,
) {
return async () => {
const result = await fn(globPatterns);
if (result.length === 0) {
throw new Error(`Result length is ${result.length}. This test is not meaningful`);
} else {
if (!logged[fnName]) {
logged[fnName] = true;
console.log(
`${fnName} found ${result.length} files for pattern ${pattern.join(
', ',
)}`,
);
}
}
};
} Somehow it produces different test results than expected. |
Ok, I'll check this out soon. |
I do not see where tinybench API is used ... are you sure the bug is filled against the right project? |
Problem:
I wrote a benchmark to compare file crawling and get quite different results compared to e.g. benchmark. With benchmark I'm able to run tests in sequence so they can all after another access the same resources. In this way the results are reliable.
With tinybench I throw out of memory or other odd errors due to parallel processing.
Suggested solution:
Implement a way to run all test cases in sequence.
Links:
The text was updated successfully, but these errors were encountered: