Skip to content

Commit

Permalink
smoke tests: add ability to target build (#4880)
Browse files Browse the repository at this point in the history 
### Intent
Restructure the Mocha test runner code to ensure that all test setup and
environment preparation steps are executed before the actual test suite
begins. This restructuring was necessary to allow the test runner to
properly target a specific Positron build using the `--build` flag.

### Approach
* Moved all environment setup and configuration logic to run before the
test suite starts.
* Refactored code to handle `--build` path validation and setup early in
the execution.
* Ensured environment variables are correctly defined during the setup
phase.

### QA Notes
To target a build:
```
yarn smoketest-pr --build /Applications/Positron.app
```

The console will log the version of the targeted build:
<img width="702" alt="Screenshot 2024-10-03 at 9 39 49 AM"
src="https://github.com/user-attachments/assets/47b6d127-2f70-496b-b275-d209de40eea4">

* Ran full test suite against PR and things look good. 👍
  • Loading branch information
@midleman
midleman authored Oct 3, 2024
1 parent 9137831 commit f1ae7a3
Show file tree
Hide file tree
Showing 6 changed files with 284 additions and 209 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/positron-full-test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ jobs:
POSITRON_PY_VER_SEL: 3.10.12
POSITRON_R_VER_SEL: 4.4.0
id: electron-smoke-tests
run: DISPLAY=:10 yarn smoketest-all --tracing --parallel --jobs 2
run: DISPLAY=:10 yarn smoketest-all --tracing --parallel --jobs 2 --skip-cleanup

- name: Run Web Smoke Tests
if: always()
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/positron-merge-to-main.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ jobs:
POSITRON_PY_VER_SEL: 3.10.12
POSITRON_R_VER_SEL: 4.4.0
id: electron-smoke-tests
run: DISPLAY=:10 yarn ${{ env.SMOKETEST_TARGET }} --tracing --parallel --jobs 2
run: DISPLAY=:10 yarn ${{ env.SMOKETEST_TARGET }} --tracing --parallel --jobs 2 --skip-cleanup

- name: Convert XUnit to JUnit
id: xunit-to-junit
Expand Down
10 changes: 10 additions & 0 deletions test/smoke/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,16 @@ The following smoketest scripts are currently available:
- `smoketest-win`: Runs tests tagged with `#win` (Windows)
- `smoketest-pr`: Runs tests tagged with `#pr`

#### Target a Positron Build

You can specify a custom build of Positron to run your tests against using the `--build` option. This allows you to point to a local installation or a specific build of the application.

```bash
yarn smoketest-pr --build /Applications/Positron.app --parallel --jobs 3
```

**Note:** During the setup phase, the script will automatically detect and display the version of Positron being tested. This helps verify that the correct build is being used.

## Test Project

Before any of the tests start executing the test framework clones down the [QA Content Examples](https://github.com/posit-dev/qa-example-content) repo. This repo contains R and Python files that are run by the automated tests and also includes data files (such as Excel, SQLite, & parquet) that support the test scripts. If you make additions to QA Content Examples for a test, please be sure that the data files are free to use in a public repository.
Expand Down
8 changes: 3 additions & 5 deletions test/smoke/src/areas/positron/dataexplorer/dataexplorer.test.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,8 @@ describe('Data Explorer #web #win', () => {
describe('Python Pandas Data Explorer #pr', () => {

before(async function () {

this.timeout(120000);
await PositronPythonFixtures.SetupFixtures(this.app as Application);

});

after(async function () {
Expand Down Expand Up @@ -204,9 +203,8 @@ df2 = pd.DataFrame(data)`;
// https://github.com/posit-dev/positron/issues/4663
describe('Python Polars Data Explorer #pr', () => {
before(async function () {

this.timeout(120000);
await PositronPythonFixtures.SetupFixtures(this.app as Application);

});

after(async function () {
Expand Down Expand Up @@ -341,8 +339,8 @@ df2 = pd.DataFrame(data)`;
describe('R Data Explorer', () => {

before(async function () {
this.timeout(120000);
await PositronRFixtures.SetupFixtures(this.app as Application);

});

it('R - Verifies basic data explorer functionality [C609620] #pr', async function () {
Expand Down
143 changes: 22 additions & 121 deletions test/smoke/src/positronUtils.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,13 @@
import * as fs from 'fs';
import * as path from 'path';
import * as mkdirp from 'mkdirp';
import * as vscodetest from '@vscode/test-electron';
import fetch from 'node-fetch';
import { MultiLogger, ConsoleLogger, FileLogger, Logger, measureAndLog, getBuildElectronPath, getBuildVersion, getDevElectronPath } from '../../automation';
import { installAllHandlers, retry } from './utils';
import { MultiLogger, ConsoleLogger, FileLogger, Logger, } from '../../automation';
import { installAllHandlers, } from './utils';

let version: string | undefined;
export const ROOT_PATH = path.join(__dirname, '..', '..', '..');
const TEST_DATA_PATH = process.env.TEST_DATA_PATH || 'TEST_DATA_PATH not set';
const WORKSPACE_PATH = path.join(TEST_DATA_PATH, 'qa-example-content');
const EXTENSIONS_PATH = path.join(TEST_DATA_PATH, 'extensions-dir');
const WORKSPACE_PATH = process.env.WORKSPACE_PATH || 'WORKSPACE_PATH not set';
const EXTENSIONS_PATH = process.env.EXTENSIONS_PATH || 'EXTENSIONS_PATH not set';
const LOGS_DIR = process.env.BUILD_ARTIFACTSTAGINGDIRECTORY || 'smoke-tests-default';

const asBoolean = (value: string | undefined): boolean | undefined => {
Expand All @@ -32,6 +29,7 @@ const OPTS: ParseOptions = {
headless: asBoolean(process.env.HEADLESS),
browser: process.env.BROWSER,
electronArgs: process.env.ELECTRON_ARGS,
version: process.env.BUILD_VERSION,
};

/**
Expand All @@ -48,8 +46,7 @@ export function setupAndStartApp(): Logger {
// Create a new logger for this suite
const logger = createLogger(logsRootPath);

// Set up environment, hooks, etc
setupTestEnvironment(logger);
// Set test defaults and before/after hooks
setTestDefaults(logger, logsRootPath, crashesRootPath);
installAllHandlers(logger);

Expand Down Expand Up @@ -83,70 +80,15 @@ function getTestFileName(): string {
}
}

function setupTestEnvironment(logger: Logger) {
//
// #### Electron Smoke Tests ####
//

if (!OPTS.web) {
let testCodePath = OPTS.build;
let electronPath: string;

if (testCodePath) {
electronPath = getBuildElectronPath(testCodePath);
version = getBuildVersion(testCodePath);
} else {
testCodePath = getDevElectronPath();
electronPath = testCodePath;
process.env.VSCODE_REPOSITORY = ROOT_PATH;
process.env.VSCODE_DEV = '1';
process.env.VSCODE_CLI = '1';
}

if (!fs.existsSync(electronPath || '')) {
throw new Error(`Cannot find VSCode at ${electronPath}. Please run VSCode once first (scripts/code.sh, scripts\\code.bat) and try again.`);
}

if (OPTS.remote) {
logger.log(`Running desktop remote smoke tests against ${electronPath}`);
} else {
logger.log(`Running desktop smoke tests against ${electronPath}`);
}
}

//
// #### Web Smoke Tests ####
//
else {
const testCodeServerPath = OPTS.build || process.env.VSCODE_REMOTE_SERVER_PATH;

if (typeof testCodeServerPath === 'string') {
if (!fs.existsSync(testCodeServerPath)) {
throw new Error(`Cannot find Code server at ${testCodeServerPath}.`);
} else {
logger.log(`Running web smoke tests against ${testCodeServerPath}`);
}
}

if (!testCodeServerPath) {
process.env.VSCODE_REPOSITORY = ROOT_PATH;
process.env.VSCODE_DEV = '1';
process.env.VSCODE_CLI = '1';

logger.log(`Running web smoke out of sources`);
}
}
}

/**
* Set the default options for the test suite.
*
* @param logger the logger instance for the test suite
* @param logsRootPath the root path for the logs
* @param crashesRootPath the root path for the crashes
*/
function setTestDefaults(logger: Logger, logsRootPath: string, crashesRootPath: string) {
before(async function () {
this.timeout(5 * 60 * 1000); // increase timeout for downloading VSCode

if (!OPTS.web && !OPTS.remote && OPTS.build) {
// Only enabled when running with --build and not in web or remote
await measureAndLog(() => ensureStableCode(TEST_DATA_PATH, logger, OPTS), 'ensureStableCode', logger);
}

this.defaultOptions = {
codePath: OPTS.build,
workspacePath: WORKSPACE_PATH,
Expand All @@ -166,7 +108,13 @@ function setTestDefaults(logger: Logger, logsRootPath: string, crashesRootPath:
});
}

function createLogger(logsRootPath: string): Logger {
/**
* Create a logger instance.
*
* @param logsRootPath the root path for the logs
* @returns Logger instance
*/
export function createLogger(logsRootPath: string, logsFileName = `smoke-test-runner.log`): Logger {
const loggers: Logger[] = [];

if (OPTS.verbose) {
Expand All @@ -176,59 +124,11 @@ function createLogger(logsRootPath: string): Logger {
fs.rmSync(logsRootPath, { recursive: true, force: true, maxRetries: 3 });
mkdirp.sync(logsRootPath);

loggers.push(new FileLogger(path.join(logsRootPath, `smoke-test-runner.log`)));
loggers.push(new FileLogger(path.join(logsRootPath, logsFileName)));

return new MultiLogger(loggers);
}

function parseVersion(version: string): { major: number; minor: number; patch: number } {
const [, major, minor, patch] = /^(\d+)\.(\d+)\.(\d+)/.exec(version)!;
return { major: parseInt(major), minor: parseInt(minor), patch: parseInt(patch) };
}


async function ensureStableCode(testDataPath: string, logger: Logger, opts: any): Promise<void> {
let stableCodePath = opts['stable-build'];

if (!stableCodePath) {
const current = parseVersion(version!);
const versionsReq = await retry(() => measureAndLog(() => fetch('https://update.code.visualstudio.com/api/releases/stable'), 'versionReq', logger), 1000, 20);

if (!versionsReq.ok) {
throw new Error('Could not fetch releases from update server');
}

const versions: string[] = await measureAndLog(() => versionsReq.json(), 'versionReq.json()', logger);
const stableVersion = versions.find(raw => {
const version = parseVersion(raw);
return version.major < current.major || (version.major === current.major && version.minor < current.minor);
});

if (!stableVersion) {
throw new Error(`Could not find suitable stable version for ${version}`);
}

logger.log(`Found VS Code v${version}, downloading previous VS Code version ${stableVersion}...`);

const stableCodeDestination = path.join(testDataPath, 's');
const stableCodeExecutable = await retry(() => measureAndLog(() => vscodetest.download({
cachePath: stableCodeDestination,
version: stableVersion,
extractSync: true,
}), 'download stable code', logger), 1000, 3);

stableCodePath = path.dirname(stableCodeExecutable);
}

if (!fs.existsSync(stableCodePath)) {
throw new Error(`Cannot find Stable VSCode at ${stableCodePath}.`);
}

logger.log(`Using stable build ${stableCodePath} for migration tests`);

opts['stable-build'] = stableCodePath;
}

type ParseOptions = {
verbose?: boolean;
remote?: boolean;
Expand All @@ -240,4 +140,5 @@ type ParseOptions = {
'stable-build'?: string;
browser?: string;
electronArgs?: string;
version?: string;
};
Loading

0 comments on commit f1ae7a3

Please sign in to comment.