verify

Note

Please read the instructions below carefully and return to them regularly to ensure you follow our process.

Verification is about ensuring that Wordplay works as intended. Your goal in this role is to write automated unit and end-to-end tests that help us know when something is broken and run manual tests that can't be automated. Success in this role means 1) writing automated tests, 2) finding and reporting defects in high-quality issues, and 3) helping fix defects. Verification is essential to our project's success because teachers and students won't use it if things are broken!

Start by ensuring you've completed the local setup instructions.

Here's our verification process.

Choose what to verify

Our current verification priorities are:

• WCAG 2.2 compliance. Every user interface in the application should comply with every WCAG guideline.
• Performance. Wordplay should be reasonably performant, even on older devices and slow internet connections.

Based on the priorities above, discuss with maintainers and the rest of the verification contributors where to focus your verification efforts. Identify a specific functionality to test, either at the unit level (e.g., a particular function or component in the implementation) or at the end-to-end level (a specific case of use that a creator might follow that we want to ensure works). Work with your manager to coordinate with others and avoid duplicating work. Think of yourself as part of a larger testing team.

Examine existing tests

For the functionality you've chosen, identify any existing tests. Unit or end-to-end tests:

• Our unit tests end with the .test.ts extension in the repository.
• Our end-to-end tests are stored in /tests and end with the .spec.ts extension.

Based on these, identify some important but not yet tested aspects of the unit or activity.

Create an issue

First, search the GitHub issues to verify that there isn't an existing issue for the verifications needed. If you find one and it's not assigned, you can assign it yourself. If contributors are already assigned, comment on the issue and ask if you can join the team and help.

If you don't find an issue, create one to represent the set of tests you want to write, labeling it a defect. Describe the functionality that has not yet been tested and the unit's scope or end-to-end tests you want to build. Assign yourself to the issue and any other collaborators you're working with.

Set up a draft pull request for the issue

Please follow the pull request instructions to get ready to work on the issue.

Design a test

For unit tests, that means identifying a set of inputs into a function that executes some aspect of the function that no test has yet done and identifying the intended output of that function. You don't need to write any code to design your test; you only need to specify inputs and outputs that should be expected when evaluating a particular function.

An end-to-end test identifies a series of user inputs to the application and a set of expected outputs at particular times that indicate that the application is or isn't working correctly. As with unit tests, you don't need to write any code to design a test; you need to decide the sequence of inputs a user would provide and the output you'd expect to be true.

Note

It's essential to think of test design as a different activity than test implementation so that you don't let implementation challenges bias what we choose to test.

Implement tests

Create a branch of main representing the tests you want to write. Ensure you're running the npm run test in a terminal to monitor test passes and failures. Running end-to-end tests is a bit slower, so you'll want to manually run those with npm run end2end.

Unit tests

Wordplay uses vitest for unit testing. You don't have to install anything other than the Wordplay package since test is already included in that install.

The basic concept behind the test is that .test.ts contains tests, and it will find them wherever they are in our project and run them. These tests are scattered throughout src and named to mirror the file they're testing. You can see hundreds of test examples if you search for files ending in .test.ts.

To run tests, run npm run test, a command defined in package.json. It will watch for changes and re-run tests whose dependencies have changed, so testing is fast and incremental.

Writing a test involves adding tests to an existing .test.ts file or creating a new .test.ts file and adding tests.

You can read more about automated testing and about vitest to start contributing automated unit tests. (I won't write a tutorial here since there are hundreds of tutorials on the web).

End-to-end tests

We use Playwright to automate user experience tests. These are typically called "end-to-end" tests because they test the full user experience and all the code involved in it. Unlike unit tests, these tests aren't a simple series of raw inputs and verified outputs but rather a series of user interactions and some expected output provided to the user.

Playwright runs these tests in real Chromium, Firefox, and WebKit browser engines, helping to find cross-browser issues.

All end-to-end tests are stored in /tests and must end with the extension test.ts to be recognized by Playwright as tests. Write new ones there, and ensure that the latest files you create have some coherent scope of functionality that they're testing.

Learn Playwright by reading its getting started guide and also its best practices guide. You can also review the tests already written in /tests as examples.

To run tests, use npm run end2end, which will run all Playwright tests. This is also run when a pull request on the main is submitted or a branch is merged into the main. Please take a look at the docs for instructions on how to run specific tests.

As with unit testing, our goal is coverage of all possible inputs. But rather than think about inputs to low-level functions, for end-to-end tests, coverage is about UX coverage: we want tests that cover everything a creator can do on the platform.

Writing end-to-end tests has its challenges, often stemming from re-creating an environment that mimics someone using a browser. The playwright has a lot of features to make this possible. Don't try to get creative; lean hard on its API, as they've probably thought of your use case before. Many people on the web have used it to verify web apps.

Manual Tests

If you genuinely can't write an automated test for something, you can write a manual one. This is like unit or end-to-end tests but a natural language description of a sequence of inputs and expected outputs because people run these tests rather than a computer need to be as unambiguous as possible.

The major downside of manual testing is that it's not repeatable: every time Wordplay's implementation changes, you have to do all that clicking and typing again, in the same order, to see if things still work. So, we generally want to avoid relying on manual testing alone.

To write a manual test, create a branch for your test(s), create a .md text file in tests/manual/ in the central repository, and follow the template in the README.md file in that folder.

Fix bugs

If your tests are successful, they may reveal bugs. Your job is to diagnose and fix them, per the develop role guidelines. Please help if you get stuck debugging them or need more knowledge about how Wordplay is architected to identify a fix.

Submit your pull request (PR)

Once your tests pass as intended and cover the scope of tests you planned to write, commit them, including Fixed #[issue number] in your commit message. Once merged, this will close the issue. Then, submit a pull request for review.

Revise your tests until approved. Iterate with the PR reviewer until your pull request is approved and merged.

Once you're done, go write more tests!