Skip to content

Commit

Permalink
Document how to add tests
Browse files Browse the repository at this point in the history
Fixes gbdev#1300's last remaining item.
Also add `checkdiff` rules to remind us to update that doc if
any of the test driver scripts are updated.
  • Loading branch information
ISSOtm committed Mar 1, 2024
1 parent 2ccdf0c commit b81ebb3
Show file tree
Hide file tree
Showing 2 changed files with 97 additions and 0 deletions.
84 changes: 84 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,3 +69,87 @@ header.
6. Create a pull request against the branch `master`.
7. Be prepared to get some comments about your code and to modify it. Tip: Use
`git rebase -i origin/master` to modify chains of commits.

## Adding a test

The test suite is a little ad-hoc, so the way tests work is different for each program being tested.

Feel free to modify how the test scripts work,

### RGBASM

Each `.asm` file corresponds to one test.
RGBASM will be invoked on the `.asm` file.

If a `.out` file exists, RGBASM's output (`println` etc.) must match its contents.
If a `.err` file exists, RGBASM's errors (`warn`, errors, etc.) must match its contents.

If a `.out.bin` file exists, the object file will be linked, and the generated ROM truncated to the length of the `.out.bin` file.
After that, the ROM must match the `.out.bin` file.

### RGBLINK

Each `.asm` file corresponds to one test, or one *set* of tests.

All tests begin by assembling the `.asm` file into an object file, which will be linked in various ways depending on the test.

#### Simple tests

These simply check that RGBLINK's output matches... *something*.
A `.out` file **must** exist, and RGBLINK's output must match that file's contents.

Additionally, if a `.out.bin` file exists, the `.gb` file generated by RGBLINK must match it.

#### Linker script tests

These allow applying various linker scripts to the same object file.
If one or more `.link` files exist, whose names start the same as the `.asm` file, then each of those files correspond to one test.

Each `.link` linker script **must** be accompanied by a `.out` file, and RGBLINK's output must match that file's contents when passed the corresponding linker script.

#### Variant tests

These allow testing RGBLINK's `-d`, `-t`, and `-w` flags.
If one or more files called `<asm file name>-<flag>.out` or `<asm file name>-no-<flag>.out` exist, then each of them corresponds to one test.

The object file will be linked with and without said flag, respectively; and in each case, RGBLINK's output must match the `.out` file's contents.

### RGBFIX

Each `.bin` file corresponds to one test, and **must** be accompanied by a `.flags` file and a `.err` file.

The `.flags` file is a text file whose first line contains flags to pass to RGBFIX.
(There may be more lines, which will be ignored; they can serve as commentsto explain what the test is about.)

RGBFIX will be invoked on the `.bin` file, and its error output must match the contents of the `.err` file.
(If no errors ought to be printed, then the `.err` file should just be empty.)

Additionally, if a `.gb` file exists, the output of RGBFIX must match the `.gb`.

### RGBGFX

Each `.png` file corresponds to one test.
RGBGFX will be invoked on the file.
If a `.flags` file exists, it will be used as part of the RGBGFX invocation (`@<file>.flags`).

If no `.err` file exists, RGBGFX is simply expected to be able to process the file normally.
If one *does* exist, RGBGFX's return status is ignored, but its output **must** match the `.err` file's contents.

### Downstream projects

1. Make sure the downstream project supports `make <target> RGBDS=<path to RGBDS>`.
While the test suite supports any Make target name, only [Make](//gnu.org/software/make) is currently supported, and the Makefile must support a `RGBDS` variable to use a non-system RGBDS.

Only projects hosted on GitHub are currently supported, too.
2. Add the project to `test/fetch-test-deps.sh`: add a new `action` line at the bottom, following the existing pattern:

```sh
action <owner>/<repo> <date of last commit> <hash of last commit>
```

(The date is used to avoid fetching too much history when cloning the repositories.)
3. Add the project to `test/run-tests.sh`: add a new `test_downstream` line at the bottom, following the existing pattern:

```sh
test_downstream <repo> <Make target>
```
13 changes: 13 additions & 0 deletions contrib/checkdiff.bash
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,16 @@ dependency src/gfx/main.cpp contrib/zsh_compl/_rgbgfx \
"Did the rgbgfx CLI change?"
dependency src/gfx/main.cpp contrib/bash_compl/_rgbgfx.bash \
"Did the rgbgfx CLI change?"

dependency test/fetch-test-deps.sh CONTRIBUTING.md \
"Did the test protocol change?"
dependency test/run-tests.sh CONTRIBUTING.md \
"Did the test protocol change?"
dependency test/asm/test.sh CONTRIBUTING.md \
"Did the RGBASM test protocol change?"
dependency test/link/test.sh CONTRIBUTING.md \
"Did the RGBLINK test protocol change?"
dependency test/fix/test.sh CONTRIBUTING.md \
"Did the RGBFIX test protocol change?"
dependency test/gfx/test.sh CONTRIBUTING.md \
"Did the RGBGFX test protocol change?"

0 comments on commit b81ebb3

Please sign in to comment.