Skip to content

Commit

Permalink
README: Add documentation on how to write new tests
Browse files Browse the repository at this point in the history
This is really just an example on repository dumping for now:
we should add detail about RepositorySimulator helpers as well.

Signed-off-by: Jussi Kukkonen <[email protected]>
  • Loading branch information
jku committed Jul 30, 2024
1 parent 880b344 commit 1074eaf
Showing 1 changed file with 86 additions and 3 deletions.
89 changes: 86 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,96 @@ make test-python-tuf
make test-go-tuf
```

It's also possible to locally run the test suite with a client-under-test CLI from another location:
It's also possible to locally run the test suite with a client-under-test CLI that is locally installed elsewhere:

```bash
pip install -e $CONFORMANCE_SUITE_DIR
pytest "$CONFORMANCE_SUITE_DIR/tuf_conformance" --entrypoint path/to/my/client-under-test/cli
make dev
./env/bin/pytest tuf_conformance --entrypoint path/to/my/client-under-test/cli
```

### Creating a new test

Let's say we want to test that clients will not accept a decreasing targets version. We start with a simple skeleton
test that doesn't assert anything but sets up a repository and runs client refresh:

```python
def test_targets_version(
client: ClientRunner, server: SimulatorServer
) -> None:
# Initialize our repository: modify targets version and make sure the version is included in snapshot
init_data, repo = server.new_test(client.test_name)
repo.targets.version = 7
repo.update_snapshot()

# initialize client-under-test, run refresh
client.init_client(init_data)
client.refresh(init_data)
```

we can now run the test:
```bash
./env/bin/pytest tuf_conformance \
-k test_targets_version # run a specific test only
--entrypoint "./clients/go-tuf/go-tuf" # use the included go-tuf client as client-under-test
--repository-dump-dir /tmp/test-repos # dump repository contents

# take a look at the repository targets.json that got debug dumped
cat /tmp/test-repos/test_targets_version/1/targets.json
cat /tmp/test-repos/test_targets_version/1/snapshot.json
```

The metadata looks as expected (targets version is 7) so we can add a modification to the end of the test:

```python
# Make an non-compliant change in repository
repo.targets.version = 6
repo.update_snapshot()

# refresh client again
client.refresh(init_data)
```

Running the test again results in a second repository version being dumped (each client refresh leads to a dump):
```bash
./env/bin/pytest tuf_conformance \
-k test_targets_version # run a specific test only
--entrypoint "./clients/go-tuf/go-tuf" # use the included go-tuf client as client-under-test
--repository-dump-dir /tmp/test-repos # dump repository contents

# take a look at targets versions in both snapshot versions that got debug dumped
cat /tmp/test-repos/test_targets_version/1/snapshot.json
cat /tmp/test-repos/test_targets_version/2/snapshot.json
```

The repository metadata looks as expected (but not spec-compliant) so we can add some asserts for client behaviour now.
The final test looks like this:

```python
def test_targets_version(
client: ClientRunner, server: SimulatorServer
) -> None:
# Initialize our repository: modify targets version and make sure it's included in snapshot
init_data, repo = server.new_test(client.test_name)
repo.targets.version = 7
repo.update_snapshot() # snapshot v2

# initialize client-under-test
client.init_client(init_data)

# Run refresh on client-under-test, expect success
assert client.refresh(init_data) == 0

# Make a non-compliant change in repository
repo.targets.version = 6
repo.update_snapshot() # snapshot v3

# refresh client again, expect failure and refusal to accept snapshot and targets
assert client.refresh(init_data) == 1
assert client.version(Snapshot.type) == 2
assert client.version(Targets.type) == 7
```


### Some design notes

* pytest is used as the test infrastructure, the client-under-test executable is given with `--entrypoint`
Expand Down

0 comments on commit 1074eaf

Please sign in to comment.