Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SanityCheck: take inspiration from what verifyproblem does (kattis tools) #87

Open
wil93 opened this issue Nov 28, 2022 · 1 comment
Open
Assignees

Comments

@wil93
Copy link
Member

wil93 commented Nov 28, 2022

The verifyproblem tool (see https://github.com/Kattis/problemtools) has many interesting sanity checks that we most likely would benefit from, for example:

  • it warns if the input / output files don't have a newline at the end of the file
  • it performs some fuzzying of the validators, it checks that at least one validator rejects inputs with:
    • spaces added where there already is whitespace
    • newlines added where there already are newlines
    • leading zeros added to integers
    • trailing zeros added to real number decimal portion
    • random junk added to the end of the file

And I guess other checks that I don't remember or that I didn't run into

@edomora97
Copy link
Collaborator

edomora97 commented Dec 2, 2022

I've skimmed through the verifyproblem source code and listed the interesting checks:

  1. Check input/output newline format (\n vs \r\n)
  2. Check attachment input/output newline format (\n vs \r\n)
  3. Check input files end with \n
  4. Check attached files end with \n
  5. Identical input files in a subtask
  6. No directory in attachments
  7. Fuzz custom checker with junk files (should reject with score 0 and no crash):
    a. an empty file
    b. a binary file with byte values 0 up to 256
    c. a text file with the ASCII characters 32 up to 127
    d. a random text file with printable ASCII characters
  8. Fuzz custom checker with junk modifications (when no grader: should not change the score):
    a. spaces added where there already is whitespace
    b. newlines added where there already are newlines
    c. leading zeros added to integers
    d. trailing zeros added to real number decimal portion
    e. random junk added to the end of the file

Most of them make sense, others may require some discussions (e.g. (1) what about with binary I/O? (5) may be useful? I guess no).
All (or most of them at least) should be quite easy to implement.

@edomora97 edomora97 self-assigned this Dec 3, 2022
edomora97 added a commit that referenced this issue Dec 3, 2022
edomora97 added a commit that referenced this issue Dec 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants